Gaussian Processes

A Gaussian process (GP) is a distribution over functions $f:\Omega \to \mathbb{R}$ such that, for every finite set of input points $x_1,\dots ,x_n\in \Omega$, the random vector $(f(x_1),\dots ,f(x_n))$ is multivariate Gaussian. A GP is characterized by its mean function $\mu (x)=\mathbb{E}[f(x)]$ (almost always taken as zero) and its covariance function or kernel $K(x,x^{\prime })=\text{Cov}(f(x),f(x^{\prime }))$. The kernel must be positive semi-definite: for any finite set $u_1,\dots ,u_N\in \Omega$, the Gram matrix $\mathbf{K}=(K(u_i,u_j){)}_{i,j}$ must satisfy ${\mathbf{v}}^{\top }\mathbf{Kv}\ge 0$ for all $\mathbf{v}$.

GPs are used in three canonical tasks:

• Interpolation: given exact function values $f(x_1),\dots ,f(x_n)$, estimate $f(x)$ at a new point.
• Regression: given noisy observations $y_i=f(x_i)+{ϵ}_i$ with ${ϵ}_i\stackrel{\text{iid}}{\sim }\mathcal{N}(0,{\sigma }^2)$, estimate $f(x)$.
• Numerical integration: given exact values, estimate $\int f(x)dx$.

For all three, the posterior mean reduces to a kernel-weighted linear combination of the observed values, and the posterior is a new GP with closed-form mean and kernel. The choice of kernel encodes the prior beliefs about smoothness, stationarity, and scale of $f$.

$$\text{(Regression posterior)}f(x)\mid \mathbf{y}\sim \mathcal{N}\left({\mathbf{k}}^{\top }(\mathbf{K}+{\sigma }^2\mathbf{I}{)}^{-1}\mathbf{y},K(x,x)-{\mathbf{k}}^{\top }(\mathbf{K}+{\sigma }^2\mathbf{I}{)}^{-1}\mathbf{k}\right),$$

where ${\mathbf{K}}_{ij}=K(x_i,x_j)$ and ${\mathbf{k}}_i=K(x,x_i)$. Setting ${\sigma }^2=0$ recovers the interpolation case.

Prerequisites

• 3.1 Bayesian Framework — prior, likelihood, posterior.
• 6.2 Multivariate Gaussian Distribution — the conditional formula for a jointly Gaussian vector is the entire engine of GP inference.
• 3.5.2 Normal-Normal Model — one-dimensional version of the same posterior.

Contents

• 3.10.1 Kernels — kernel design (Brownian motion, scaled Brownian plus constant, Integrated Brownian Motion, squared-exponential/RBF); algebra of valid kernels.
• 3.10.2 GP Regression — posterior for noisy observations; Sherman-Morrison-Woodbury for the infinite-variance intercept; hyperparameter estimation.
• 3.10.3 Integrated Brownian Motion — canonical kernel whose posterior mean is a natural cubic spline.

Shared Structure

Every GP application in this folder starts from the same joint Gaussian assumption. Write $\mathbf{f}=(f(x_1),\dots ,f(x_n){)}^{\top }$ and $\mathbf{K}$ for the $n\times n$ Gram matrix. For exact observations (interpolation), the joint distribution

and the standard Gaussian conditioning formula give

For noisy observations (regression), $\mathbf{K}$ is replaced by $\mathbf{K}+{\sigma }^2\mathbf{I}$. The only thing that changes across applications is the choice of kernel.

When to Use Which Kernel

Three signals determine kernel choice:

1. Smoothness prior. How differentiable is $f$ believed to be? Brownian motion gives continuous-but-nowhere-differentiable samples, Integrated Brownian Motion gives once-differentiable samples and produces cubic-spline posteriors, and squared-exponential gives infinitely-differentiable samples.
2. Boundary behavior. Brownian motion forces $f(0)=0$; scaled-BM-plus-constant drops this. Integrated-Brownian-Motion-plus-linear-term relaxes further by letting the linear component have unrestricted intercept and slope.
3. Stationarity. The squared-exponential and Matérn kernels depend only on $\Vert x-x^{\prime }\Vert$, so they are stationary and isotropic — the prior looks the same everywhere in $\Omega$. Brownian and Integrated Brownian kernels are non-stationary: variance grows with $x$.

The scaled-IBM-plus-linear-term kernel is the workhorse for STAT 238’s Bayesian nonparametric regression because (a) the posterior mean is a natural cubic spline — a widely used smoother — and (b) it connects directly to the ReLU-basis high-dimensional linear regression in 3.8.1 Bayesian Linear Regression.

Connection to Other Bayesian Methods

The GP with ReLU basis is obtained as a limit of high-dimensional linear regression. Placing priors ${\beta }_2,\dots ,{\beta }_m\stackrel{\text{iid}}{\sim }\mathcal{N}(0,{\tau }^2)$ on the coefficients of a ReLU basis $\{1,x,\text{ReLU}(x-1),\dots ,\text{ReLU}(x-(m-1))\}$ and sending the grid density to infinity gives exactly the Integrated-Brownian-Motion-plus-linear-term GP. This makes GP regression the nonparametric limit of Bayesian ridge regression with a specific basis.

Inference when the kernel has hyperparameters (noise $\sigma$, signal scale $\gamma$) proceeds by computing the marginal likelihood $p(\mathbf{y}\mid \sigma ,\gamma )$ and either maximizing it (empirical Bayes) or integrating it against a prior — see 3.7 Marginal Likelihood. The Sherman-Morrison-Woodbury identity is the key tool for handling improper $\mathcal{N}(0,C)$ priors on linear-basis coefficients in the $C\to \infty$ limit.

See Also

• 3. Bayesian Statistics — parent index.
• 3.8 Bayesian Regression — parametric cousin; GP regression is the nonparametric limit.
• 3.6 Hierarchical Models — hyperparameter priors give hierarchical GPs.
• 3.9 Laplace Approximation — used when the likelihood is non-Gaussian (classification).
• 4.6 Brownian Motion — underlying stochastic process for the canonical kernels.