Normal-Normal Model

Normal-Normal Conjugacy (Known Variance)

Let the unknown parameter be a mean $\mu \in \mathbb{R}$ and the observations be i.i.d. Gaussian with known variance ${\sigma }^2$. The Normal-Normal conjugate pair is
$\mu \sim N({\mu }_0,{\tau }_0^2),y_i\mid \mu \stackrel{\text{iid}}{\sim }N(\mu ,{\sigma }^2),i=1,\dots ,n,$
which produces the closed-form posterior
$\mu \mid y\sim N({\mu }_n,{\tau }_n^2),\frac{1}{{\tau }_n^2}=\frac{1}{{\tau }_0^2}+\frac{n}{{\sigma }^2},{\mu }_n={\tau }_n^2\left(\frac{{\mu }_0}{{\tau }_0^2}+\frac{n\bar{y}}{{\sigma }^2}\right).$
Here $\bar{y}=n^{-1}{\sum }_i{y}_i$ is the sample mean. The update rule is clean in the precision parametrization $\kappa =1/{\tau }^2$: posterior precision equals prior precision plus data precision, and the posterior mean is the precision-weighted average
${\mu }_n=\frac{{\kappa }_0{\mu }_0+(n/{\sigma }^2)\bar{y}}{{\kappa }_0+n/{\sigma }^2}.$
The prior’s “effective sample size” is $n_0={\sigma }^2/{\tau }_0^2$: the prior carries as much information as $n_0$ pseudo-observations of value ${\mu }_0$.

Posterior summaries:

• Mean, median, mode all equal ${\mu }_n$ by symmetry of the Gaussian.
• Posterior variance ${\tau }_n^2=1/({\kappa }_0+n/{\sigma }^2)$ shrinks like $O(1/n)$ as data accumulate.
• Posterior predictive for a new observation $\tilde{y}$: $\tilde{y}\mid y\sim N({\mu }_n,{\tau }_n^2+{\sigma }^2)$. The predictive variance decomposes into posterior uncertainty about $\mu$ plus irreducible observation noise.

Noninformative limit. Taking ${\tau }_0^2\to \infty$ (precision $\to 0$) recovers the flat prior on $\mu$ and produces $\mu \mid y\sim N(\bar{y},{\sigma }^2/n)$, numerically identical to the frequentist sampling distribution of $\bar{y}$. The posterior mean equals the MLE and the posterior variance equals the squared standard error.

Normal-Inverse-Gamma Conjugacy (Unknown Mean and Variance)

When both the mean $\mu$ and variance ${\sigma }^2$ are unknown, the conjugate prior is the Normal-Inverse-Gamma family, written $(\mu ,{\sigma }^2)\sim \text{NIG}({\mu }_0,{\kappa }_0,{\alpha }_0,{\beta }_0)$:
${\sigma }^2\sim \text{InvGamma}({\alpha }_0,{\beta }_0),\mu \mid {\sigma }^2\sim N\left({\mu }_0,{\sigma }^2/{\kappa }_0\right).$
The parameter ${\kappa }_0$ is the prior effective sample size for $\mu$, and $({\alpha }_0,{\beta }_0)$ parametrize the prior for ${\sigma }^2$ with prior mean ${\beta }_0/({\alpha }_0-1)$ for ${\alpha }_0>1$. Given $n$ observations with sample mean $\bar{y}$ and sample sum of squares $S={\sum }_i(y_i-\bar{y}{)}^2$, the posterior is $\text{NIG}({\mu }_n,{\kappa }_n,{\alpha }_n,{\beta }_n)$ with
${\kappa }_n={\kappa }_0+n,{\mu }_n=\frac{{\kappa }_0{\mu }_0+n\bar{y}}{{\kappa }_0+n},{\alpha }_n={\alpha }_0+\frac{n}{2},$
${\beta }_n={\beta }_0+\frac{1}{2}S+\frac{{\kappa }_{0}n(\bar{y}-{\mu }_0{)}^2}{2({\kappa }_0+n)}.$
The three contributions to ${\beta }_n$ are the prior scale, the within-sample dispersion, and a “disagreement penalty” for how far the data mean sits from the prior mean.

Marginal posterior for $\mu$ (integrating out ${\sigma }^2$): Student-$t$ with $2{\alpha }_n$ degrees of freedom,
$\mu \mid y\sim t_{2{\alpha }_n}\left({\mu }_n,\frac{{\beta }_n}{{\alpha }_n{\kappa }_n}\right),$
which collapses to the Gaussian of the known-${\sigma }^2$ case as ${\alpha }_n\to \infty$. This Student-$t$ posterior is why $t$-intervals are the default for credible intervals on a Gaussian mean when the variance is not known in advance; see 3.2 Credible Intervals.

Prerequisites

• 3.1 Bayesian Framework — prior, likelihood, posterior, Bayes’ rule.
• 3.5 Conjugate Families — the general conjugacy pattern this note instantiates.
• 2.2c.3 Gaussian Distribution — the workhorse distribution.
• 2.2c.6 Gamma Distribution — parent of Inverse-Gamma; used for ${\sigma }^2$.
• 2.2c.8 Student’s t Distribution — Student-$t$ arises as the marginal posterior for $\mu$ with unknown ${\sigma }^2$.

Intuition

Gaussian densities are the unique family (up to scaling) whose logarithm is a quadratic in the unknown. Multiplying two Gaussians in $\mu$ therefore produces another Gaussian, because the sum of two quadratics is again a quadratic. The prior $N({\mu }_0,{\tau }_0^2)$ contributes the quadratic $-\frac{1}{2{\tau }_0^2}(\mu -{\mu }_0{)}^2$ and the likelihood contributes $-\frac{n}{2{\sigma }^2}(\mu -\bar{y}{)}^2$; completing the square in $\mu$ yields the posterior Gaussian with precision equal to the sum of the prior and likelihood precisions.

The central mechanism is precision addition. Precision measures the sharpness of information: a prior that asserts $\mu$ is known to within standard deviation ${\tau }_0$ has precision $1/{\tau }_0^2$, and $n$ observations with noise standard deviation $\sigma$ jointly carry precision $n/{\sigma }^2$. These add, and the posterior mean is the weighted average with weights equal to precisions. This is the Bayesian analog of how a Kalman filter fuses information from two sensors, and it generalizes directly to multivariate Gaussians (where precision is the inverse covariance matrix).

When the variance is unknown, the Student-$t$ structure emerges because the posterior for ${\sigma }^2$ is heavy-tailed (Inverse-Gamma), and averaging a Gaussian with variance ${\sigma }^2$ over an Inverse-Gamma for ${\sigma }^2$ produces a Student-$t$ for $\mu$. The degrees-of-freedom parameter tracks how much data has been seen: with $n$ observations and a weakly informative prior, the posterior for $\mu$ is approximately $t_{n-1}$, which recovers the frequentist $t$-distribution of the sample mean.

Properties

(Precision Addition) Posterior precision equals prior precision plus data precision; posterior means are precision-weighted averages.

Writing $\kappa =1/{\tau }^2$ for precision, the Normal-Normal update is
${\kappa }_n={\kappa }_0+n{\kappa }_{\text{data}},{\mu }_n=\frac{{\kappa }_0{\mu }_0+n{\kappa }_{\text{data}}\bar{y}}{{\kappa }_n},$
where ${\kappa }_{\text{data}}=1/{\sigma }^2$. This makes the prior interpretable as $n_0={\kappa }_0/{\kappa }_{\text{data}}={\sigma }^2/{\tau }_0^2$ pseudo-observations of value ${\mu }_0$. The multivariate extension replaces scalar precisions with precision matrices $\Lambda ={\Sigma }^{-1}$, and the same additive law holds: ${\Lambda }_n={\Lambda }_0+n{\Sigma }^{-1}$, ${\mu }_n={\Lambda }_n^{-1}({\Lambda }_0{\mu }_0+n{\Sigma }^{-1}\bar{y})$.

(Shrinkage to the Prior Mean) The posterior mean is between the prior mean and the sample mean, biased toward whichever carries more precision.

Rewriting ${\mu }_n=w{\mu }_0+(1-w)\bar{y}$ with $w={\kappa }_0/{\kappa }_n\in (0,1)$, the posterior mean is a convex combination. When $n$ is small relative to the prior’s effective sample size $n_0$, the posterior sits near ${\mu }_0$; when $n\gg n_0$, it sits near $\bar{y}$. This pattern is the univariate case of the shrinkage behaviour that drives hierarchical models and James-Stein estimation.

(Posterior Predictive is Gaussian with Inflated Variance) New observations are predicted with variance ${\tau }_n^2+{\sigma }^2$, never just ${\sigma }^2$.

For the known-${\sigma }^2$ model, $\tilde{y}\mid y\sim N({\mu }_n,{\tau }_n^2+{\sigma }^2)$. The first term ${\tau }_n^2$ captures remaining uncertainty about $\mu$; the second term ${\sigma }^2$ is irreducible observation noise. A Bayesian predictive interval is strictly wider than the plug-in interval using $\hat{\mu }$, and this is precisely what protects a Bayesian forecast from the overconfidence of plug-in prediction. Under the unknown-${\sigma }^2$ model, the predictive for $\tilde{y}$ is Student-$t$ rather than Gaussian, with location ${\mu }_n$ and scale $\sqrt{{\beta }_n({\kappa }_n+1)/({\alpha }_n{\kappa }_n)}$ on $2{\alpha }_n$ degrees of freedom.

(Sequential Updating) Posterior after $n_1+n_2$ observations equals the posterior after $n_1$ observations used as the prior for the next $n_2$.

Because precision addition is associative, processing data in batches and collapsing to a single posterior yields the same result as processing all data at once. This is the Gaussian special case of the general (Sequential Updating) property in 3.1 Bayesian Framework: treating the posterior as the next prior gives identical inference regardless of batching. This is the fundamental structure of online learning and the Kalman filter.

(Bernstein-von Mises for Normal-Normal) With any proper prior, the posterior converges to $N(\bar{y},{\sigma }^2/n)$ as $n\to \infty$.

The posterior mean ${\mu }_n\to \bar{y}$ and posterior variance ${\tau }_n^2\to {\sigma }^2/n$ because the prior contribution $1/{\tau }_0^2$ becomes negligible against the data contribution $n/{\sigma }^2$. The posterior therefore recovers the frequentist sampling distribution of the MLE, delivering the (Prior Washing Out) property of 3.1 Bayesian Framework exactly and explicitly — for Normal-Normal, BvM is not asymptotic but happens cleanly at every $n$.

(Marginal Posterior is Student-t When ${\sigma }^2$ is Unknown) Integrating out ${\sigma }^2$ from the NIG posterior gives a Student-$t$ marginal for $\mu$.

$\mu \mid y\sim t_{2{\alpha }_n}\left({\mu }_n,\frac{{\beta }_n}{{\alpha }_n{\kappa }_n}\right),$
with $2{\alpha }_n$ degrees of freedom. With a Jeffreys-style noninformative prior $p(\mu ,{\sigma }^2)\propto 1/{\sigma }^2$ (the limit ${\kappa }_0,{\alpha }_0,{\beta }_0\to 0$), this collapses to the classical Student-$t$ result $\mu \mid y\sim t_{n-1}(\bar{y},s^2/n)$, where $s^2=S/(n-1)$ is the unbiased sample variance — the Bayesian posterior and the frequentist sampling distribution coincide numerically. This is why the familiar $t$-interval is also a Bayesian credible interval under the Jeffreys prior.

(Noninformative Prior and Exchangeability with Frequentist Inference) The ${\tau }_0\to \infty$ limit reproduces frequentist formulas exactly.

For known ${\sigma }^2$, the flat-prior posterior is $N(\bar{y},{\sigma }^2/n)$, matching the sampling distribution of $\bar{y}$. For unknown ${\sigma }^2$ with the reference prior $p(\mu ,{\sigma }^2)\propto {\sigma }^{-2}$, the marginal posterior for $\mu$ is $t_{n-1}(\bar{y},s^2/n)$. The Bayesian $(1-\alpha )$ credible interval then coincides with the frequentist $(1-\alpha )$ confidence interval, even though their interpretations differ — see 3.2 Credible Intervals for the contrast.

(Multivariate Extension) For unknown mean vector $\mathbf{\mu }\in {\mathbb{R}}^d$ with known covariance $\mathbf{\Sigma }$, the conjugate prior is multivariate Gaussian.

If $\mathbf{\mu }\sim N({\mathbf{\mu }}_0,{\mathbf{\Sigma }}_0)$ and ${\mathbf{y}}_i\mid \mathbf{\mu }\stackrel{\text{iid}}{\sim }N(\mathbf{\mu },\mathbf{\Sigma })$, then $\mathbf{\mu }\mid \mathbf{y}\sim N({\mathbf{\mu }}_n,{\mathbf{\Sigma }}_n)$ with ${\mathbf{\Sigma }}_n^{-1}={\mathbf{\Sigma }}_0^{-1}+n{\mathbf{\Sigma }}^{-1}$ and ${\mathbf{\mu }}_n={\mathbf{\Sigma }}_n({\mathbf{\Sigma }}_0^{-1}{\mathbf{\mu }}_0+n{\mathbf{\Sigma }}^{-1}\bar{\mathbf{y}})$. For unknown covariance, the conjugate prior is the Normal-Inverse-Wishart, the multivariate analog of NIG, used throughout Bayesian linear regression; see 3.8.1 Bayesian Linear Regression.

Examples

Example 1: Measurement Calibration (Known ${\sigma }^2$)

A scale has calibration mean $\mu$ (unknown) and known measurement noise standard deviation $\sigma =0.1$ g. A prior based on the manufacturer’s specification is $\mu \sim N(100,1)$ (prior mean 100 g, prior standard deviation 1 g). Ten calibration weighings of a reference mass yield sample mean $\bar{y}=99.87$ g.

Prior precision: ${\kappa }_0=1/1^2=1$.
Data precision: $n/{\sigma }^2=10/0.01=1000$.
Posterior precision: ${\kappa }_n=1001$, hence ${\tau }_n^2=1/1001\approx 0.000999$, ${\tau }_n\approx 0.0316$ g.
Posterior mean: ${\mu }_n=(1\cdot 100+1000\cdot 99.87)/1001=99.87013$ g.

The prior’s effective sample size is $n_0={\sigma }^2/{\tau }_0^2=0.01$, so 10 observations overwhelm the prior entirely. The posterior sits essentially at the sample mean with a tight credible interval $[{\mu }_n-1.96{\tau }_n,{\mu }_n+1.96{\tau }_n]\approx [99.81,99.93]$ g.

Predictive for the next weighing: $\tilde{y}\mid y\sim N(99.870,{\tau }_n^2+{\sigma }^2)=N(99.870,0.01100)$, i.e. a predictive standard deviation of $\approx 0.105$ g — only slightly wider than pure noise because the posterior uncertainty about $\mu$ is already very small.

Example 2: Polling with an Informative Prior

Suppose pre-election support for a candidate has been running at 52% with historical standard deviation 2%: prior $\mu \sim N(0.52,0.02^2)$. A new poll samples $n=400$ respondents with observed sample proportion $\bar{y}=0.48$ and reported standard error $\sigma =\sqrt{0.48\cdot 0.52/400}\approx 0.025$. Treating the sample proportion as approximately Gaussian with known $\sigma$:

Precisions: ${\kappa }_0=1/0.02^2=2500$; ${\kappa }_{\text{data}}=1/{\sigma }^2=1/0.025^2=1600$.
Posterior precision: ${\kappa }_n=2500+1600=4100$, so ${\tau }_n=1/\sqrt{4100}\approx 0.01562$.
Posterior mean: ${\mu }_n=(2500\cdot 0.52+1600\cdot 0.48)/4100\approx 0.5044$.

The posterior sits closer to the informative prior than to the new poll because the prior carries more precision than the single poll. A frequentist reading the poll alone would see $0.48\pm 1.96\cdot 0.025=[0.431,0.529]$; the Bayesian credible interval $[0.5044-1.96\cdot 0.0156,0.5044+1.96\cdot 0.0156]=[0.474,0.535]$ is narrower and shifted, reflecting the prior information. As more polls accumulate, the prior’s 2500 pseudo-observations are gradually outweighed by the accumulating $n$.

Example 3: Unknown Variance — Student- $t$ Credible Interval

A sample of $n=8$ i.i.d. observations from $N(\mu ,{\sigma }^2)$ (both unknown) has sample mean $\bar{y}=5.2$ and sample sum of squares $S=14.0$. Using the reference prior $p(\mu ,{\sigma }^2)\propto 1/{\sigma }^2$ (equivalent to ${\kappa }_0,{\alpha }_0,{\beta }_0\to 0$ in NIG):

Posterior parameters: ${\kappa }_n=n=8$, ${\mu }_n=\bar{y}=5.2$, ${\alpha }_n=n/2=4$, ${\beta }_n=S/2=7.0$.
Marginal posterior for $\mu$: $\mu \mid y\sim t_{2{\alpha }_n}\left({\mu }_n,{\beta }_n/({\alpha }_n{\kappa }_n)\right)=t_8\left(5.2,0.21875\right)$.

The posterior scale is $\sqrt{0.21875}\approx 0.468$. A 95% equal-tail credible interval uses the 0.975 quantile of $t_8$, which is 2.306:
$[5.2-2.306\cdot 0.468,5.2+2.306\cdot 0.468]=[4.12,6.28].$
This matches the classical 95% $t$-interval $\bar{y}\pm t_{n-1,0.975}s/\sqrt{n}$ with $s^2=S/(n-1)=2.0$, so $s/\sqrt{n}=\sqrt{2/8}\approx 0.500$ and $t_{7,0.975}=2.365$, giving $[4.02,6.38]$. The slight numerical difference comes from using the posterior’s $2{\alpha }_n=8$ degrees of freedom instead of $n-1=7$; it reflects the choice between the Jeffreys prior (${\alpha }_0=0$) and the marginally stronger “classical” convention that debits one degree of freedom for estimating ${\sigma }^2$.

Derivation

Goal: derive the posterior $\mu \mid y\sim N({\mu }_n,{\tau }_n^2)$ under the known-variance Normal-Normal model.

Starting from Bayes’ rule, the log posterior is
$\log p(\mu \mid y)=\log p(\mu )+{\sum }_{i=1}^n\log p(y_i\mid \mu )+\text{const},$
with both terms quadratic in $\mu$:
$\log p(\mu )=-\frac{1}{2{\tau }_0^2}(\mu -{\mu }_0{)}^2+c_1,$
${\sum }_i\log p(y_i\mid \mu )=-\frac{1}{2{\sigma }^2}{\sum }_i(y_i-\mu {)}^2+c_2=-\frac{n}{2{\sigma }^2}(\mu -\bar{y}{)}^2+c_3.$
Adding the two quadratics and completing the square in $\mu$:
$\log p(\mu \mid y)=-\frac{1}{2}\left(\frac{1}{{\tau }_0^2}+\frac{n}{{\sigma }^2}\right){\left(\mu -{\mu }_n\right)}^2+c,$
where ${\mu }_n$ is the weighted average ${\tau }_n^2({\mu }_0/{\tau }_0^2+n\bar{y}/{\sigma }^2)$ and ${\tau }_n^2=(1/{\tau }_0^2+n/{\sigma }^2{)}^{-1}$. Exponentiating gives the claimed $N({\mu }_n,{\tau }_n^2)$ posterior.

The posterior-predictive derivation is similar: marginalize $\tilde{y}$ over the Gaussian posterior for $\mu$,
$\tilde{y}=\mu +ϵ,\mu \sim N({\mu }_n,{\tau }_n^2),ϵ\sim N(0,{\sigma }^2),$
with $\mu$ and $ϵ$ independent. The sum of independent Gaussians is Gaussian with added variances: $\tilde{y}\mid y\sim N({\mu }_n,{\tau }_n^2+{\sigma }^2)$.

See Also

• 3.5 Conjugate Families — parent index.
• 3.5.1 Beta-Binomial Model — the canonical conjugate-pair template.
• 3.5.3 Gamma-Poisson Model — count-data analog.
• 3.5.4 Dirichlet-Multinomial Model — categorical analog.
• 3.2 Credible Intervals — Student-$t$ credible intervals arise from the NIG posterior.
• 3.6 Hierarchical Models — Normal-Normal is the building block for Gaussian hierarchies and James-Stein shrinkage.
• 3.8.1 Bayesian Linear Regression — multivariate Normal-Normal extended to regression coefficients.
• 2.2c.3 Gaussian Distribution — underlying distribution.
• 2.2c.8 Student’s t Distribution — arises as the marginal posterior for $\mu$ with unknown variance.