Bayesian Framework

Bayesian Framework

The Bayesian framework treats an unknown parameter $\theta \in \Theta$ as a random variable with a probability distribution. Before observing data, beliefs about $\theta$ are encoded in a prior density $p(\theta )$; the likelihood $p(y\mid \theta )$ specifies how the observed data $y$ are generated given $\theta$. Bayes’ rule combines them into the posterior
$p(\theta \mid y)=\frac{p(\theta )p(y\mid \theta )}{p(y)}\propto p(\theta )p(y\mid \theta ),$
where the denominator $p(y)={\int }_{\Theta }p(\theta )p(y\mid \theta )d\theta$ is the marginal likelihood (also called the evidence). The posterior is the complete Bayesian answer: every inferential question (point estimates, credible intervals, model comparison, predictions) is a deterministic functional of $p(\theta \mid y)$.

Point estimates are summaries of the posterior:

• Posterior mean: ${\hat{\theta }}_{\text{PM}}=\mathbb{E}[\theta \mid y]=\int \theta p(\theta \mid y)d\theta$ — minimizes posterior squared-error loss.
• Posterior mode (MAP): ${\hat{\theta }}_{\text{MAP}}=\arg {\max }_{\theta }p(\theta \mid y)=\arg {\max }_{\theta }[p(\theta )p(y\mid \theta )]$ — minimizes posterior $0$-$1$ loss. See MAP.
• Posterior median: minimizes posterior absolute-error loss.

The posterior predictive distribution of a new observation $\tilde{y}$ given data $y$ is
$p(\tilde{y}\mid y)={\int }_{\Theta }p(\tilde{y}\mid \theta )p(\theta \mid y)d\theta ,$
which averages the sampling distribution $p(\tilde{y}\mid \theta )$ over posterior uncertainty in $\theta$.

Likelihood principle. All evidence about $\theta$ contained in data $y$ enters inference only through the likelihood function $\ell (\theta )=p(y\mid \theta )$, viewed as a function of $\theta$ with $y$ fixed. Two experiments whose likelihood functions differ only by a $\theta$-free multiplicative constant must yield the same posterior (and therefore the same Bayesian inference). One practical consequence is stopping-rule invariance: a data-dependent stopping rule (e.g., “stop when $3$ successes are seen” vs. “stop after $n$ trials”) does not affect the posterior, because the part of the sampling distribution that depends on the stopping rule is $\theta$-free and cancels. This contrasts sharply with frequentist $p$-values, which do depend on the stopping rule.

Because $p(y)$ does not depend on $\theta$, the unnormalized posterior $p(\theta )p(y\mid \theta )$ is enough to identify the posterior shape, and most Bayesian computation (including MCMC) works with the unnormalized form and never computes $p(y)$.

Intuition

The prior encodes what is believed about $\theta$ before seeing data; the likelihood encodes what the data say about $\theta$. Bayes’ rule multiplies them together and renormalizes — the posterior is a compromise between the two, pulled toward the prior where the prior is concentrated and pulled toward the MLE where the likelihood is concentrated. As the sample size grows, the likelihood sharpens and eventually dominates any fixed prior, so the Bayesian posterior concentrates around the MLE; for small samples the prior can meaningfully shift the answer, which is both the strength and the controversy of the framework.

The likelihood principle is what makes this machinery internally consistent. If the sole role of data is to multiply the prior by $p(y\mid \theta )$, then anything about the data-collection process that does not change the shape of $p(y\mid \theta )$ as a function of $\theta$ is irrelevant to inference. Stopping rules, sampling plans, and the experimenter’s intentions all fall into this category: they change the normalizing constant of the likelihood but not its shape, so they change nothing. Frequentist procedures, which average over the full sampling distribution of the data, are not invariant to these choices and can give different answers from the same observed likelihood.

Properties

(Posterior Proportionality) Only the $\theta$-dependence of prior $\times$ likelihood matters; the evidence $p(y)$ is a normalizing constant.

Writing $p(\theta \mid y)\propto p(\theta )p(y\mid \theta )$ and discarding factors that do not depend on $\theta$ is the workhorse computation of Bayesian statistics. For conjugate families, the right-hand side is proportional to a known distribution’s density and the posterior is identified by inspection. For non-conjugate problems, the shape of the unnormalized posterior is all MCMC needs: the ratio $p({\theta }^{\prime })p(y\mid {\theta }^{\prime })/p(\theta )p(y\mid \theta )$ that appears in the Metropolis-Hastings acceptance probability cancels the evidence entirely.

(Sequential Updating) Data can be absorbed one observation at a time and the final posterior is the same as using all data at once.

For conditionally independent data $y_1,\dots ,y_n$ given $\theta$, Bayes’ rule factors as
$p(\theta \mid y_{1:n})\propto p(\theta ){\prod }_{i=1}^{n}p(y_i\mid \theta ),$
so applying Bayes’ rule with $p(\theta \mid y_{1:k})$ as the “prior” for observation $y_{k+1}$ yields $p(\theta \mid y_{1:k+1})$. Today’s posterior becomes tomorrow’s prior. This makes the framework natural for online learning and recursive filtering, and it is why the conjugate-update formulas are stated in the “prior-$\to$-posterior” form: the hyperparameters of the posterior are the hyperparameters of the prior plus a contribution from the data.

(Likelihood Principle) Two datasets whose likelihoods agree up to a $\theta$-free constant yield the same Bayesian inferences.

If ${\ell }_1(\theta )=c\cdot {\ell }_2(\theta )$ for some constant $c>0$ that does not depend on $\theta$, then $p(\theta ){\ell }_1(\theta )\propto p(\theta ){\ell }_2(\theta )$ and the two posteriors are identical. Stopping-rule invariance is the canonical consequence. Consider two experiments on a Bernoulli parameter $\theta$: (a) fix $n=12$ trials and count successes, observing $y=3$; (b) keep sampling until $3$ successes are observed, requiring $n=12$ total trials. The binomial and negative-binomial likelihoods are
${\ell }_a(\theta )=\left(\genfrac{}{}{0ex}{}{12}{3}\right){\theta }^3(1-\theta {)}^9,{\ell }_b(\theta )=(\genfrac{}{}{0ex}{}{11}{2}){\theta }^3(1-\theta {)}^9,$
which differ only by the combinatorial prefactor. Both produce the same posterior under any prior on $\theta$. A frequentist $p$-value testing $H_0:\theta =0.5$ is $0.073$ under the binomial design and $0.033$ under the negative-binomial design — the “significance” depends on the experimenter’s intent, which the Bayesian calculation ignores.

(Prior Washing Out) For regular models and large $n$, the posterior concentrates around the MLE at rate $n^{-1/2}$.

Under standard regularity (identifiability, smooth density, interior true parameter), the Bernstein-von Mises theorem gives
$p(\theta \mid y_{1:n})\approx N\left({\hat{\theta }}_{\text{MLE}},\mathcal{I}({\hat{\theta }}_{\text{MLE}}{)}^{-1}/n\right)$
for any fixed prior with positive density at the true $\theta$, where $\mathcal{I}$ is the Fisher information. As $n\to \infty$ the prior’s influence is $O(n^{-1})$ and the likelihood’s influence is $O(1)$, so the posterior tracks the likelihood. This justifies the common heuristic “when the data are plentiful, the prior doesn’t matter” and connects the Laplace approximation to a large-$n$ Gaussian posterior.

(Posterior Predictive as Calibration) Predictions automatically inherit parameter uncertainty via the posterior predictive.

The plug-in predictive $p(\tilde{y}\mid \hat{\theta })$ uses a single parameter value and underestimates predictive variance — when $\hat{\theta }$ is itself uncertain, the true predictive spread is wider than a single likelihood under $\hat{\theta }$ suggests. The posterior predictive $p(\tilde{y}\mid y)=\int p(\tilde{y}\mid \theta )p(\theta \mid y)d\theta$ integrates out $\theta$, so predictive intervals automatically widen when the posterior is diffuse and tighten as $n$ grows. For conjugate models this integral is often closed-form (e.g., Beta-Binomial gives a beta-binomial predictive); for non-conjugate models it is approximated by drawing ${\theta }^{(m)}\sim p(\theta \mid y)$ and then ${\tilde{y}}^{(m)}\sim p(\tilde{y}\mid {\theta }^{(m)})$.

(Loss-Matched Point Estimates) The “best” point estimate depends on the loss function, which is why the posterior yields several of them.

A Bayesian point estimate is the minimizer of expected posterior loss: under squared-error loss $L(\theta ,a)=(\theta -a{)}^2$ the minimizer is the posterior mean; under absolute-error loss $|\theta -a|$, the posterior median; under $0$-$1$ loss $1\{\theta \ne a\}$, the posterior mode (MAP). The loss function is an application-level choice, not a property of the posterior, and different losses can give meaningfully different estimates for skewed posteriors. When the posterior is symmetric (e.g., Gaussian) all three collapse to the same point.

(Improper Priors) Priors that do not integrate to one are allowed if the posterior is proper.

A uniform prior on $\theta \in \mathbb{R}$ or a Jeffreys prior $p(\theta )\propto 1/\theta$ on $\theta >0$ is not a probability density, but the posterior $p(\theta \mid y)\propto p(\theta )p(y\mid \theta )$ can still be a proper density if the likelihood provides enough integrability. This is routine in practice — flat priors are the standard starting point for the Bayesian linear regression posterior, which is proper as long as the design matrix has full column rank. Checking posterior propriety is mandatory whenever an improper prior is used: the unnormalized form $p(\theta )p(y\mid \theta )$ looks sensible even when it integrates to $\infty$, and sampling from a non-integrable target can silently produce nonsense.

Examples

Beta-Binomial: the canonical calculation

Let $\theta \in [0,1]$ be an unknown coin-bias and place a $\text{Beta}(\alpha ,\beta )$ prior on $\theta$:
$p(\theta )=\frac{1}{B(\alpha ,\beta )}{\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}.$
Observe $y$ successes in $n$ flips, a $\text{Binomial}(n,\theta )$ sample with likelihood
$p(y\mid \theta )=\left(\genfrac{}{}{0ex}{}{n}{y}\right){\theta }^y(1-\theta {)}^{n-y}.$
The unnormalized posterior is a product of powers of $\theta$ and $1-\theta$:
$p(\theta \mid y)\propto {\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}\cdot {\theta }^y(1-\theta {)}^{n-y}={\theta }^{\alpha +y-1}(1-\theta {)}^{\beta +n-y-1}.$
Recognizing the kernel of a beta density,
$\theta \mid y\sim \text{Beta}(\alpha +y,\beta +n-y).$
The posterior hyperparameters are the prior hyperparameters plus a contribution from the data: $\alpha \mapsto \alpha +y$ and $\beta \mapsto \beta +n-y$. The posterior mean is
$\mathbb{E}[\theta \mid y]=\frac{\alpha +y}{\alpha +\beta +n}=\underset{w_{\text{prior}}}{\underbrace{\frac{\alpha +\beta }{\alpha +\beta +n}}}\cdot \underset{\text{prior mean}}{\underbrace{\frac{\alpha }{\alpha +\beta }}}+\underset{w_{\text{data}}}{\underbrace{\frac{n}{\alpha +\beta +n}}}\cdot \underset{\text{MLE}}{\underbrace{\frac{y}{n}}},$
a weighted average of the prior mean and the MLE with weights determined by “prior sample size” $\alpha +\beta$ vs. data sample size $n$. As $n\to \infty$, the weight on the MLE tends to $1$ — the prior washes out. See 3.5.1 Beta-Binomial Model for the full conjugate development.

Likelihood principle in action: binomial vs. negative-binomial

Alice flips a coin $n=12$ times and records $y=3$ heads (binomial design). Bob flips the same coin until $k=3$ heads appear and records that this took $n=12$ flips (negative-binomial design). Both observe the same $(n,y)=(12,3)$.

The binomial likelihood is ${\ell }_A(\theta )=\left(\genfrac{}{}{0ex}{}{12}{3}\right){\theta }^3(1-\theta {)}^9$. The negative-binomial likelihood is ${\ell }_B(\theta )=\left(\genfrac{}{}{0ex}{}{11}{2}\right){\theta }^3(1-\theta {)}^9$ — the final flip must be a head, so only $11$ flips remain for the first $2$ heads to occur. The two likelihoods differ only by the $\theta$-free combinatorial prefactors $\left(\genfrac{}{}{0ex}{}{12}{3}\right)=220$ and $\left(\genfrac{}{}{0ex}{}{11}{2}\right)=55$.

Under any prior $p(\theta )$ on $[0,1]$, the two posteriors are
$p_A(\theta \mid y)\propto p(\theta ){\theta }^3(1-\theta {)}^9,p_B(\theta \mid n)\propto p(\theta ){\theta }^3(1-\theta {)}^9,$
identical up to renormalization. Every Bayesian summary — posterior mean, MAP, credible interval, Bayes factor against $\theta =0.5$ — is therefore the same for Alice and Bob.

A frequentist, by contrast, computes $P(Y\ge 3\mid \theta =0.5,n=12)=0.073$ under Alice’s binomial design (not significant at $\alpha =0.05$) and $P(N\ge 12\mid \theta =0.5,k=3)=0.033$ under Bob’s negative-binomial design (significant). The “significance” of the observation depends on which experiment Alice or Bob intended to run, even though the data are identical — a violation of the likelihood principle that has no analog in Bayesian analysis.

Posterior predictive for a new coin flip

Continuing the Beta-Binomial setup: after observing $y$ heads in $n$ flips with posterior $\theta \mid y\sim \text{Beta}(\alpha +y,\beta +n-y)$, what is the probability that a new, independent flip is heads?

By the tower rule,
$P(\tilde{y}=1\mid y)=\mathbb{E}[\theta \mid y]=\frac{\alpha +y}{\alpha +\beta +n}.$
Plug-in prediction using the MAP would give ${\hat{\theta }}_{\text{MAP}}(1-{\hat{\theta }}_{\text{MAP}})$ for the predictive variance of $\tilde{y}$; the correct posterior predictive variance is strictly larger because it accounts for uncertainty in $\theta$:
$\mathrm{Var}(\tilde{y}\mid y)=\mathbb{E}[\theta \mid y](1-\mathbb{E}[\theta \mid y]).$
For $m$ new flips, the posterior predictive distribution of the number of heads is a beta-binomial distribution $\text{BetaBin}(m,\alpha +y,\beta +n-y)$, which has fatter tails than a $\text{Binomial}(m,\hat{\theta })$ — the extra variance comes from integrating out $\theta$. In the limit $n\to \infty$ the posterior concentrates at the truth and the beta-binomial converges to the binomial.

See Also

• 3. Bayesian Statistics — parent index.
• 1.6 Bayes’ Rule — the event-level form of Bayes’ rule that the parameter-level version generalizes.
• 1.2.1 Maximum Likelihood Estimation — frequentist counterpart; the MLE is the mode of a flat-prior posterior.
• 1.2.2 Maximum a Posteriori Estimation — the MAP estimate as a posterior summary.
• 3.2 Credible Intervals — interval summaries of the posterior.
• 3.3 Bayes Factors — model comparison via marginal-likelihood ratios.
• 3.5 Conjugate Families — prior-likelihood pairs for which the posterior is closed-form.
• 3.9 Laplace Approximation — Gaussian approximation around the MAP, justified by Bernstein-von Mises.
• 3.11 MCMC — how to sample from posteriors when closed forms fail.