Bayes Factors

Bayes Factor

A Bayes factor compares two models $M_0$ and $M_1$ by the ratio of their marginal likelihoods (also called the evidence of each model):
${\mathrm{BF}}_{10}=\frac{p(y\mid M_1)}{p(y\mid M_0)}=\frac{\int p({\theta }_1\mid M_1)p(y\mid {\theta }_1,M_1)d{\theta }_1}{\int p({\theta }_0\mid M_0)p(y\mid {\theta }_0,M_0)d{\theta }_0}.$
The Bayes factor is the evidence update for the odds between the two models: combined with the prior odds $\pi (M_1)/\pi (M_0)$ and Bayes’ rule,
$\underset{\text{posterior odds}}{\underbrace{\frac{P(M_1\mid y)}{P(M_0\mid y)}}}=\underset{\text{data-driven}}{\underbrace{{\mathrm{BF}}_{10}}}\times \underset{\text{prior odds}}{\underbrace{\frac{\pi (M_1)}{\pi (M_0)}}}.$
${\mathrm{BF}}_{10}>1$ is evidence for $M_1$ relative to $M_0$; ${\mathrm{BF}}_{10}<1$ is evidence for $M_0$.

Interpretation scale (Jeffreys, 1961). Common verbal descriptors of ${\mathrm{BF}}_{10}$:

• $1$ to $3$ — not worth more than a bare mention.
• $3$ to $10$ — substantial evidence for $M_1$.
• $10$ to $30$ — strong evidence.
• $30$ to $100$ — very strong evidence.
• $>100$ — decisive.

These are rules of thumb, not a replacement for reporting $\log {\mathrm{BF}}_{10}$ directly. Log-Bayes-factors are more numerically stable and additive over independent experiments.

Unlike a frequentist $p$-value, the Bayes factor (a) compares two specific models rather than rejecting a null against an unspecified alternative, (b) can provide evidence for the null (small ${\mathrm{BF}}_{10}$ means the data support $M_0$, not “insufficient evidence to reject”), and (c) does not depend on the stopping rule or sampling plan — stopping-rule invariance is inherited from the likelihood principle.

The Jeffreys-Lindley paradox. For a point-null $M_0:\theta ={\theta }_0$ vs. a diffuse alternative $M_1:\theta \sim p_1(\theta )$ with large variance, the Bayes factor can strongly favor $M_0$ while the frequentist $p$-value rejects $M_0$. Concretely, if the prior $p_1$ is flat over a wide range, the marginal likelihood under $M_1$ averages $p(y\mid \theta )$ over regions where $\theta$ is implausible, diluting it; meanwhile the $p$-value sees only the likelihood at ${\theta }_0$ relative to the sampling distribution and can be small. The paradox is a genuine divergence between Bayesian and frequentist testing, not a bug in either framework — it reflects that the two procedures answer different questions.

Computation. Closed-form for conjugate pairs; otherwise use the Laplace approximation, bridge sampling, thermodynamic integration, or reversible-jump MCMC.

Intuition

The marginal likelihood $p(y\mid M)=\int p(\theta \mid M)p(y\mid \theta ,M)d\theta$ is the average of the likelihood under the prior. A model with a sharp prior concentrated near the truth has a large marginal likelihood; a model with a diffuse prior spreads its bet and has a smaller one. This is the built-in Occam penalty for Bayesian model comparison: a model with more parameters or a wider prior pays a cost for integrating over parameter values that fit the data poorly, even if its best fit is as good as the simpler model’s. No explicit complexity penalty needs to be added — it is baked into the evidence.

The Bayes factor is sensitive to the prior in a way point estimates and credible intervals are not. The posterior under a given model “washes out” a vague prior for large $n$, but the evidence for that model depends on the prior directly through $\int p(\theta \mid M)p(y\mid \theta ,M)d\theta$. This is both a strength (the prior’s honesty is what makes the Occam penalty work) and a weakness (vague priors can be manipulated to favor either hypothesis). The Jeffreys-Lindley paradox is the cleanest illustration of this sensitivity.

Properties

(Evidence Update) The Bayes factor transforms prior odds into posterior odds.

Applying Bayes’ rule to the two models gives $P(M_k\mid y)=p(y\mid M_k)\pi (M_k)/p(y)$, and dividing the two ratios cancels $p(y)$:
$\frac{P(M_1\mid y)}{P(M_0\mid y)}=\frac{p(y\mid M_1)}{p(y\mid M_0)}\cdot \frac{\pi (M_1)}{\pi (M_0)}={\mathrm{BF}}_{10}\cdot \frac{\pi (M_1)}{\pi (M_0)}.$
When prior odds are $1:1$ (flat over models), the posterior odds equal the Bayes factor. When prior odds are different, multiply. This factorization separates the data-driven part (${\mathrm{BF}}_{10}$, depending only on the likelihoods and parameter priors) from the model prior ($\pi (M_k)$, depending only on pre-data beliefs about which model is correct), which is the main conceptual advantage over a raw posterior-odds report.

(Built-in Occam Penalty) Integrating the likelihood over the prior automatically penalizes complex models.

The marginal likelihood $p(y\mid M)={\mathbb{E}}_{\theta \sim p(\cdot \mid M)}[p(y\mid \theta ,M)]$ averages the likelihood against the prior. A model whose parameter space is large but whose prior does not concentrate the mass near high-likelihood values will have small marginal likelihood despite possibly fitting well at its MLE. A simpler model with a tighter prior near the truth pays no such penalty. The Bayesian Information Criterion (BIC)
$\text{BIC}=-2\log p(y\mid {\hat{\theta }}_{\text{MLE}})+p\log n$
is the leading-order approximation of $-2\log p(y\mid M)$ under a Laplace approximation as $n\to \infty$; the $p\log n$ term is the explicit complexity penalty that emerges from integrating the Laplace Gaussian against the prior.

(Evidence for the Null) Bayes factors can support either model; frequentist $p$-values can only reject.

A small ${\mathrm{BF}}_{10}$ (say ${\mathrm{BF}}_{10}=0.05$) is direct evidence that $M_0$ explains the data $20\times$ better than $M_1$ — a meaningful affirmative claim for $M_0$. A large $p$-value, by contrast, is only “failure to reject” and gives no statement of confidence in the null. This asymmetry matters most in cases where null effects are scientifically interesting: bioequivalence trials, replication studies, and checks that a proposed predictor has no effect.

(Stopping-Rule Invariance) The Bayes factor inherits the likelihood principle; the data-collection plan is irrelevant.

Two experiments that produce the same observed likelihood yield the same marginal likelihood (up to a data-only constant that cancels in the ratio) and therefore the same Bayes factor. Stopping when a specific $p$-value threshold is hit, sampling until a fixed number of successes, or running until a budget is exhausted — all produce identical Bayesian evidence if the resulting likelihood is the same. Frequentist testing, by contrast, has different Type-I error rates under different stopping rules, which is the source of “$p$-hacking” controversies.

(Jeffreys-Lindley Paradox) Vague priors can make the Bayes factor favor the null arbitrarily strongly, while the $p$-value rejects.

Consider testing $M_0:\theta =0$ vs. $M_1:\theta \sim N(0,{\tau }^2)$ with a Gaussian likelihood $y\sim N(\theta ,1/n)$ for a sample mean. The marginal likelihoods are
$p(y\mid M_0)=\varphi (y;0,1/n),p(y\mid M_1)=\varphi (y;0,1/n+{\tau }^2),$
and
${\mathrm{BF}}_{10}=\frac{p(y\mid M_1)}{p(y\mid M_0)}=\frac{1}{\sqrt{1+n{\tau }^2}}\exp \left(\frac{n{y}^2}{2}\cdot \frac{n{\tau }^2}{1+n{\tau }^2}\right).$
For fixed $y$ with $|y|=c/\sqrt{n}$ (i.e., $z$-score $c$ fixed as $n$ grows), sending $\tau \to \infty$ makes ${\mathrm{BF}}_{10}\to 0$ even as the $z$-score stays fixed at a “significant” value like $c=2$. Concretely, a $p$-value of $0.05$ can coexist with ${\mathrm{BF}}_{10}$ as small as $0.1$ for wide enough priors — the data are “significant” under frequentist testing but the Bayes factor calls them evidence for the null. This is not a paradox in the logical sense; it is a predictable consequence of using a vague prior with a point-null, and it disappears when the prior is chosen proportionate to the scientific question.

(Not a Likelihood Ratio) $\mathrm{BF}$ and the frequentist likelihood ratio $\Lambda =p(y\mid {\hat{\theta }}_1)/p(y\mid {\hat{\theta }}_0)$ coincide only for two point hypotheses.

When both $M_0$ and $M_1$ specify single parameter values with no free parameters, the marginal likelihoods reduce to the likelihoods and ${\mathrm{BF}}_{10}=\Lambda$. As soon as either model has a prior (i.e., a free parameter that is integrated out), ${\mathrm{BF}}_{10}$ and $\Lambda$ diverge: $\Lambda$ uses the maximum-likelihood fit under each model, while ${\mathrm{BF}}_{10}$ uses the prior-averaged fit. The two generally have the same sign (both pick the better-fitting model) but different magnitudes, with $\Lambda$ typically favoring the more complex model because it uses its best fit while the Bayes factor penalizes it for diffuse priors.

(Computation) Marginal likelihoods are usually the hardest Bayesian quantity to compute.

For conjugate models, the marginal likelihood has a closed form (see conjugate families — e.g., the Beta-Binomial evidence is $B(\alpha +y,\beta +n-y)\cdot \left(\genfrac{}{}{0ex}{}{n}{y}\right)/B(\alpha ,\beta )$). For non-conjugate models, three standard approaches:

1. Laplace approximation: $\log p(y\mid M)\approx \ell (\hat{\theta })+\frac{d}{2}\log (2\pi )-\frac{1}{2}\log |-H(\hat{\theta })|$. Simple and fast; $O(n^{-1})$-accurate for smooth unimodal posteriors. See 3.9 Laplace Approximation.
2. Bridge sampling / thermodynamic integration: uses MCMC draws to produce unbiased marginal-likelihood estimates; expensive but asymptotically exact.
3. Reversible-jump MCMC: samples jointly across models in a single chain; gives posterior model probabilities directly without ever computing $p(y\mid M_k)$ separately.

Examples

Point-null binomial with beta prior

Test $M_0:\theta =0.5$ vs. $M_1:\theta \sim \text{Beta}(1,1)$ with data $y$ successes in $n$ trials.

Marginal under $M_0$: $p(y\mid M_0)=\left(\genfrac{}{}{0ex}{}{n}{y}\right)(0.5{)}^n$.

Marginal under $M_1$: $p(y\mid M_1)={\int }_0^1\left(\genfrac{}{}{0ex}{}{n}{y}\right){\theta }^y(1-\theta {)}^{n-y}d\theta =(\genfrac{}{}{0ex}{}{n}{y})B(y+1,n-y+1)=(\genfrac{}{}{0ex}{}{n}{y})\frac{y!(n-y)!}{(n+1)!}=\frac{1}{n+1}$.

Bayes factor: ${\mathrm{BF}}_{10}=\frac{1/(n+1)}{\left(\genfrac{}{}{0ex}{}{n}{y}\right)(0.5{)}^n}=\frac{2^n}{(n+1)\left(\genfrac{}{}{0ex}{}{n}{y}\right)}$.

For $n=10$, $y=7$: ${\mathrm{BF}}_{10}=\frac{1024}{11\cdot 120}\approx 0.776$, inconclusive. For $n=10$, $y=10$: ${\mathrm{BF}}_{10}=\frac{1024}{11\cdot 1}\approx 93$, very strong evidence against the fair coin. For $n=100$, $y=50$: ${\mathrm{BF}}_{10}\approx 0.126$, meaningful evidence for the fair coin — observing the expected frequency is positive support for $M_0$, not just “failure to reject.”

Laplace-approximated Bayes factor for logistic regression

Compare $M_1$: logistic regression with covariates $x_1,x_2$ vs. $M_0$: intercept-only, under flat priors on $\beta$. For each model, find the MAP ${\hat{\beta }}^{(k)}$ by Newton’s method and the observed information $-H^{(k)}({\hat{\beta }}^{(k)})$. The Laplace log-evidences are
$\log p(y\mid M_k)\approx {\ell }^{(k)}({\hat{\beta }}^{(k)})+\frac{p_k}{2}\log (2\pi )-\frac{1}{2}\log |-H^{(k)}({\hat{\beta }}^{(k)})|.$

The log-Bayes-factor is
$\log {\mathrm{BF}}_{10}\approx \underset{\text{fit improvement}}{\underbrace{{\ell }^{(1)}({\hat{\beta }}^{(1)})-{\ell }^{(0)}({\hat{\beta }}^{(0)})}}+\underset{\text{constant in }n}{\underbrace{\frac{p_1-p_0}{2}\log (2\pi )}}-\underset{\text{Occam penalty}}{\underbrace{\frac{1}{2}(\log |-H^{(1)}|-\log |-H^{(0)}|)}}.$

For large $n$, $\log |-H^{(k)}|\approx p_k\log n$ and the formula reduces to $\log {\mathrm{BF}}_{10}\approx ({\ell }^{(1)}-{\ell }^{(0)})-\frac{p_1-p_0}{2}\log n$, which is (up to a constant) the negative difference in BIC. Two added predictors buy you $\Delta \ell \approx \frac{3}{2}\log n$ worth of fit before the Bayes factor starts penalizing the larger model.

Jeffreys-Lindley paradox in action

Consider a single observation $y\sim N(\theta ,1/n)$ with $n=1000$. Observe $y=0.0632$, which gives a $z$-score $z=y\sqrt{n}=2.0$ and a two-sided $p$-value of $0.046$ — “significant at $\alpha =0.05$.”

Test $M_0:\theta =0$ vs. $M_1:\theta \sim N(0,{\tau }^2)$ with $\tau =10$ (a vague prior). Plugging into the formula:
${\mathrm{BF}}_{10}=\frac{1}{\sqrt{1+1000\cdot 100}}\exp \left(\frac{1000\cdot 0.0632^2}{2}\cdot \frac{100000}{100001}\right)=\frac{1}{316.2}\exp (1.997)\approx \frac{7.37}{316.2}\approx \mathrm{0.023.}$

So ${\mathrm{BF}}_{10}=0.023$, or ${\mathrm{BF}}_{01}\approx 43$: under this prior, the data are $43\times$ more likely under $M_0$ than $M_1$ — strong evidence for the null — even though the frequentist $p$-value rejects $M_0$. The mechanism is that $\tau =10$ treats $\theta =\pm 10$ as equally plausible a priori as $\theta \approx 0$, and observing $|\theta |\approx 0.063$ rules out most of $M_1$‘s prior mass. The “paradox” dissolves once the prior is chosen to represent scientifically plausible effect sizes (e.g., $\tau =0.1$ if effects of that magnitude are expected); the sensitivity to the prior is a genuine feature of Bayesian testing, not a mistake.

See Also

• 3.1 Bayesian Framework — Bayes’ rule applied at the model level gives the evidence-update formula.
• 3.7 Marginal Likelihood — how to compute $p(y\mid M)$ for conjugate and non-conjugate models.
• 3.9 Laplace Approximation — the standard shortcut for non-conjugate evidence; yields BIC in the large-$n$ limit.
• 3.2 Credible Intervals — parameter-level posterior summary; complementary to model-level testing.
• 1.4 Hypothesis Testing — frequentist counterpart.
• 1.4.1 Neyman-Pearson Testing — classical likelihood-ratio framework.
• 3. Bayesian Statistics — parent index.