Beta-Binomial Model

Beta-Binomial Conjugacy

Let $\theta \in [0,1]$ be an unknown Bernoulli parameter (success probability). The Beta-Binomial conjugate pair is the prior-likelihood combination
$\theta \sim \text{Beta}(\alpha ,\beta ),y\mid \theta \sim \text{Binomial}(n,\theta ),$
which produces the closed-form posterior
$\theta \mid y\sim \text{Beta}(\alpha +y,\beta +n-y).$
The prior hyperparameters $(\alpha ,\beta )$ update by adding successes $y$ to $\alpha$ and failures $n-y$ to $\beta$. Equivalently, the prior is interpreted as $\alpha$ “pseudo-successes” and $\beta$ “pseudo-failures” observed before the data.

Posterior summaries:

• Posterior mean: $\mathbb{E}[\theta \mid y]=\frac{\alpha +y}{\alpha +\beta +n}$ — a weighted average of the prior mean $\alpha /(\alpha +\beta )$ and the MLE $y/n$, with weights proportional to prior and data sample sizes.
• Posterior mode (MAP): $\frac{\alpha +y-1}{\alpha +\beta +n-2}$ when $\alpha +y\ge 1$ and $\beta +n-y\ge 1$; equals the MLE under a uniform prior $\text{Beta}(1,1)$.
• Posterior variance: $\frac{(\alpha +y)(\beta +n-y)}{(\alpha +\beta +n{)}^2(\alpha +\beta +n+1)}$; shrinks like $O(1/n)$.

Common prior choices:

• $\text{Beta}(1,1)=\text{Uniform}(0,1)$: flat prior; posterior is $\text{Beta}(1+y,1+n-y)$ and the MAP equals the MLE.
• $\text{Beta}(0.5,0.5)$ (Jeffreys prior): noninformative prior based on the Fisher information, $p(\theta )\propto {\theta }^{-1/2}(1-\theta {)}^{-1/2}$; has a U-shape favoring extremes.
• $\text{Beta}(\alpha ,\beta )$ with small $\alpha +\beta$: weakly informative; centered at $\alpha /(\alpha +\beta )$ with “prior sample size” $\alpha +\beta$.

Posterior predictive for a new flip or new batch of flips:

• Single new flip: $P(\tilde{y}=1\mid y)=\mathbb{E}[\theta \mid y]=\frac{\alpha +y}{\alpha +\beta +n}$.
• $m$ new flips: $\tilde{y}\mid y\sim \text{BetaBin}(m,\alpha +y,\beta +n-y)$, a beta-binomial with variance strictly larger than the plug-in $\text{Binomial}(m,\hat{\theta })$.

The $k$-success form (Bernoulli likelihood) generalizes trivially: for $n$ i.i.d. Bernoulli observations $y_1,\dots ,y_n$ with $\sum y_i=y$, the posterior is identical to the single-binomial case.

Intuition

The Beta prior’s density ${\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}$ has exactly the same shape in $\theta$ as the binomial likelihood ${\theta }^y(1-\theta {)}^{n-y}$. Multiplying the two adds exponents: $\alpha -1+y$ and $\beta -1+n-y$. Renormalizing recognizes the result as another Beta density. This shape-matching is the entire mechanism — Beta is conjugate to Binomial because their densities live in the same two-parameter exponential family parameterized by powers of $\theta$ and $1-\theta$.

The update has a clean pseudo-data interpretation. A $\text{Beta}(\alpha ,\beta )$ prior is statistically indistinguishable from observing $\alpha$ prior successes and $\beta$ prior failures. Once real data arrive, the posterior counts are just $\alpha +y$ successes and $\beta +n-y$ failures total. The ratio $\alpha /(\alpha +\beta )$ is the prior mean; the sum $\alpha +\beta$ is the effective prior sample size. Small $\alpha +\beta$ means weak prior and fast washout; large $\alpha +\beta$ means strong prior that competes with the data.

Properties

(Posterior Mean as Shrinkage) The posterior mean is a convex combination of the prior mean and the MLE.

Writing ${\hat{\theta }}_{\text{MLE}}=y/n$ and ${\mu }_0=\alpha /(\alpha +\beta )$ for the prior mean,
$\mathbb{E}[\theta \mid y]=\frac{\alpha +y}{\alpha +\beta +n}=\underset{w_0}{\underbrace{\frac{\alpha +\beta }{\alpha +\beta +n}}}{\mu }_0+\underset{w_d}{\underbrace{\frac{n}{\alpha +\beta +n}}}{\hat{\theta }}_{\text{MLE}},$
with weights summing to $1$. When $n\ll \alpha +\beta$ the posterior is dominated by the prior; when $n\gg \alpha +\beta$ it is dominated by the MLE. The boundary $n\approx \alpha +\beta$ is the regime where both contribute comparably. This shrinkage structure is the reason Bayesian inference stabilizes estimates for small $n$: the MLE $y/n$ has variance $\theta (1-\theta )/n$ that blows up relative to the parameter’s range for $n$ of order $1$, while the posterior mean is automatically pulled toward the prior and never degenerates.

(Uniform Prior Recovers MLE) A $\text{Beta}(1,1)$ prior makes the MAP equal the MLE.

Plugging $\alpha =\beta =1$ into the MAP formula gives $\frac{y}{n}$, exactly the binomial MLE. This is a general pattern: under a flat prior, Bayesian MAP estimation coincides with maximum likelihood, because the prior does not shift the location of the mode. The posterior mean, however, differs from the MLE even under a flat prior — $(y+1)/(n+2)$ rather than $y/n$ — because the Beta density is asymmetric unless $\alpha =\beta$.

(Jeffreys Prior) The $\text{Beta}(1/2,1/2)$ prior is the Jeffreys prior for the Bernoulli likelihood and is invariant under reparameterization.

The Jeffreys prior is $p(\theta )\propto \sqrt{\mathcal{I}(\theta )}$ where $\mathcal{I}$ is the Fisher information; for the Bernoulli, $\mathcal{I}(\theta )=1/[\theta (1-\theta )]$, giving $p(\theta )\propto {\theta }^{-1/2}(1-\theta {)}^{-1/2}$, which is $\text{Beta}(1/2,1/2)$. Its defining feature is that inference under the Jeffreys prior is invariant to one-to-one reparameterizations of $\theta$: posterior inference on $\theta$ and on, say, $\eta =\log [\theta /(1-\theta )]$ yields the same “objective” results. The Jeffreys prior is U-shaped, concentrating mass at $0$ and $1$, which can be a feature (encodes the principle of indifference on a bounded scale) or a bug (injects nontrivial information into the tails) depending on the application.

(Predictive Distribution Is Beta-Binomial) The posterior predictive distribution for $m$ new flips is a beta-binomial, not a binomial.

Marginalizing $\theta$ out of $m$ new flips,
$P(\tilde{y}=k\mid y)={\int }_0^1\left(\genfrac{}{}{0ex}{}{m}{k}\right){\theta }^k(1-\theta {)}^{m-k}p(\theta \mid y)d\theta =(\genfrac{}{}{0ex}{}{m}{k})\frac{B(\alpha +y+k,\beta +n-y+m-k)}{B(\alpha +y,\beta +n-y)},$
which is the $\text{BetaBin}(m,\alpha +y,\beta +n-y)$ pmf. Its variance is
$\mathrm{Var}(\tilde{y}\mid y)=m\mathbb{E}[\theta \mid y](1-\mathbb{E}[\theta \mid y])\frac{\alpha +\beta +n+m}{\alpha +\beta +n+1},$
strictly larger than the plug-in binomial variance $m\mathbb{E}[\theta \mid y](1-\mathbb{E}[\theta \mid y])$ — the inflation factor exceeds $1$ and accounts for posterior uncertainty in $\theta$. The inflation vanishes as $n\to \infty$ and the posterior concentrates.

(Sequential Update) The posterior after observing $k$ flips is the prior for the $(k+1)$-th flip.

Beta-Binomial conjugacy is sequentially self-updating: if $\theta \sim \text{Beta}(\alpha ,\beta )$ a priori and we observe flips one at a time, after seeing $y_1,\dots ,y_k$ with total successes $s_k$, the posterior is $\text{Beta}(\alpha +s_k,\beta +k-s_k)$. Using this as the prior for flip $k+1$ and updating with the single new observation produces $\text{Beta}(\alpha +s_{k+1},\beta +k+1-s_{k+1})$, the same as updating with all $k+1$ flips at once. This is the concrete form of the sequential updating property from 3.1 Bayesian Framework for this conjugate pair.

(Laplace’s Rule of Succession) Under a uniform prior, the posterior probability that the next flip is a success is $(y+1)/(n+2)$.

Plugging $\alpha =\beta =1$ into the posterior mean gives $P(\tilde{y}=1\mid y)=(y+1)/(n+2)$, known as Laplace’s rule of succession. It is the oldest explicit Bayesian predictive formula (Laplace, 1774) and a classic example of the benefit of shrinkage: after seeing $n=0$ flips the predictive probability is $1/2$ (not undefined); after seeing $y=n$ successes the predictive is $(n+1)/(n+2)$ (not the degenerate $1$). The MLE’s pathologies at the boundary are gone.

Examples

Three new flips after a uniform prior

Place a $\text{Beta}(1,1)$ (uniform) prior on $\theta$ and observe $y=2$ heads in $n=3$ flips. The posterior is $\theta \mid y\sim \text{Beta}(3,2)$.

Posterior mean: $\mathbb{E}[\theta \mid y]=3/5=0.6$. Posterior mode (MAP): $(3-1)/(3+2-2)=2/3\approx 0.667$. The MAP equals the MLE $2/3$, as expected for the uniform prior.

$95\%$ equal-tail credible interval for $\theta$: using the $\text{Beta}(3,2)$ quantiles,
$[q_{0.025},q_{0.975}]\approx [0.194,0.932].$
The Wald confidence interval $\hat{\theta }\pm 1.96\sqrt{\hat{\theta }(1-\hat{\theta })/n}=0.667\pm 0.533=[0.13,1.20]$ is much wider and extends past $1$ — the Bayesian interval is automatically bounded inside $[0,1]$.

Predictive for $m=3$ new flips: $\tilde{y}\mid y\sim \text{BetaBin}(3,3,2)$. Evaluating the pmf,
$P(\tilde{y}=0)=\frac{1}{10},P(\tilde{y}=1)=\frac{3}{10},P(\tilde{y}=2)=\frac{3}{10},P(\tilde{y}=3)=\frac{3}{10}.$

The plug-in binomial $\text{Bin}(3,0.6)$ would give $P(\tilde{y}=3)=0.6^3=0.216$; the beta-binomial’s $0.3$ correctly accounts for posterior uncertainty, giving fatter tails.

Using a strongly informative prior

A manufacturer tests a new coin and claims the bias is very close to fair: $\theta \approx 0.5$ with $95\%$ prior certainty. Encode this as $\text{Beta}(100,100)$, which has mean $0.5$ and standard deviation $\sqrt{100\cdot 100/(200^2\cdot 201)}\approx 0.035$. A tester flips the coin $20$ times and observes $y=15$ heads.

MLE-only analysis: ${\hat{\theta }}_{\text{MLE}}=15/20=0.75$; the coin appears strongly biased.

Bayesian posterior: $\theta \mid y\sim \text{Beta}(115,105)$ with posterior mean $115/220\approx 0.523$, barely shifted from the prior mean $0.5$. Posterior variance is $\approx 0.00113$, standard deviation $\approx 0.034$ — essentially the prior’s width. The strong prior swamps the small sample.

$95\%$ credible interval: $[0.455,0.589]$. The point estimate of $0.75$ from the MLE is excluded by the Bayesian credible interval. To override the prior, the tester needs substantially more data — if the coin is truly biased at $0.75$, the posterior mean will cross $0.6$ only after roughly $n\approx 100$ observations with the same success rate. This illustrates both the strength (prior pulls toward believable values) and the risk (misspecified prior suppresses real signal) of Bayesian inference with informative priors.

Sequential updating

Start with $\text{Beta}(2,2)$ and observe three flips in sequence: T, H, T.

After flip 1 (T): posterior is $\text{Beta}(2,3)$, mean $2/5=0.4$.
After flip 2 (H): posterior is $\text{Beta}(3,3)$, mean $3/6=0.5$.
After flip 3 (T): posterior is $\text{Beta}(3,4)$, mean $3/7\approx 0.429$.

Updating with all three flips at once: prior $\text{Beta}(2,2)$, likelihood $\text{Bin}(3,\theta )$ with $y=1$. Posterior: $\text{Beta}(2+1,2+2)=\text{Beta}(3,4)$. Identical to the sequential result, as the sequential-updating property guarantees. Every batch of data can be absorbed in any order or granularity; the final posterior depends only on the total successes and total failures, which is the sufficient statistic for the Bernoulli model.

See Also

• 3.5 Conjugate Families — parent index.
• 3.1 Bayesian Framework — prior, likelihood, posterior, posterior predictive; this note is the canonical worked example.
• 3.2 Credible Intervals — the Beta posterior’s credible intervals are the standard worked example.
• 3.5.4 Dirichlet-Multinomial Model — multivariate generalization: Beta is $\text{Dir}$ in dimension $1$.
• 2.2b.3 Binomial Distribution — the underlying likelihood.
• 1.2.1 Maximum Likelihood Estimation — frequentist counterpart; the MLE is the MAP under $\text{Beta}(1,1)$.
• 1.5.2 Bayesian Bootstrap — a Dirichlet-weighted analog that generalizes beyond Bernoulli data.