Dirichlet-Multinomial Model

Dirichlet-Multinomial Conjugacy

Let $\mathbf{\theta }=({\theta }_1,\dots ,{\theta }_K)$ be an unknown probability vector on a $K$-category simplex (${\theta }_k\ge 0$, ${\sum }_k{\theta }_k=1$). The Dirichlet-Multinomial conjugate pair is
$\mathbf{\theta }\sim \text{Dirichlet}(\mathbf{\alpha }),\mathbf{y}\mid \mathbf{\theta }\sim \text{Multinomial}(n,\mathbf{\theta }),$
where $\mathbf{\alpha }=({\alpha }_1,\dots ,{\alpha }_K)$ with ${\alpha }_k>0$ and $\mathbf{y}=(y_1,\dots ,y_K)$ with ${\sum }_k{y}_k=n$. The posterior is
$\mathbf{\theta }\mid \mathbf{y}\sim \text{Dirichlet}(\mathbf{\alpha }+\mathbf{y})=\text{Dirichlet}({\alpha }_1+y_1,\dots ,{\alpha }_K+y_K).$
The prior hyperparameters update componentwise: add the observed count in each category to the prior’s concentration for that category. The prior is interpretable as ${\alpha }_k$ “pseudo-observations” in category $k$, for a total prior sample size ${\alpha }_0:={\sum }_k{\alpha }_k$.

Posterior summaries: writing ${\alpha }_0={\sum }_k{\alpha }_k$,

• Posterior mean of ${\theta }_k$: $\mathbb{E}[{\theta }_k\mid \mathbf{y}]=\frac{{\alpha }_k+y_k}{{\alpha }_0+n}$ — a weighted average of the prior mean ${\alpha }_k/{\alpha }_0$ and the MLE $y_k/n$.
• Posterior mode (MAP) of ${\theta }_k$: $\frac{{\alpha }_k+y_k-1}{{\alpha }_0+n-K}$ when ${\alpha }_k+y_k\ge 1$ for all $k$.
• Posterior marginal of a single ${\theta }_k$: $\text{Beta}({\alpha }_k+y_k,{\alpha }_0+n-{\alpha }_k-y_k)$ — reduces to Beta-Binomial when $K=2$.
• Posterior covariance: $\text{Cov}({\theta }_j,{\theta }_k\mid \mathbf{y})=-\frac{({\alpha }_j+y_j)({\alpha }_k+y_k)}{({\alpha }_0+n{)}^2({\alpha }_0+n+1)}$ for $j\ne k$. Components are always negatively correlated because they share the unit budget.

Common prior choices:

• $\text{Dirichlet}(1,\dots ,1)=\text{Uniform}$ on the simplex: flat prior; posterior mode equals the MLE $y_k/n$ (with a small shift).
• $\text{Dirichlet}(1/K,\dots ,1/K)$ (Perks-Jeffreys-like, sparse): small ${\alpha }_0=1$ favors mass concentrating on a few categories.
• $\text{Dirichlet}(1/2,\dots ,1/2)$ (Jeffreys prior): reparametrization-invariant; gentle shrinkage toward the simplex center.

Posterior predictive for a new observation $\tilde{y}$ (single draw from one of $K$ categories):
$P(\tilde{y}=k\mid \mathbf{y})=\mathbb{E}[{\theta }_k\mid \mathbf{y}]=\frac{{\alpha }_k+y_k}{{\alpha }_0+n}.$
For $m$ new i.i.d. draws $\tilde{\mathbf{y}}$, the predictive is Dirichlet-Multinomial (also called Pólya / multivariate hypergeometric): $\tilde{\mathbf{y}}\mid \mathbf{y}\sim \text{DirMult}(m,\mathbf{\alpha }+\mathbf{y})$, with variance strictly larger than the plug-in Multinomial.

Prerequisites

• 3.1 Bayesian Framework — prior, likelihood, posterior, Bayes’ rule.
• 3.5 Conjugate Families — the general conjugacy pattern this note instantiates.
• 3.5.1 Beta-Binomial Model — two-category special case; all intuitions carry over.
• 2.2b.4 Multinomial Distribution — the likelihood.
• 2.2c.10 Dirichlet Distribution — the prior family.

Intuition

The Dirichlet distribution is the multivariate generalization of the Beta: instead of one probability on $[0,1]$, it models a full probability vector on the simplex. Its density is proportional to ${\prod }_k{\theta }_k^{{\alpha }_k-1}$, exactly the kernel of the Multinomial likelihood viewed as a function of $\mathbf{\theta }$. Multiplying prior by likelihood adds exponents componentwise, and the result is again Dirichlet with concentrations incremented by the observed category counts.

The shape parameters ${\alpha }_k$ play two roles at once. First, their ratios ${\alpha }_k/{\alpha }_0$ set the prior mean (expected probability for each category). Second, their total ${\alpha }_0={\sum }_k{\alpha }_k$ sets the prior’s concentration — a large ${\alpha }_0$ produces a prior tightly focused around its mean vector, and a small ${\alpha }_0$ produces a prior that is dispersed across the simplex. This two-parameter control is why the Dirichlet is the standard prior over discrete distributions: it separates “what I expect” from “how confident I am.”

The negative posterior correlation between components is an unavoidable structural fact. Evidence that category $j$ has more probability mass than thought necessarily implies that some other category(s) have less, because the components must sum to one. No conjugate prior on the simplex could avoid this; it is a property of simplex geometry, not of Dirichlet specifically.

Properties

(Reduces to Beta-Binomial When $K=2$) The two-category Dirichlet-Multinomial is the Beta-Binomial model.

With $K=2$, $\mathbf{\alpha }=(\alpha ,\beta )$, and $\mathbf{y}=(y,n-y)$, the Dirichlet prior is $\text{Beta}(\alpha ,\beta )$ and the Multinomial likelihood is $\text{Binomial}(n,{\theta }_1)$. The posterior $\text{Dirichlet}(\alpha +y,\beta +n-y)$ is exactly the Beta-Binomial posterior. All Beta-Binomial results lift to the multicategory case.

(Posterior Marginal of Each ${\theta }_k$ is Beta) Marginalizing over the other $K-1$ components leaves a Beta distribution.

${\theta }_k\mid \mathbf{y}\sim \text{Beta}({\alpha }_k+y_k,{\alpha }_0+n-{\alpha }_k-y_k).$
This follows from the aggregation property of the Dirichlet: merging all categories other than $k$ into one aggregate category collapses the Dirichlet to a Beta on $({\theta }_k,1-{\theta }_k)$. Credible intervals for a single category probability reuse the Beta machinery from the 3.5.1 Beta-Binomial Model.

(Aggregation and Subcomposition) Merging categories keeps the posterior Dirichlet; conditioning on a subset is also Dirichlet.

If categories $\{1,\dots ,K\}$ are partitioned into groups $G_1,\dots ,G_m$ with ${\tilde{\theta }}_j={\sum }_{k\in G_j}{\theta }_k$ and ${\tilde{y}}_j={\sum }_{k\in G_j}{y}_k$, then $({\tilde{\theta }}_1,\dots ,{\tilde{\theta }}_m)\mid \mathbf{y}\sim \text{Dirichlet}({\tilde{\alpha }}_1+{\tilde{y}}_1,\dots ,{\tilde{\alpha }}_m+{\tilde{y}}_m)$. Conditional on the total probability in a subset, the within-subset probabilities (renormalized) are independently Dirichlet distributed. This is why Dirichlet is the natural model for hierarchical categorical data.

(Posterior Mean as Simplex Shrinkage) Posterior mean lies between the prior mean vector and the MLE, weighted by effective sample sizes.

$\mathbb{E}[\mathbf{\theta }\mid \mathbf{y}]=w\frac{\mathbf{\alpha }}{{\alpha }_0}+(1-w)\frac{\mathbf{y}}{n},w=\frac{{\alpha }_0}{{\alpha }_0+n}.$
A Dirichlet prior with total mass ${\alpha }_0$ acts like ${\alpha }_0$ prior observations distributed across the $K$ categories. A category with zero observed count still receives nonzero posterior mass ${\alpha }_k/({\alpha }_0+n)$, a form of Laplace smoothing that prevents posterior probabilities from collapsing to zero on unseen categories. This is widely used in natural-language models under the name “additive smoothing.”

(Sequential Updating) Componentwise accumulation of category counts yields the same posterior as batch updating.

After seeing ${\mathbf{y}}^{(1)}$ of total size $n_1$ and then ${\mathbf{y}}^{(2)}$ of size $n_2$, the posterior is $\text{Dirichlet}(\mathbf{\alpha }+{\mathbf{y}}^{(1)}+{\mathbf{y}}^{(2)})$ — associative in the count vector. See 3.1 Bayesian Framework.

(Posterior Predictive is Dirichlet-Multinomial) Predictive for $m$ new draws has extra variance compared to the plug-in Multinomial.

$\tilde{\mathbf{y}}\mid \mathbf{y}\sim \text{DirMult}(m,\mathbf{\alpha }+\mathbf{y}).$
The marginal for each component is Beta-Binomial, and the variance of ${\tilde{y}}_k$ is inflated above $m{p}_k(1-p_k)$ (the Multinomial marginal variance) by a factor $({\alpha }_0+n+m)/({\alpha }_0+n+1)$ that approaches 1 only as the posterior concentrates. The Dirichlet-Multinomial is the multicategory analog of the Beta-Binomial predictive distribution.

(Negative Covariance is Structural) $\text{Cov}({\theta }_j,{\theta }_k\mid \mathbf{y})<0$ for all $j\ne k$.

Because ${\sum }_k{\theta }_k=1$ is a hard constraint, any Bayesian model on the simplex must assign negative posterior covariance between distinct components. The exact formula is
$\text{Cov}({\theta }_j,{\theta }_k\mid \mathbf{y})=-\frac{({\alpha }_j+y_j)({\alpha }_k+y_k)}{({\alpha }_0+n{)}^2({\alpha }_0+n+1)}.$
If a model’s inferential goal depends on independent category probabilities, this is a sign that the underlying problem is miscast — the simplex constraint is real.

(Dirichlet Multinomial as a Count-Data Alternative) Marginalizing $\mathbf{\theta }$ away yields a Dirichlet-Multinomial likelihood.

For applications where the parameter is a nuisance, integrating out $\mathbf{\theta }$ gives the marginal distribution of the counts, $p(\mathbf{y}\mid \mathbf{\alpha },n)$, known as the Dirichlet-Multinomial or Pólya distribution. This serves as a more flexible likelihood than Multinomial itself in overdispersed count-category data, and is the basis of hyperparameter tuning via empirical Bayes.

Examples

Example 1: Survey on a Three-Option Question

A survey asks $n=120$ respondents to choose among three options A, B, C. A weakly informative uniform prior is $\mathbf{\alpha }=(1,1,1)$ (with ${\alpha }_0=3$). Observed counts are $\mathbf{y}=(55,40,25)$.

Posterior: $\mathbf{\theta }\mid \mathbf{y}\sim \text{Dirichlet}(56,41,26)$ with total ${\alpha }_0+n=123$.
Posterior means: $(56/123,41/123,26/123)=(0.455,0.333,0.211)$ — essentially the sample proportions because the prior is very weak.
Marginal credible interval for ${\theta }_A$: ${\theta }_A\mid \mathbf{y}\sim \text{Beta}(56,67)$, 95% equal-tail CI $\approx [0.367,0.545]$.
Posterior predictive for one new respondent: $P(\tilde{y}=A)=56/123=0.455$, $P(\tilde{y}=B)=0.333$, $P(\tilde{y}=C)=0.211$ — the predictive plug-in uses the posterior mean, not the MLE.

Example 2: Laplace Smoothing in a Language Model

Over a vocabulary of $K=10,000$ words, a corpus of $n=10^6$ words is counted. Many words appear zero times. An MLE would assign probability zero to these words — disastrous for computing the likelihood of any future text containing one of them.

Using a $\text{Dirichlet}(\alpha ,\dots ,\alpha )$ prior with $\alpha =1$ (equivalent to adding one pseudo-count per word):
${\hat{\theta }}_k=\frac{y_k+1}{n+K}=\frac{y_k+1}{10^6+10^4}\approx \frac{y_k+1}{1.01\times 10^6}.$
For an unseen word ($y_k=0$), the posterior mean is $1/(n+K)\approx 10^{-6}$ — nonzero. For a common word appearing $y_k=1000$ times, $(1001)/(1.01\times 10^6)\approx 9.91\times 10^{-4}$, only slightly pulled toward uniform from the raw rate $10^{-3}$. This is add-one (Laplace) smoothing, a standard regularization in $n$-gram language models derived directly from the Dirichlet-Multinomial posterior mean.

Example 3: A/B/C Test with Sparse Data

A trial of three website variants records no conversions for variant C out of 20 visitors: $\mathbf{y}=(4,6,0)$. Naively, the MLE says variant C has zero conversion rate. Using a uniform Dirichlet prior $\mathbf{\alpha }=(1,1,1)$:

Posterior: $\text{Dirichlet}(5,7,1)$.
Posterior means: $(5/13,7/13,1/13)=(0.385,0.538,0.077)$.
Credible interval for ${\theta }_C$ (Beta marginal $\text{Beta}(1,12)$): 95% equal-tail CI $\approx [0.0021,0.2647]$ — wide, reflecting genuine uncertainty.

A frequentist Wald confidence interval for ${\theta }_C$ would be undefined (standard error zero with ${\hat{\theta }}_C=0$) or degenerate; the Bayesian analysis gives a usable interval directly. Running variant C further to resolve the uncertainty is the correct next step.

Derivation

Goal: derive the $\text{Dirichlet}(\mathbf{\alpha }+\mathbf{y})$ posterior.

The Dirichlet prior is
$p(\mathbf{\theta })=\frac{1}{B(\mathbf{\alpha })}{\prod }_{k=1}^K{\theta }_k^{{\alpha }_k-1},B(\mathbf{\alpha })=\frac{{\prod }_k\Gamma ({\alpha }_k)}{\Gamma ({\alpha }_0)},$
supported on the simplex. The Multinomial likelihood for $\mathbf{y}$ with ${\sum }_k{y}_k=n$ is
$p(\mathbf{y}\mid \mathbf{\theta })=\left(\genfrac{}{}{0ex}{}{n}{y_1,\dots ,y_K}\right){\prod }_k{\theta }_k^{y_k}\propto {\prod }_k{\theta }_k^{y_k}.$
Multiplying and dropping $\mathbf{\theta }$-free constants:
$p(\mathbf{\theta }\mid \mathbf{y})\propto {\prod }_k{\theta }_k^{{\alpha }_k-1}\cdot {\prod }_k{\theta }_k^{y_k}={\prod }_k{\theta }_k^{({\alpha }_k+y_k)-1},$
the kernel of $\text{Dirichlet}(\mathbf{\alpha }+\mathbf{y})$, with normalizing constant $1/B(\mathbf{\alpha }+\mathbf{y})$.

Posterior predictive: integrating out $\mathbf{\theta }$ for a single new category draw,
$P(\tilde{y}=k\mid \mathbf{y})=\int {\theta }_{k}p(\mathbf{\theta }\mid \mathbf{y})d\mathbf{\theta }=\mathbb{E}[{\theta }_k\mid \mathbf{y}]=\frac{{\alpha }_k+y_k}{{\alpha }_0+n}.$
For $m$ new draws $\tilde{\mathbf{y}}$, the integral of $\text{Multinomial}(m,\mathbf{\theta })$ against the Dirichlet posterior yields the closed-form Dirichlet-Multinomial $\text{DirMult}(m,\mathbf{\alpha }+\mathbf{y})$.

See Also

• 3.5 Conjugate Families — parent index.
• 3.5.1 Beta-Binomial Model — two-category special case.
• 3.5.2 Normal-Normal Model — continuous analog on the real line.
• 3.5.3 Gamma-Poisson Model — count-data analog on the positive reals.
• 3.6 Hierarchical Models — Dirichlet hyperprior over topic or cluster proportions.
• 3.7 Marginal Likelihood — Dirichlet-Multinomial evidence enables empirical-Bayes tuning of $\mathbf{\alpha }$.
• 2.2b.4 Multinomial Distribution — the likelihood.
• 2.2c.10 Dirichlet Distribution — the prior.