2.2c.10 Dirichlet Distribution

Dirichlet Distribution

A continuous random vector $\mathbf{\Theta }=({\Theta }_1,\dots ,{\Theta }_k)$ on the $(k-1)$-simplex ${\Delta }_{k-1}=\{\mathbf{\theta }\in {\mathbb{R}}^k:{\theta }_i\ge 0,{\sum }_{i=1}^k{\theta }_i=1\}$ follows the Dirichlet distribution $\mathbf{\Theta }\sim \text{Dir}({\alpha }_1,\dots ,{\alpha }_k)$ with concentration parameters ${\alpha }_i>0$, defined by the probability density function
$f(\mathbf{\theta })=\frac{1}{B(\mathbf{\alpha })}{\prod }_{i=1}^k{\theta }_i^{{\alpha }_i-1},\mathbf{\theta }\in {\Delta }_{k-1},$
where $B(\mathbf{\alpha })=\frac{{\prod }_{i=1}^k\Gamma ({\alpha }_i)}{\Gamma \left({\sum }_{i=1}^k{\alpha }_i\right)}$ is the multivariate Beta function, generalizing the two-parameter Beta function $B(\alpha ,\beta )$.

The Dirichlet distribution is the natural extension of the Beta distribution from a single proportion to a vector of proportions that must sum to one. It models random probability vectors—any collection of non-negative quantities that partition a whole.

Special cases:

• If $k=2$, the Dirichlet reduces to the Beta: ${\Theta }_1\sim \text{Beta}({\alpha }_1,{\alpha }_2)$ and ${\Theta }_2=1-{\Theta }_1$.
• If ${\alpha }_1={\alpha }_2=\cdots ={\alpha }_k=1$, then $\mathbf{\Theta }\sim \text{Uniform}({\Delta }_{k-1})$—the uniform distribution on the simplex.
• If all ${\alpha }_i=\alpha$ (the symmetric Dirichlet), the density is invariant to permutations of the components.

The concentration sum ${\alpha }_0={\sum }_{i=1}^k{\alpha }_i$ controls how tightly the distribution concentrates around its mean. Large ${\alpha }_0$ yields vectors close to $({\alpha }_1/{\alpha }_0,\dots ,{\alpha }_k/{\alpha }_0)$; small ${\alpha }_0$ pushes mass toward the corners of the simplex (sparse vectors where one component dominates).

Bayesian Interpretation conjugate prior for the Multinomial likelihood. If $\mathbf{\Theta }\sim \text{Dir}(\mathbf{\alpha })$ and $\mathbf{X}\mid \mathbf{\Theta }\sim \text{Mult}(n,\mathbf{\Theta })$, then the posterior is $\mathbf{\Theta }\mid \mathbf{X}\sim \text{Dir}({\alpha }_1+x_1,\dots ,{\alpha }_k+x_k).$ The parameters ${\alpha }_i$ act as pseudo-counts: ${\alpha }_i-1$ prior observations of category $i$. This generalizes Beta–Binomial conjugacy to the multivariate setting.

The Dirichlet is the

Intuition

The Dirichlet distribution is best understood as a distribution over probability vectors. Just as the Beta distribution models uncertainty about a single probability $\theta \in [0,1]$, the Dirichlet models uncertainty about a vector of probabilities $({\theta }_1,\dots ,{\theta }_k)$ constrained to lie on the simplex.

Imagine you are modeling a $k$-sided die whose face probabilities are unknown. Before rolling, your prior belief about $\mathbf{\theta }=({\theta }_1,\dots ,{\theta }_k)$ is $\text{Dir}(\mathbf{\alpha })$. Each ${\alpha }_i$ represents how strongly you believe face $i$ will appear—larger ${\alpha }_i$ pulls more probability toward face $i$. After observing $x_i$ occurrences of face $i$ in $n$ rolls, the posterior is $\text{Dir}({\alpha }_1+x_1,\dots ,{\alpha }_k+x_k)$, just as in the Beta-Binomial case but for all $k$ categories simultaneously.

The concentration sum ${\alpha }_0$ controls confidence: a symmetric $\text{Dir}(100,100,100)$ is tightly concentrated near $(1/3,1/3,1/3)$, expressing strong belief in a fair die, while $\text{Dir}(0.1,0.1,0.1)$ concentrates near the corners, expressing a belief that one face dominates.

Properties

(Mean) $\mathbb{E}[{\Theta }_i]=\frac{{\alpha }_i}{{\alpha }_0}$, where ${\alpha }_0={\sum }_{j=1}^k{\alpha }_j$.

(Variance) $\text{Var}({\Theta }_i)=\frac{{\alpha }_i({\alpha }_0-{\alpha }_i)}{{\alpha }_0^2({\alpha }_0+1)}$.

As ${\alpha }_0\to \infty$, the variance shrinks like $1/{\alpha }_0$, concentrating the distribution around its mean.

(Covariance) For $i\ne j$, $\text{Cov}({\Theta }_i,{\Theta }_j)=\frac{-{\alpha }_i{\alpha }_j}{{\alpha }_0^2({\alpha }_0+1)}$.

The components are negatively correlated, mirroring the simplex constraint $\sum {\Theta }_i=1$: if one component increases, the others must collectively decrease.

(Marginals) Each component is Beta: ${\Theta }_i\sim \text{Beta}({\alpha }_i,{\alpha }_0-{\alpha }_i)$.

(Aggregation) If components $i$ and $j$ are merged, ${\Theta }_i+{\Theta }_j\sim \text{Beta}({\alpha }_i+{\alpha }_j,{\alpha }_0-{\alpha }_i-{\alpha }_j)$. More generally, any partition of $\{1,\dots ,k\}$ into $m$ groups produces a $\text{Dir}$ with $m$ components whose parameters are the group sums.

(Gamma Construction) If $Y_i\sim \text{Gamma}({\alpha }_i,1)$ are independent, then

$\left(\frac{Y_1}{{\sum }_{j=1}^k{Y}_j},\dots ,\frac{Y_k}{{\sum }_{j=1}^k{Y}_j}\right)\sim \text{Dir}({\alpha }_1,\dots ,{\alpha }_k).$
This is the standard constructive definition of the Dirichlet, generalizing the Beta’s Gamma construction. It gives a concrete sampling algorithm: draw $k$ independent Gamma random variables and normalize.

(Neutrality) For any subset $S\subset \{1,\dots ,k\}$, the sum ${\sum }_{i\in S}{\Theta }_i$ is independent of the ratios ${\Theta }_i/{\sum }_{j\in S}{\Theta }_j$ for $i\in S$. This “neutral” property means the total share allocated to a group is independent of how that share is distributed within the group.

(Posterior Predictive) Under a $\text{Dir}(\mathbf{\alpha })$ prior and Multinomial likelihood, the posterior predictive probability of the next observation falling in category $i$ after seeing counts $\mathbf{x}$ is $\frac{{\alpha }_i+x_i}{{\alpha }_0+n}$, the Laplace smoothing estimator.

Examples

Document Topic Modeling

In a topic model, each document has a topic distribution $\mathbf{\theta }=({\theta }_1,{\theta }_2,{\theta }_3)$ drawn from $\text{Dir}(0.5,0.5,0.5)$. A document contains $n=100$ words, with each word assigned to a topic according to $\text{Mult}(100,\mathbf{\theta })$. Suppose the observed topic counts are $(x_1,x_2,x_3)=(60,30,10)$.

What is the posterior distribution over topic proportions? $\mathbf{\theta }\mid \mathbf{x}\sim \text{Dir}(0.5+60,0.5+30,0.5+10)=\text{Dir}(60.5,30.5,10.5).$ The posterior mean is $\left(\frac{60.5}{101.5},\frac{30.5}{101.5},\frac{10.5}{101.5}\right)\approx (0.596,0.300,0.103)$, close to the observed proportions but slightly smoothed toward equality by the symmetric prior. The posterior is concentrated (large ${\alpha }_0=101.5$), reflecting the substantial sample size.

By Dirichlet-Multinomial conjugacy,

Simplex Uniformity

Let $\mathbf{\Theta }\sim \text{Dir}(1,1,1)$—the uniform distribution on the $2$-simplex $\{({\theta }_1,{\theta }_2,{\theta }_3):{\theta }_i\ge 0,\sum {\theta }_i=1\}$.

What is the marginal distribution and expected value of ${\Theta }_1$? By the marginals property, ${\Theta }_1\sim \text{Beta}(1,2)$, which has density $f({\theta }_1)=2(1-{\theta }_1)$ on $[0,1]$. Its expected value is $\mathbb{E}[{\Theta }_1]=\frac{1}{1+2}=\frac{1}{3}$. Although the distribution on the simplex is uniform, each marginal is not uniform on $[0,1]$—it is skewed toward $0$ because the constraint $\sum {\theta }_i=1$ limits how large any single component can be.