2.2b.4 Multinomial Distribution

Multinomial Distribution

A discrete random vector $\mathbf{X}=(X_1,\dots ,X_k)$ follows the Multinomial distribution $\mathbf{X}\sim \text{Mult}(n,\mathbf{p})$ with $n\in \mathbb{N}$ trials and probability vector $\mathbf{p}=(p_1,\dots ,p_k)$ satisfying $p_i>0$ and ${\sum }_{i=1}^k{p}_i=1$, defined by the probability mass function
$p_{\mathbf{X}}(x_1,\dots ,x_k)=\frac{n!}{x_1!x_2!\cdots x_k!}{p}_1^{x_1}{p}_2^{x_2}\cdots p_k^{x_k}=\left(\genfrac{}{}{0ex}{}{n}{x_1,\dots ,x_k}\right){\prod }_{i=1}^k{p}_i^{x_i},$
where $x_i\in \{0,1,\dots ,n\}$ and ${\sum }_{i=1}^k{x}_i=n$. The multinomial coefficient $\left(\genfrac{}{}{0ex}{}{n}{x_1,\dots ,x_k}\right)=\frac{n!}{x_1!\cdots x_k!}$ counts the number of ways to partition $n$ objects into $k$ groups of sizes $x_1,\dots ,x_k$.

The Multinomial distribution models the counts of outcomes when $n$ independent trials are performed, each resulting in exactly one of $k\ge 2$ mutually exclusive categories. It is the natural extension of the Binomial distribution from two outcomes to $k$ outcomes.

Special cases:

• If $k=2$, the Multinomial reduces to the Binomial: $X_1\sim \text{Bin}(n,p_1)$ and $X_2=n-X_1$.

• If $n=1$, the Multinomial reduces to a single categorical trial (the multi-category analogue of Bernoulli).

Components Are Not Independent $n$ trials are independent, the components $X_1,\dots ,X_k$ of a Multinomial vector are negatively correlated because they are constrained to sum to $n$. For $i\ne j$, $\text{Cov}(X_i,X_j)=-n{p}_i{p}_j<0$: observing more outcomes in category $i$ necessarily leaves fewer for category $j$.

Although the

Intuition

The Multinomial distribution generalizes the Binomial from coin flips (two outcomes) to dice rolls ($k$ outcomes). Imagine rolling a (possibly weighted) $k$-sided die $n$ times and counting how many times each face appears. The result is a vector of counts $(X_1,\dots ,X_k)$ with $\sum X_i=n$, and this vector follows $\text{Mult}(n,\mathbf{p})$.

Because the counts must sum to $n$, specifying any $k-1$ of them determines the last. This constraint introduces negative dependence between components—if one category gets more than expected, the others must collectively get less. Despite this marginal dependence, each individual component $X_i$ is simply $\text{Bin}(n,p_i)$, and all the complexity lies in the joint structure.

Properties

(Marginals) Each component is Binomial: $X_i\sim \text{Bin}(n,p_i)$ for $i=1,\dots ,k$.

(Mean) $\mathbb{E}[X_i]=n{p}_i$.

(Variance) $\text{Var}(X_i)=n{p}_i(1-p_i)$.

(Covariance) For $i\ne j$, $\text{Cov}(X_i,X_j)=-n{p}_i{p}_j$.

This follows from the constraint $\sum X_i=n$: taking the variance of both sides gives ${\sum }_i\text{Var}(X_i)+2{\sum }_{i<j}\text{Cov}(X_i,X_j)=0$, which, combined with the marginal variances, yields the result.

(Correlation) For $i\ne j$, $\rho (X_i,X_j)=\frac{-p_i{p}_j}{\sqrt{p_i(1-p_i)p_j(1-p_j)}}=-\sqrt{\frac{p_i{p}_j}{(1-p_i)(1-p_j)}}$.

(Aggregation) If categories $i$ and $j$ are merged into a single category with probability $p_i+p_j$, the result is $\text{Mult}(n,{\mathbf{p}}^{\prime })$ with $k-1$ categories. Multinomial structure is preserved under grouping.

(Conditioning) Conditional on $X_1=x_1$, the remaining counts $(X_2,\dots ,X_k)$ follow $\text{Mult}\left(n-x_1,\frac{p_2}{1-p_1},\dots ,\frac{p_k}{1-p_1}\right)$.

(Sum of Independent Multinomials) If $\mathbf{X}\sim \text{Mult}(n,\mathbf{p})$ and $\mathbf{Y}\sim \text{Mult}(m,\mathbf{p})$ are independent, then $\mathbf{X}+\mathbf{Y}\sim \text{Mult}(n+m,\mathbf{p})$.

(Bayesian Conjugacy) The Dirichlet distribution is the conjugate prior for the Multinomial likelihood. If $\mathbf{\Theta }\sim \text{Dir}(\mathbf{\alpha })$ and $\mathbf{X}\mid \mathbf{\Theta }\sim \text{Mult}(n,\mathbf{\Theta })$, then $\mathbf{\Theta }\mid \mathbf{X}\sim \text{Dir}({\alpha }_1+x_1,\dots ,{\alpha }_k+x_k)$.

Examples

Genetic Offspring Ratios

In a genetics experiment, offspring of a particular cross fall into three phenotype categories with theoretical probabilities $\mathbf{p}=(9/16,3/16,4/16)$ under Mendelian inheritance. In a sample of $n=160$ offspring, the observed counts are $(x_1,x_2,x_3)=(86,35,39)$.

What are the expected counts, and how can goodness of fit be assessed? $\text{Mult}(160,\mathbf{p})$, the expected counts are $\mathbb{E}[X_1]=160\cdot \frac{9}{16}=90,\mathbb{E}[X_2]=160\cdot \frac{3}{16}=30,\mathbb{E}[X_3]=160\cdot \frac{4}{16}=40.$ The Pearson chi-square statistic is ${\chi }^2={\sum }_{i=1}^3\frac{(x_i-n{p}_i{)}^2}{n{p}_i}=\frac{(86-90{)}^2}{90}+\frac{(35-30{)}^2}{30}+\frac{(39-40{)}^2}{40}=\frac{16}{90}+\frac{25}{30}+\frac{1}{40}\approx \mathrm{1.04.}$ Under the null hypothesis, ${\chi }^2\stackrel{\text{approx}}{\sim }{\chi }_{k-1}^2={\chi }_2^2$. Since $1.04$ is well below the $95\%$ critical value of $5.99$, there is no evidence against the Mendelian model.

Under

Dice Fairness

A six-sided die is rolled $n=600$ times, yielding counts $(X_1,\dots ,X_6)$.

What is the covariance between $X_1$ (count of ones) and $X_6$ (count of sixes)? If the die is fair, $\mathbf{p}=(1/6,\dots ,1/6)$. By the covariance property, $\text{Cov}(X_1,X_6)=-n{p}_1{p}_6=-600\cdot \frac{1}{6}\cdot \frac{1}{6}=-\frac{600}{36}=-\frac{50}{3}\approx -\mathrm{16.67.}$ The negative covariance confirms the intuition: more ones in $600$ rolls means fewer sixes. The correlation is $\rho (X_1,X_6)=-\frac{1/36}{\sqrt{(5/36{)}^2}}=-\frac{1}{5}=-0.2$.