3.2. Gaussian Mixture Models

Gaussian Mixture Models

The Gaussian Mixture Model (GMM) is a probabilistic model that represents the distribution of data as a weighted combination of $K$ multivariate Gaussian components:
$p(\mathbf{x}\mid \pi ,{\mu }_1,{\Sigma }_1,\dots ,{\mu }_K,{\Sigma }_K)={\sum }_{k=1}^K{\pi }_k\mathcal{N}(\mathbf{x}\mid {\mu }_k,{\Sigma }_k),$
where $\pi \in {\mathbb{R}}^K$ is a vector, such that each ${\pi }_z=P(Z)$ is a prior on the $Z$-th cluster center (i.e., $\sum {\pi }_k=1$). Given a data point $x$, we can use total probability to compute the marginal $p(x_n)$, and Bayes’ rule to compute the probability that $x_n$ belongs to the $k$-th cluster:
$p(z=k\mid \mathbf{x},\mathbf{\pi },\{{\mathbf{\mu }}_k,{\mathbf{\Sigma }}_k\})=\frac{{\pi }_k\mathcal{N}(\mathbf{x}\mid {\mathbf{\mu }}_k,{\mathbf{\Sigma }}_k)}{{\sum }_{j=1}^K{\pi }_j\mathcal{N}(\mathbf{x}\mid {\mathbf{\mu }}_j,{\mathbf{\Sigma }}_j)}.$
These posterior probabilities $p(z=k\mid \mathbf{x})$ represent the soft assignments of data points to clusters, meaning that each point is assigned a probability of being in each cluster.

Because there is no closed-form solution for the log-likelihood $p(x\mid \pi )$, the Expectation-Minimization (EM) algorithm provides an iterative way to optimize this likelihood. To begin, initialize by making guesses about the distributions $\{{\pi }_k,{\mu }_k,{\Sigma }_k{\}}_{k=1}^K$. Then:

1. E-Step: Compute the soft-assignments (probabilities) of each point $x_n$ being in cluster $k$, ${\gamma }_{nk}=p(z_n=k\mid x_n)\in [0,1]$, and the expected log-likelihood objective function,

Q(\pi, { \mu_{k}, \Sigma_{k}}{k=1}^{K})&= \mathbb{E}{Z}[\ln p(X,Z \mid \theta)]\
&=\sum_{n=1}^{N}\sum_{k=1}^{K}\gamma_{nk}(\ln \pi_{k} + \ln\mathcal{N}(x_{n} \mid \mu_{k}, \Sigma_{k})).
\end{align*}$$

1. M-Step: Find the new parameters of $\arg \underset{\pi ,\mu ,\Sigma }{\max }Q(\pi ,\{{\mu }_k,{\Sigma }_k{\}}_{k=1}^K)$:

$N_k={\sum }_{n=1}^N{\gamma }_{nk}⟹{\pi }_k=\frac{N_k}{N},{\mu }_k=\frac{1}{N_k}{\sum }_{n=1}^N{\gamma }_{nk}{x}_n,{\Sigma }_k=\frac{1}{N_k}{\sum }_{n=1}^N{\gamma }_{nk}(x_n-{\mu }_k{)}^T(x_n-{\mu }_k).$