2. Mutual Information

Mutual Information

The mutual information between two random variables $X$ and $Y$ is a non-negative measure of the reduction in entropy of one variable given knowledge of the other. For discrete variables, it is defined as

I(X;Y)&=\mathbb{E}\left( \log_{2}\frac{p_{X,Y}(x,y)}{p_{X}(x)p_{Y}(y)} \right) \
&= \sum_{x,y \in \mathcal{X} \times \mathcal{Y}}p_{X,Y}(x,y)\log_{2}\frac{p_{X,Y}(x,y)}{p_{X}(x)p_{Y}(y)} \
&=H(X)-H(X \mid Y) \
&= H(X)+H(Y)-H(X,Y)
\end{align}$$

Independence $I(X;Y)=0$.

Note that for independent variables,

Rationale

We begin by rationalizing about the definition of mutual information. Per the definition, mutual information quantifies the reduction in entropy given two variables $X$ and $Y$, compared to just $X$ or $Y$. It is only logical that $I(X;Y)=0$ if $X$ and $Y$ are independent; one will gain no information by observing $X$ (resp. $Y$) given $Y$ (resp. $X$). However, we must reason about the key equation here: $p_{X\mid Y}(x\mid y)=\frac{p_{X,Y}(x,y)}{p_X(x)p_Y(y)}.$

Properties

(Data )