6.2 Multivariate Gaussian Distribution

Jointly Gaussian Random Variables

Joint Gaussians are random variables whose joint behavior is completely characterized by a mean vector and covariance matrix. Let $X={\left[\begin{array}{ccc}{X}_1& \dots & X_n\end{array}\right]}^T$ be a random vector, and let $Z\in {\mathbb{R}}^{\ell }$ be the standard normal vector (i.e., $Z_i\sim \mathcal{N}(0,1)$ for $i=1,\dots ,\ell$ i.i.d.). Then, $X_1,\dots ,X_n$ are jointly Gaussian if and only if:

1. There exist some mean $\mu =\mathbb{E}[X]\in {\mathbb{R}}^n$ and real matrix $A\in {\mathbb{R}}^{n\times \ell }$, s. t. $X\sim \mathcal{N}(\mu ,A{A}^T)$, and
   $X=AZ+\mu .$
2. All elements of the affine hull of $X$ (i.e., all affine combinations $u\in \mathrm{aff}(X_1,\dots ,X_n)$) follow a normal distribution, and therefore $\mathrm{aff}(X_1,\dots ,X_n)$ is a Gaussian subspace.

The PDF of joint Gaussians is defined by a positive definite covariance matrix $\Sigma =A{A}^T\in {\mathbb{S}}_{++}^n$ with entries ${\Sigma }_{i,j}=\mathrm{Cov}(X_i,X_j)$, such that
$f_X(x_1,\dots ,x_n)=\frac{1}{\sqrt{(2\pi {)}^n\det (\Sigma )}}\exp \left(-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )\right).$

Properties

(Independence) Joint Gaussians $X_1,\dots ,X_n$ are indep. iff $\Sigma =\mathrm{diag}({\sigma }_1^2,\dots ,{\sigma }_n^2)$.

Without loss of generality, we will consider the case of two jointly Gaussian random variables. Let $X_1,X_2$ be uncorrelated joint Gaussians. Therefore,
$\Sigma =\left[\begin{array}{cc}{\sigma }_1^2& 0\\ 0& {\sigma }_2^2\end{array}\right].$
Substituting ${\Sigma }^{-1}$ into the joint PDF, we have

f_{X}(x_{1},x_{2})&=\frac{1}{\sqrt{ (2\pi)^2 }\sigma_{1}\sigma_{2}}\exp\left( -\frac{1}{2}\left[ \frac{(x_{1}-\mu_{1})^2}{\sigma_{1}^2}+\frac{(x_{2}-\mu_{2})^2}{\sigma_{2}^2} \right] \right) \\ &= \frac{1}{\sqrt{ 2\pi }\sigma_{1}}\exp\left( -\frac{1}{2} \frac{(x_{1}-\mu_{1})^2}{\sigma_{1}^2} \right) \cdot \frac{1}{\sqrt{ 2\pi }\sigma_{2}}\exp\left( -\frac{1}{2} \frac{(x_{2}-\mu_{2})^2}{\sigma_{2}^2} \right) \\ &= f_{X_{1}}(x_{1}) \cdot f_{X_{2}}(x_{2}). \end{align}$$ #### (*Linear Combinations*) Any $U$ s.t. $U_{i} \in \operatorname{aff}(X_{1}, \ldots, X_{n})$ is jointly Gaussian. Let $U$ be a random vector, such that each $U_{i}=\gamma_{i}^TX+\beta_{i}$ is a linear combination of $X=[X_{1}, \ldots, X_{n}]^T$ for some vector of coefficients $\gamma \in \mathbb{R}^n$. We also know that $$X_{i}=a_{i}^TZ+\mu_{i},$$ and therefore, $$\begin{align} U_{i}&=\gamma_{1}(a_{1}^TZ+\mu_{1})+\gamma_{n}(a_{n}^TZ+\mu_{n})+\beta_{1} \\ &= \gamma^T(AZ+\mu)+\beta_{1} \\ &= \gamma^TAZ+\gamma^T\mu+\beta_{1}, \end{align}$$ which is simply an affine combination of $\gamma_{i}X_{i}+\beta_{1}$. As such,

U=\begin{bmatrix} U_{1} \ \vdots \ U_{n} \end{bmatrix} = \Gamma AZ+(\Gamma \mu+\beta)=BZ+\lambda,

which satisfies definition 1 with coefficients $B=\Gamma A$ and mean $\lambda=\Gamma\mu+\beta$, and therefore $U \sim \mathcal{N}(\Gamma \mu+\beta, \Gamma AA^T\Gamma^T)$ #### (*Covariance*) For $X=AZ+\mu$, it follows that $\Sigma=AA^T$. Let $\Sigma_{i,j}=\operatorname{Cov}(X_{i}, X_{j})$. This means that $$\begin{align} \Sigma_{i,j}&=\mathbb{E}[X_{i}X_{j}]-\mathbb{E}[X_{i}]\mathbb{E}[X_{j}] \\ &= \mathbb{E}[(a_{i}^TZ+\mu_{i})(a_{j}^TZ+\mu_{j})]-(a_{i}^T\mathbb{E}[Z]+\mu_{i})(a_{j}^T\mathbb{E}[Z]+\mu_{j}) \\ &= \mathbb{E}[a_{i}^TZa^T_{j}Z+a_{j}^TZ\mu_{i}+a_{i}^TZ\mu_{j}+\mu_{i}\mu_{j}]-\mu_{i}\mu_{j} \\ &= a_{i}^T\mathbb{E}[ZZ^T]a_{j} \\ &= a_{i}^Ta_{j} \end{align}$$ #### (*Covariance Partition*) For random vectors $X,Y$, $\Sigma=\begin{bmatrix}\Sigma_{X} & \Sigma_{XY} \\ \Sigma_{YX} & \Sigma_{Y}\end{bmatrix}$. #### (*Covariance LLSE*) For any joint Gaussians, $\mathbb{L}(Y \mid X)=\mu_{Y}+\Sigma_{YX}\Sigma^{-1}_{X}(X-\mu_{X})$. #### (LLSE/MMSE Equivalence) For any joint Gaussians, $\mathbb{L}(Y \mid X)=\mathbb{M}(Y \mid X)$. # Example >[!example] I. MMSE for Joint Gaussians > >Find $\mathbb{E}(X \mid Y,Z)$ for the jointly Gaussian random vector > >$$ \begin{bmatrix} X\\ Y\\ Z \end{bmatrix} \sim \mathcal{N}\left( \mu = \begin{bmatrix}0\\0\\0\end{bmatrix}, \Sigma= \begin{bmatrix} 5 & 3 & 1\\ 3 & 9 & 3\\ 1 & 3 & 2 \end{bmatrix} \right). >$$ > >>[!example]- Solution. >>We proceed by acknowledging that $\operatorname{Cov}(X,Y)=3$ and $\operatorname{Cov}(X,Z)=1$, as well as that $\operatorname{Var}(X)=5$. Since LLSE is equal to the MMSE, we use the formula for general partition of the LLSE: >> >>$$\begin{align} \mathbb{L}[X \mid Y,Z]&=\mathbb{L}[X \mid Y]+\mathbb{L}[X \mid Z-\mathbb{L}[Z \mid Y]] \\ &= \mathbb{L}[X \mid Y]+\mathbb{L}\left[ X \mid Z-\frac{1}{3}Y \right]. \end{align}$$ >>The variable $Z-\frac{1}{3}Y$ has a mean of 0 (since both $Y,Z$ are zero-mean), and a variance of $\operatorname{Var}(Z)-\frac{1}{9}\operatorname{Var}(Y)=2-1=1$. We also have by bilinearity that $\operatorname{Cov}\left( X,Z-\frac{1}{3}Y \right)=\operatorname{Cov}(X,Z)-\frac{1}{3}\operatorname{Cov}(X,Y)=1-1=0$. Since the variables are jointly Gaussian, this implies that $X$ and $Z-\frac{1}{3}Y$ are independent, and therefore, >>$$\begin{align} \mathbb{L}\left[X \mid Z-\frac{1}{3}Y\right]=0. \end{align}$$ >>Finally, we have that >>$$\begin{align} \mathbb{L}[X \mid Y,Z]&=\mathbb{L}[X,Y] \\ &= \boxed{\frac{1}{3}Y}. \end{align}$$ >[!example] II. Joint Gaussians As Linear Transformations of IID Gaussians > >Let $X=\begin{bmatrix}X_{1} & X_{2}\end{bmatrix}^T$ be a jointly Gaussian random vector with mean $\mu=\begin{bmatrix}0 & 0\end{bmatrix}^T$ and covariance >$$ \Sigma_X = \begin{bmatrix} 2 & 1\\ 1 & 2 >\end{bmatrix} = >\begin{bmatrix} \frac{\sqrt{3}+1}{2} & \frac{\sqrt{3}-1}{2}\\[6pt] \frac{\sqrt{3}-1}{2} & \frac{\sqrt{3}+1}{2} \end{bmatrix} \begin{bmatrix} \frac{\sqrt{3}+1}{2} & \frac{\sqrt{3}-1}{2}\\[6pt] \frac{\sqrt{3}-1}{2} & \frac{\sqrt{3}+1}{2} >\end{bmatrix}^T = A A^T.$$ > >**A)** Express $X_{1},X_{2}$ as linear combinations of i.i.d. standard Gaussian random variables. >>[!example]- A) Solution. >>Let $Z$ be a random vector, such that $Z_{i} \sim \mathcal{N}(0, 1)$. Since $\mu=0$, we have >>$$\begin{bmatrix} X_{1} \\ X_2 >>\end{bmatrix} = >>A\begin{bmatrix} Z_{1} \\ Z_{2} >>\end{bmatrix} = >>\begin{bmatrix} \dfrac{\sqrt{ 3 }+1}{2}Z_{1}+\dfrac{\sqrt{ 3 }-1}{2}Z_{2} \\ \dfrac{\sqrt{ 3 }-1}{2}Z_{1}+\dfrac{\sqrt{ 3 }+1}{2}Z_{2} >>\end{bmatrix}.$$ > >**B)** Find a matrix $B$ such that the components of $BX$ are i.i.d. $\mathcal{N}(0,1)$. >>[!example]- B) Solution. >>We need to find some $B$, such that $BX=Z$. We observe that $X=AZ$. Since $A$ is invertible (i.e., it has a nonzero determinant), $A^{-1}X=Z$, and therefore $B=A^{-1}$. We proceed by finding $\det A=\sqrt{3}$, and therefore, >>$$B=A^{-1}=\frac{1}{\sqrt{ 3 }}\begin{bmatrix} \dfrac{\sqrt{ 3 }+1}{2} & \dfrac{1-\sqrt{ 3 }}{2} \\ \dfrac{1-\sqrt{ 3 }}{2} &\dfrac{\sqrt{ 3 }+1}{2} \end{bmatrix}$$ >[!example] III. Gaussian Random Vector MMSE > >Consider the Gaussian random vector