1.2.3 Linear Least Squares Estimation

Linear Least Squares Estimate (LLSE)

The linear least‐squares estimator (LLSE) of a random variable $Y$ given another random variable $X$ is the best linear predictor of $Y$. Concretely, we seek constants $a$ and $b$ such that the linear function $\hat{Y}=aX+b$ minimizes the squared error $\mathbb{E}[(Y-\hat{Y}{)}^2]$:
$\mathbb{L}[Y\mid X]={\min }_{a,b}\mathbb{E}[(Y-(aX+b){)}^2],$
which is achieved by the optimal value
$\mathbb{L}[Y\mid X]=\mathbb{E}[Y]+\frac{\mathrm{Cov}(X,Y)}{\mathrm{Var}(X)}(X-\mathbb{E}[X]).$
The LLSE is unbiased, meaning that $\mathbb{E}[{\hat{Y}}_{\text{LLSE}}]=\mathbb{E}[Y]$.

Intuition

In the geometric sense, things become much simpler to explain. Let the random variables $X,Y$ be equivalent to vectors $x,y\in L^n(\Omega )$ in the Hilbert space of dimension $n$, such that $x,y,u$, are linearly independent, and their span forms the 3-dimensional Hilbert subspace of random variables. Here, $u$ is the unit-norm uniform vector (i.e., $u=\frac{1}{\Vert 1{\Vert }_2}$). Then:

1. $\mathbb{E}[XY]\equiv ⟨x,y⟩$.
2. $\mathbb{E}[X^2]\equiv \Vert x{\Vert }_2^2$.
3. $\rho =\frac{\mathbb{E}[XY]}{\sqrt{\mathbb{E}[X]\mathbb{E}[Y]}}\equiv \frac{⟨x,y⟩}{\Vert x{\Vert }_2\Vert y{\Vert }_2}$.

Now, let $\bar{x}=\frac{x-⟨x,u⟩u}{\Vert x-⟨x,u⟩u{\Vert }_2}$ be the Gram-Schmidt orthonormalized vector of $x$, such that $\{\bar{x},u\}$ form a basis in 2-dimensional space. We can project $y$ onto this basis, such that
$\mathbb{L}[Y\mid X]={\mathrm{proj}}_{\mathrm{Span}(\bar{x},u)}(x)={\mathrm{proj}}_{\mathrm{Span}(u)}(x)+{\mathrm{proj}}_{\mathrm{Span}(\bar{x})}(x).$
This is equivalent to
$\mathbb{L}(Y\mid X)=⟨y,u⟩u+⟨y,\bar{x}⟩\bar{x}=\mathbb{E}[Y]+\mathbb{E}[\bar{X}Y]\cdot \bar{X}.$
Here, $\bar{X}=\frac{X-\mathbb{E}[X]}{\sqrt{\mathrm{Var}(X)}}$. Substituting, we get the standard LLSE equation:

\mathbb{L}(Y \mid X)&=\mathbb{E}[Y]+\frac{\mathbb{E}[XY]-\mathbb{E}[X] \cdot \mathbb{E}[Y]}{\sqrt{ \operatorname{Var}(X)} } \cdot \frac{X-\mathbb{E}[X]}{\sqrt{ \operatorname{Var}(X) }} \\ \\ &= \mathbb{E}[Y] + \frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(X)}(X-\mathbb{E}[X]). \end{align}$$ # Properties #### (*Orthogonality*) $\operatorname{Cov}(E, Y)=0$, where $E=Y-\hat{Y}_{\text{LLSE}}$. #### (*Independence Partition*) It holds that $\mathbb{L}[Y \mid X,Z]=\mathbb{L}[Y \mid X]+\mathbb{L}[Y \mid Z]$ if $X, Z$ are independent and $\mu=\begin{bmatrix}0 & 0 & 0\end{bmatrix}^T$. Let $x,y,z$ represent $X,Y,Z$ in the Hilbert space. Since $y,z$ are orthogonal, $\left\{ \bar{y}=\frac{y}{\|y\|_{2}}, \bar{z}=\frac{z}{\|z\|_{2}} \right\}$ form an orthonormal basis. We can decompose the projection into $$\begin{align} \mathbb{L}(X \mid Y,Z) &= \operatorname{proj}_{\operatorname{span}(\bar{y},\bar{z})}(x) \\ &= \operatorname{proj}_{\operatorname{span}(\bar{y})}(x)+\operatorname{proj}_{\operatorname{span}(\bar{z})}(x) \\ &= \langle x, \bar{y} \rangle \bar{y} + \langle x, \bar{z} \rangle \bar{z} \\ &= \mathbb{E}[X\bar{Y}]\bar{Y}+\mathbb{E}[X\bar{Z}]\bar{Z}, \end{align}$$ where $\bar{Y}=\frac{Y}{\sqrt{ \mathbb{E}[Y^2] }}$ and $\bar{Z}=\frac{Z}{\sqrt{ \mathbb{E}[Z^2] }}$. As such, $$\begin{align} \mathbb{L}(X \mid Y,Z)&=\frac{\mathbb{E}[XY]}{\mathbb{E}[Y^2]}Y+\frac{\mathbb{E}[XZ]}{\mathbb{E}[Z^2]}Z \\ &= \mathbb{L}(X \mid Y) + \mathbb{L}(X \mid Z). \end{align}$$ #### (Zero-Mean Partition) $\mathbb{L}[Y \mid X,Z]=\mathbb{L}[Y \mid X]+\mathbb{L}[Y \mid Z - \mathbb{L}[Z \mid X]]$ for any $X,Y,Z$ if $\mu=\begin{bmatrix}0 & 0 & 0\end{bmatrix}^T$. We proceed in the same manner as above, but with an extra step: Gram-Schmidt. # Examples >[!example] Orthogonal LLSE > >Consider zero-mean random variables $X,Y,Z$ with $Y,Z$ orthogonal. > >**A)** Show that >$$\mathbb{L}(X \mid Y,Z) = \mathbb{L}(X \mid Y) + \mathbb{L}(X \mid Z).$$ >>[!example]- A) Solution. >>Let $x,y,z$ represent $X,Y,Z$ in the Hilbert space. Since $y,z$ are orthogonal, $\left\{ \bar{y}=\frac{y}{\|y\|_{2}}, \bar{z}=\frac{z}{\|z\|_{2}} \right\}$ form an orthonormal basis. We can decompose the projection into >>$$\begin{align} \mathbb{L}(X \mid Y,Z) &= \operatorname{proj}_{\operatorname{span}(\bar{y},\bar{z})}(x) \\ &= \operatorname{proj}_{\operatorname{span}(\bar{y})}(x)+\operatorname{proj}_{\operatorname{span}(\bar{z})}(x) \\ &= \langle x, \bar{y} \rangle \bar{y} + \langle x, \bar{z} \rangle \bar{z} \\ &= \mathbb{E}[X\bar{Y}]\bar{Y}+\mathbb{E}[X\bar{Z}]\bar{Z}, \end{align}$$ where $\bar{Y}=\frac{Y}{\sqrt{ \mathbb{E}[Y^2] }}$ and $\bar{Z}=\frac{Z}{\sqrt{ \mathbb{E}[Z^2] }}$. As such, >>$$\begin{align} \mathbb{L}(X \mid Y,Z)&=\frac{\mathbb{E}[XY]}{\mathbb{E}[Y^2]}Y+\frac{\mathbb{E}[XZ]}{\mathbb{E}[Z^2]}Z \\ &= \mathbb{L}(X \mid Y) + \mathbb{L}(X \mid Z), \end{align}$$ given that $X,Y,Z$ are all zero-mean (note that the means are all absent from the expression, which is why there is a mismatch between the general formula and the zero-mean formula). > >**B)** Now, for any zero-mean random variables $X,Y,Z$, explain why it holds that >>[!example]- B) Solution. >>We proceed in the same manner as in part **A)**, but with an extra step: Gram-Schmidt. >[!example] Photodetector LLSE > >[!example] Balls in Bins Estimation >