1.2.1 Maximum Likelihood Estimation

Maximum Likelihood (ML) Estimate

The maximum-likelihood (ML) estimate chooses the estimator under which the observations are “most probable” by maximizing their likelihood. Let $X=(X_1,\dots ,X_n)$ be observed data described by the joint PMF $p_X(x;\theta )$ whose form depends on the unknown parameter $\theta$, known as the likelihood function. Then, the maximum-likelihood estimate of $\theta$ is that value which maximizes the likelihood function, such that
${\hat{\theta }}_{\text{ML}}(x)=\arg {\max }_{\theta }{p}_{X\mid \Theta }(x_1,\dots ,x_n\mid \theta ).$
If the observations in $X$ are independent, the likelihood function can be decomposed as $p_{X\mid \Theta }(x_1,\dots ,x_n\mid \theta )=\prod _{i=1}^n{p}_{X_i\mid \Theta }(x_i\mid \theta )$. Then, one may employ the log-likelihood function, denoted
$\ln p_{X\mid \Theta }(x_1,\dots ,x_n\mid \theta )={\sum }_{i=1}^n\ln p_{X_i\mid \Theta }(x_i\mid \theta ).$

Working With Continous Data $X$ consists of continuous data, one may simply use the likelihood function $f_X(x;\theta )$.

For the case where the observation vector

Properties

(Invariance) For some function $h$, if $\zeta =h(\theta )$, then ${\hat{\zeta }}_n\in h({\hat{\Theta }}_n)$.

(Asymptotic Normality) If $\theta$ is scalar, $\frac{{\hat{\Theta }}_n-\theta }{\sigma ({\hat{\Theta }}_n)}\to \mathcal{N}(0,1)$.

Examples

MLE of Uniform Distribution

Find the MLE of $\theta$ given $X_1,\dots ,X_n{\sim }_{\text{iid}}\text{Uniform}([0,\theta ])$.

Solution $p_X(x;\theta )=\frac{1}{\theta }{1}_{\{0\le x\le \theta \}}$. Given i.i.d. variables, the likelihood function can be written using log-likelihood as $p_X(x_1,\dots ,x_n;\theta )={\sum }_{i=1}^n\ln p_X(x_i;\theta )1_{\{0\le x_i\le \theta \}}.$ When $\hat{\theta }={\max }_i{X}_i$, $p_X(x_1,\dots ,x_n;\theta )=-n\ln \theta .$ Taking the derivative of the log-likelihood on the domain $[{\max }_i{X}_i,+\infty )$, we have $\frac{\delta }{\delta \theta }(-n\ln \theta )=-\frac{n}{\theta }<0.$ This means that the likelihood is strictly decreasing in $\theta$. As such, $\hat{\theta }={\max }_i{X}_i$.

Since the distribution is uniform, it follows that

MLE of the Gaussian

Consider the Gaussian distribution $\mathcal{N}(\mu ,{\sigma }^2)$, such that
$P(x)=\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right).$

Now, let $\{X_i{\}}_{i=1}^n$ be a set of iid Gaussian random variables. We can find the maximum likelihood estimates $\hat{\mu }$ and ${\hat{\sigma }}^2$ by first taking the maximum likelihood as
\begin{align}P(X \mid \mu, \sigma^2) &= \prod^{n}_{i=1} P(X_{i} \mid \mu, \sigma^2) \\ &= \sum^{n}_{i=1}\ln P(X_{i} \mid \mu, \sigma^2) \\ &\propto -N\frac{(x-\mu)^2}{2\sigma^2}\end{align}

Maximum Likelihood Estimation for the Binary Symmetric Channel

You are testing a digital link that corresponds to a BSC with some error probability $ϵ\in [0,0.5]$.
Suppose that you observe an input bit $X$ and an output bit $Y$. Calculate the MLE of $ϵ$.

Solution. ${\widehat{ϵ}}_{\text{MLE}}=\arg \underset{ϵ\in [0,0.5]}{\max }{\mathbb{P}}_{ϵ}(X=x,Y=y),$ i.e., we need to first figure out the joint distribution ${\mathbb{P}}_{ϵ}(X=x,Y=y)$.

Here, we want to find