1.2.2 Maximum a Posteriori Estimation

Maximum a Posteriori (MAP) Estimate

The maximum‑a posteriori (MAP) estimate chooses the estimator under which the parameter $\theta$ is most probable after observing the data, by maximizing the posterior distribution. Let $X=(X_1,\dots ,X_n)$ be observed data with likelihood $p_X(x\mid \theta )$, and assume a prior distribution $p_{\Theta }(\theta )$ on the random parameter $\Theta$. Then, the MAP estimate of $\theta$ is the value that maximizes the posterior $p_{\Theta }(\theta \mid X)$, such that
${\hat{\theta }}_{\text{MAP}}(x)=\arg {\max }_{\theta }{p}_{\Theta \mid X}(\theta \mid x_1,\dots ,x_n)=\arg {\max }_{\theta }\{p_{X\mid \Theta }(x_1,\dots ,x_n\mid \theta )\cdot p_{\Theta }(\theta )\}.$

Examples

I. Poisson Process MAP

Customers arrive to a store according to a Poisson process with rate $\lambda =1$. The store manager learns of a rumor that one of the employees is sending every other customer to the rival store, so that deterministically, every odd-numbered customer is sent away.

Let $X=1$ be the hypothesis that the rumor is true and $X=0$ the rumor is false, assuming that both hypotheses are equally likely. Suppose a customer arrives to the store at time $0$. After that, the manager observes $T_1,\dots ,T_n$, where $T_i$ is the time of the $i$-th subsequent sale, $i=1,\dots ,n$. Derive the MAP rule to determine whether the rumor was true or not.

Solution. $f_{T_1,\dots ,T_n\mid X}(t_1,\dots ,t_n\mid 1)={\prod }_{i=1}^n(t_i-t_{i-1})e^{-(t_i-t_{i-1})}$

Since both hypotheses are equally likely, MAP is equivalent to MLE. We form the posterior

II. Laplace Prior and ${\ell }^1$-Regularization (LASSO)

Suppose you draw $n$ i.i.d. data points $(x_1,y_1),\dots ,(x_n,y_n)$, where the true relationship is given by $Y=WX+ϵ$ for $ϵ\sim \mathcal{N}(0,{\sigma }^2)$. In other words, $Y$ has a linear dependence on $X$ with additive Gaussian noise. Further suppose that $W$ has a Laplace prior distribution,
$f_W(w)=\frac{1}{2\beta }{e}^{-|w|/\beta }.$
Show that finding the MAP estimate of $W$ given the data points $\{x_i,y_i{\}}_{i=1}^n$ is equivalent to minimizing the cost function
$J(w)={\sum }_{i=1}^n(y_i-w{x}_i{)}^2+\lambda |w|,$
and determine $\lambda$.

Solution. $f_W(w)$, we need to find $f_{X,Y\mid W}(x_i,y_i\mid w)$, which is equivalent to $ϵ=Y-WX\sim \mathcal{N}(0,{\sigma }^2)$. As such, $f_{E\mid W}(ϵ\mid w)={\prod }_{i=1}^n{f}_{E_i}({ϵ}_i)={\prod }_{i=1}^n\frac{1}{\sqrt{2\pi }\sigma }{e}^{-(y_i-w{x}_i{)}^2/(2{\sigma }^2)}.$ This means that our likelihood function is $L(w)=\frac{1}{2\beta }{e}^{-|w|/\beta }{\prod }_{i=1}^n\frac{1}{\sqrt{2\pi }\sigma }{e}^{-(y_i-w{x}_i{)}^2/(2{\sigma }^2)}\propto e^{-|w|/\beta }{\prod }_{i=1}^n{e}^{-(y_i-w{x}_i{)}^2/(2{\sigma }^2)}.$ Taking the log-likelihood, we have $LL(w)=-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(y_i-w{x}_i{)}^2-\frac{|w|}{\beta }.$ Since we are maximizing for $LL(w)$, which is concave on $w$ (positive absolute values are convex, and we have a negative absolute value), then we are minimizing for $-LL(w)=\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(y_i-w{x}_i{)}^2+\frac{|w|}{\beta },$ where $J(w)=-2{\sigma }^{2}LL(w)$ with $\lambda =2{\sigma }^2/\beta$.

Since we know

III. Bayesian Estimation of Exponential Distribution

Suppose that $X$ is Exponential with unknown rate $\Lambda$. As a Bayesian practitioner, you have a prior belief that the random variable $\Lambda$ is equally likely to be ${\lambda }_1$ or ${\lambda }_2$.

A) You collect one sample $X_1$ from $X$. Find the posterior distribution $\mathbb{P}(\Lambda ={\lambda }_1\mid X_1=x_1)$.

A) Solution. $\mathbb{P}(\Lambda ={\lambda }_1\mid X_1=x_1)=\frac{\mathbb{P}(X_1=x_1\mid \Lambda ={\lambda }_1)\mathbb{P}(\Lambda ={\lambda }_1)}{\mathbb{P}(X_1=x_1)}$ Here, by definition we have that the prior is $\mathbb{P}(\Lambda ={\lambda }_1)=1/2$, and $\mathbb{P}(X_1=x_1\mid \Lambda ={\lambda }_1)={\lambda }_1{e}^{-{\lambda }_1{x}_1}$. Then, by total probability, we also have $\mathbb{P}(X_1=x_1)=1/2\cdot [{\lambda }_1{e}^{-{\lambda }_1{x}_1}+{\lambda }_2{e}^{-{\lambda }_2{x}_1}]$. As such, $\mathbb{P}(\Lambda ={\lambda }_1\mid X_1=x_1)=\frac{{\lambda }_1{e}^{-{\lambda }_1{x}_1}}{{\lambda }_1{e}^{-{\lambda }_1{x}_1}+{\lambda }_2{e}^{-{\lambda }_2{x}_1}}.$

We know by Bayes that

B) Instead of using MAP or MLE and fixing some $\lambda$ of $\Lambda$, we will keep the full information of the posterior distribution of $\Lambda$. Compute the distribution of $X$ as
$f_X(x)={\sum }_{\lambda \in \{{\lambda }_1,{\lambda }_2\}}{f}_{X\mid \Lambda }(x\mid \lambda )\cdot \mathbb{P}(\Lambda =\lambda \mid X_1=x_1).$

B) Solution. $f_{X\mid \Lambda }(x\mid \lambda )=\lambda e^{-\lambda x}$. Using the answer from the previous part, we have $f_X(x)=\frac{{\lambda }_1^2{e}^{-{\lambda }_1(x_1+x)}+{\lambda }_2^2{e}^{-{\lambda }_2(x_1+x)}}{{\lambda }_1{e}^{-{\lambda }_1{x}_1}+{\lambda }_2{e}^{-{\lambda }_2{x}_1}}\cdot$

We find that

C) The fully Bayesian approach from part B) is often computationally intractable, which is one of the main reasons why point estimates are common in practice. Supposing that ${\lambda }_1>{\lambda }_2$, compute the MAP estimate for $\Lambda$, and calculate $f_X(x)$ again using the MAP rule.

C) Solution. $\lambda$ is uniform) estimate as ${\hat{\lambda }}_{\text{MAP}}=\arg {\max }_{\lambda \in \{{\lambda }_1,{\lambda }_2\}}\mathbb{P}(X=x_1\mid \Lambda =\lambda )\cdot \mathbb{P}(\Lambda =\lambda )=\arg {\max }_{\lambda \in \{{\lambda }_1,{\lambda }_2\}}\mathbb{P}(X=x_1\mid \Lambda =\lambda ),$ where our likelihood function $\mathbb{P}(X=x_1\mid \Lambda =\lambda )=\lambda e^{-\lambda x_1}$ turns into $\ln \lambda -\lambda x_1$ when log-likelihood is applied. Given ${\lambda }_1>{\lambda }_2$, it also follows that $\ln {\lambda }_1-{\lambda }_1{x}_1>\ln {\lambda }_2-{\lambda }_2{x}_1$ only if $x_1<\frac{\ln ({\lambda }_1/{\lambda }_2)}{{\lambda }_1-{\lambda }_2}=\alpha .$ Since the value of $X_1$ determines the parameter $\lambda$ for all subsequent observations, this result implies that

We formulate our MAP/ML (equivalent here, since the prior of

\lambda_{1} & \text{if } x_{1} < \alpha \
\lambda_{2} & \text{if } x_{1} \geq \alpha.
\end{cases}$$

As such, we get that

\lambda_{1}e^{-\lambda_{1}x} & \text{if } x_{1} < \alpha \
\lambda_{2}e^{-\lambda_{2}x} & \text{if } x_{1} \geq \alpha,
\end{cases}$$
where $\alpha =\frac{\ln ({\lambda }_1/{\lambda }_2)}{{\lambda }_1-{\lambda }_2}$ is the intersection of the distributions with parameters ${\lambda }_1$ and ${\lambda }_2$.