2.4. Logistic Regression

Logistic Regression

Logistic regression models a binary output $y\in \{0,1\}$as a probability generated by a linear function of the inputs $x$. Given a dataset $\{(x_n,y_n){\}}_{n=1}^N$, where $x_n\in {\mathbb{R}}^D$ and $y_n\in \{0,1\}$, we assume the model
$p(t=1|x,w)=\sigma (w_0+w_1{x}_1+\cdots +w_D{x}_D)=\sigma (w^{\top }x),$
where $\sigma (a)=\frac{1}{1+e^{-a}}$ is the sigmoid (logistic) function that squashes real values into $[0,1]$.

One way to estimate the optimal parameters $w^{\ast }$ is by maximizing the likelihood of the observed binary (Bernoulli) labels under this probabilistic model. For each observation, it follows that
$p(y_n|x_n,w)=({\hat{y}}_n{)}^{y_n}(1-{\hat{y}}_n{)}^{1-y_n},$
where ${\hat{y}}_n=\sigma (w^{\top }{x}_n)$. We minimize the negative log-likelihood, also known as the cross-entropy loss. Note here since the points are independent, the joint likelihood is just a product of the individual probabilities:
$E(w)=-{\sum }_{n=1}^N[y_n\ln {\hat{y}}_n+(1-y_n)\ln (1-{\hat{y}}_n)]=\{\begin{array}{ll}-\ln y_n& t_n=1\\ -\ln (1-y_n)& t_n=0.\end{array}$
This loss is convex, ensuring a unique global minimum. Unlike linear regression, there is no closed-form solution for logistic regression, because $y_n=\sigma (w^{\top }{x}_n)$ depends nonlinearly on $w$.

Derivation

To reason about logistic regression, we need to think in terms of probabilities. Take in the following sentence: logistic regression models the log-odds of class $C_1$ versus $C_2$ as a linear model. Essentially, we have
$\ln \left(\frac{P(C_1\mid x)}{P(C_2\mid x)}\right)=\ln \left(\frac{P(C_1\mid x)}{1-P(C_1\mid x)}\right)=w^{T}x,$
which when solved yields the sigmoid function $\sigma (w^{T}x)=\frac{1}{1+e^{-w^{T}x}}$, and looks like this: