Linear Regression

Linear Regression

Linear regression models a scalar response $y$ as an affine function of one or more inputs $\mathbf{x}$. Given a dataset $\{(x_i,y_i){\}}_{i=1}^n$ with $x_i\in \mathbb{R}$, the simple linear regression model is
$y_i={\beta }_0+{\beta }_1{x}_i+{\epsilon }_i,i=1,\dots ,n,$
where ${\beta }_0$ is the intercept, ${\beta }_1$ is the slope, and ${\epsilon }_i$ is a noise term with $\mathbb{E}[{\epsilon }_i]=0$. The parameters are estimated by ordinary least squares (OLS), minimizing the residual sum of squares
$Q({\beta }_0,{\beta }_1)={\sum }_{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2.$
Setting the partial derivatives to zero yields the closed-form estimators
${\hat{\beta }}_1=\frac{{\sum }_i(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_i(x_i-\bar{x}{)}^2}=\frac{\widehat{\mathrm{Cov}}(x,y)}{\widehat{\mathrm{Var}}(x)},{\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}.$
The fitted line always passes through the centroid $(\bar{x},\bar{y})$.

The general case with $p-1$ predictors (the multiple linear regression model) writes each observation as $y_i={\mathbf{x}}_i^{\top }\mathbf{\beta }+{\epsilon }_i$ and produces the matrix-form OLS solution $\hat{\mathbf{\beta }}=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$, covered in its own note.

Under the additional assumption that the noise is i.i.d. Gaussian, ${\epsilon }_i\stackrel{\text{iid}}{\sim }\mathcal{N}(0,{\sigma }^2)$, the OLS coefficients coincide with the maximum-likelihood estimates, and the framework gains exact distributional theory for inference: confidence intervals, $t$-tests on individual coefficients, and $F$-tests on groups. The Gauss-Markov theorem says that even without Gaussianity, OLS is the best linear unbiased estimator when the noise is uncorrelated with constant variance.

Intuition

Linear regression is the simplest model that explains how one quantity changes with another. The slope ${\beta }_1$ is the expected change in $y$ for a unit change in $x$; the intercept ${\beta }_0$ is the expected $y$ when $x=0$. Fitting reduces to minimizing the squared vertical distance between the observed points and the line, which is what least squares means literally. The closed-form solution exists because the squared-error objective is a quadratic in $({\beta }_0,{\beta }_1)$, with a unique minimum that the partial-derivative equations recover exactly.

The estimator ${\hat{\beta }}_1=\widehat{\mathrm{Cov}}(x,y)/\widehat{\mathrm{Var}}(x)$ has an interpretable structure. The numerator measures how strongly $y$ moves with $x$ in the data; the denominator rescales by how much $x$ varies. So the slope is a normalized covariance, with units of ”$y$ per unit of $x$.” If $x$ and $y$ are uncorrelated, ${\hat{\beta }}_1=0$ and the best linear prediction is the constant $\bar{y}$ — knowing $x$ does not help.

The MLE perspective adds a probabilistic story on top. If the noise really is Gaussian, the likelihood of the data factors into $n$ Gaussian densities, and maximizing the log-likelihood is mathematically identical to minimizing the sum of squared errors. So OLS and MLE agree on the regression coefficients; the only thing the Gaussian assumption adds is a closed-form variance estimator and the distributional theory for inference. Without Gaussianity the coefficient estimates remain valid by Gauss-Markov; only the standard errors and $p$-values become approximate (relying on a CLT) rather than exact.

Properties

(Slope and Intercept Estimators) The OLS coefficients have closed-form expressions in terms of sample moments.

Setting $\partial Q/\partial {\beta }_1=0$ and $\partial Q/\partial {\beta }_0=0$ and solving the resulting normal equations gives
${\hat{\beta }}_1=\frac{{\sum }_i(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_i(x_i-\bar{x}{)}^2},{\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}.$
The slope is the sample covariance of $x$ and $y$ divided by the sample variance of $x$; the intercept is whatever shift makes the line pass through $(\bar{x},\bar{y})$. Both estimators are linear in $\mathbf{y}$, which is what makes their distributional properties clean under noise assumptions.

(Population Interpretation) OLS minimizes the population mean-squared prediction error.

Viewing $(x,y)$ as a random pair, the population objective $\mathbb{E}[(y-{\beta }_0-{\beta }_{1}x{)}^2]$ is minimized at ${\beta }_1=\mathrm{Cov}(x,y)/\mathrm{Var}(x)$ and ${\beta }_0=\mathbb{E}[y]-{\beta }_1\mathbb{E}[x]$, the population analogues of the sample formulas. So OLS estimates the parameters of the best linear predictor of $y$ from $x$, regardless of whether the true conditional expectation $\mathbb{E}[y\mid x]$ is actually linear. When it is not, OLS still recovers the linear approximation that minimizes squared error.

(Variance Decomposition) Total variability in $y$ splits into explained and residual parts.

The identity ${\sum }_i(y_i-\bar{y}{)}^2={\sum }_i({\hat{y}}_i-\bar{y}{)}^2+{\sum }_i(y_i-{\hat{y}}_i{)}^2$ decomposes the total sum of squares (SST) into the regression sum of squares (SSR, explained by the fit) and the residual sum of squares (SSE, left over). The decomposition is a Pythagorean identity from the orthogonality of the residual vector to the fitted vector, and it is the foundation of the coefficient of determination
$R^2=\frac{\mathrm{SSR}}{\mathrm{SST}}=1-\frac{\mathrm{SSE}}{\mathrm{SST}},$
the fraction of $y$‘s variability captured by the regression. In simple regression $R^2$ equals the squared sample correlation between $x$ and $y$.

(Maximum Likelihood Under Gaussian Noise) OLS coincides with the MLE when ${\epsilon }_i\stackrel{\text{iid}}{\sim }\mathcal{N}(0,{\sigma }^2)$.

Under the Gaussian noise assumption, each $y_i$ is independently $\mathcal{N}({\beta }_0+{\beta }_1{x}_i,{\sigma }^2)$, and the log-likelihood is
$\ell ({\beta }_0,{\beta }_1,{\sigma }^2)=-\frac{n}{2}\log (2\pi {\sigma }^2)-\frac{1}{2{\sigma }^2}{\sum }_i(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2.$
Maximizing over $({\beta }_0,{\beta }_1)$ is identical to minimizing the sum of squared errors, so the MLE coefficients are the OLS coefficients exactly. Maximizing over ${\sigma }^2$ gives the biased estimator ${\hat{\sigma }}_{\text{MLE}}^2=Q({\hat{\beta }}_0,{\hat{\beta }}_1)/n$; the unbiased version divides by $n-2$ instead, the residual sum of squares per degree of freedom remaining after fitting two parameters.

(Gauss-Markov Theorem) OLS is the best linear unbiased estimator under uncorrelated, constant-variance noise.

Without any distributional assumption beyond $\mathbb{E}[\mathbf{\epsilon }]=0$ and $\mathrm{Cov}(\mathbf{\epsilon })={\sigma }^2\mathbf{I}$, no other linear unbiased estimator of $\mathbf{\beta }$ has smaller variance than OLS. The result does not require Gaussianity; it relies only on the noise covariance being scalar. When the noise is correlated or heteroskedastic ($\mathrm{Cov}(\mathbf{\epsilon })=\mathbf{\Sigma }\ne {\sigma }^2\mathbf{I}$), the BLUE becomes the generalized least squares estimator ${\hat{\mathbf{\beta }}}_{\text{GLS}}=({\mathbf{X}}^{\top }{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }{\mathbf{\Sigma }}^{-1}\mathbf{y}$.

(Inference Under Normality) Coefficients have a $t$-distribution.

If the Gaussian noise assumption holds, $\hat{\mathbf{\beta }}\sim \mathcal{N}(\mathbf{\beta },{\sigma }^2({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1})$, and the standardized statistic $t_j={\hat{\beta }}_j/\mathrm{SE}({\hat{\beta }}_j)$ follows a $t_{n-p}$ distribution under $H_0:{\beta }_j=0$. This gives exact-finite-sample $p$-values and confidence intervals. Without Gaussianity, the same statistic is approximately Normal by the central limit theorem when $n$ is large, and the $t$-tables remain a good approximation.

(Residual Diagnostics) Plot the residuals before trusting the fit.

The standard checks after fitting are: a residual vs. fitted plot to spot heteroskedasticity (fanning) or model misspecification (curvature), a Q-Q plot of residuals against the Normal distribution to check the noise assumption, and a serial-correlation diagnostic (e.g., the ACF of residuals) when the data are sequential. Non-trivial structure in the residuals signals that the linear-Gaussian assumptions are broken and the standard errors are unreliable.

Examples

Heights of fathers and sons

Pairs of father-son heights $(x_i,y_i)$ are recorded for $n$ families. Fit $y={\beta }_0+{\beta }_{1}x+\epsilon$ and interpret.

Solution ${\hat{\beta }}_1=\widehat{\mathrm{Cov}}(x,y)/\widehat{\mathrm{Var}}(x)$ measures how many inches taller a son is, on average, when his father is one inch taller. Galton's classic finding was ${\hat{\beta }}_1\approx 0.5$, much less than 1: tall fathers tend to have sons closer to the population mean than they themselves are. This is the original "regression to the mean" observation that gave the technique its name. The intercept ${\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}$ then absorbs whatever shift makes the line pass through the data centroid.

Predicting house price from square footage

$y_i$ is the sale price in thousands of dollars; $x_i$ is the square footage. After fitting OLS on $n=200$ houses, ${\hat{\beta }}_1=0.18$ and ${\hat{\beta }}_0=30$ with standard errors $0.02$ and $15$.

Interpretation $0.18$ thousand dollars ($180) to the predicted price, with $95\%$ confidence interval roughly $0.18\pm 2\cdot 0.02=(0.14,0.22)$. The intercept of $30k is the predicted price of a hypothetical zero-square-foot house, which is not physically meaningful but allows the line to fit the data; the large standard error on the intercept reflects that the data probably do not extend down to $x=0$. The corresponding $R^2$ would tell you the fraction of price variance the model explains, e.g., $R^2=0.65$ would mean square footage alone explains 65% of the price variation.

Each additional square foot adds

See Also

• 1.2a. Multiple Linear Regression: the matrix-form generalization to many predictors.
• 1.4. Nonlinear Regression: when the response is not linear in the parameters.
• 1.5. Regularization: ridge and lasso for high-dimensional or correlated designs.
• 1.6. Cross-Validation: model-selection machinery for regression and classification.
• 3.8.1 Bayesian Linear Regression: the Bayesian counterpart, where $\mathbf{\beta }$ has a prior and the output is a posterior distribution.
• 2.4 Time Series Regression: how OLS adapts to time series with correlated errors and time-indexed predictors.
• 1.2.3 Linear Least Squares Estimation: the inferential framework’s treatment of linear least squares.