Regularization

Regularization

Regularization adds a penalty on the coefficients of a regression model, trading a small amount of bias for a large reduction in variance. The penalized objective is
${\hat{\mathbf{\beta }}}_{\text{pen}}=\arg {\min }_{\mathbf{\beta }}\Vert \mathbf{y}-\mathbf{X}\mathbf{\beta }{\Vert }_2^2+\lambda P(\mathbf{\beta }),$
where $\lambda \ge 0$ is the regularization strength and $P(\mathbf{\beta })$ is a penalty function. The two canonical choices are the ridge (or $L_2$) penalty $P(\mathbf{\beta })=\Vert \mathbf{\beta }{\Vert }_2^2={\sum }_j{\beta }_j^2$, and the lasso (or $L_1$) penalty $P(\mathbf{\beta })=\Vert \mathbf{\beta }{\Vert }_1={\sum }_j|{\beta }_j|$. Their elastic net combination $P(\mathbf{\beta })=\alpha \Vert \mathbf{\beta }{\Vert }_1+(1-\alpha )\Vert \mathbf{\beta }{\Vert }_2^2$ blends the two.

Ridge has a closed-form solution
${\hat{\mathbf{\beta }}}_{\text{ridge}}=({\mathbf{X}}^{\top }\mathbf{X}+\lambda \mathbf{I}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y},$
obtained by minimizing the augmented normal equations. The added $\lambda \mathbf{I}$ makes the matrix invertible even when ${\mathbf{X}}^{\top }\mathbf{X}$ is singular, which is exactly what happens when predictors are collinear or when there are more predictors than observations ($p>n$). Lasso has no closed form because $|{\beta }_j|$ is non-differentiable at zero, but coordinate descent and proximal gradient methods solve it efficiently. The non-differentiability is the feature, not the bug: the kink at zero is what produces sparse solutions, where many ${\hat{\beta }}_j$ are exactly zero rather than just small.

Both estimators are biased, with the bias scaling with $\lambda$. As $\lambda \to 0$ they reduce to OLS; as $\lambda \to \infty$ they shrink toward zero. The $\lambda$ that minimizes out-of-sample prediction error is chosen by cross-validation. Predictors should be standardized to unit variance before fitting so the penalty acts comparably across coefficients with different natural scales.

Standardize Before Regularizing ${\sum }_j{\beta }_j^2$ or ${\sum }_j|{\beta }_j|$ treats every coordinate of $\mathbf{\beta }$ the same way, so a coefficient measured in millimeters is penalized the same as one measured in kilometers. Without standardization the regularizer behaves arbitrarily, depending on the units of each predictor. Standard practice: center every column of $\mathbf{X}$ to mean zero and scale to unit variance before fitting, leave the intercept unpenalized, and rescale the coefficients back at the end.

The penalty

Intuition

OLS is unbiased but has high variance when predictors are correlated or numerous. Two correlated predictors can have OLS coefficients with enormous standard errors, even flipping sign across resamples, because the data cannot pin down which one carries the signal. Regularization fixes this by adding a penalty that pulls coefficients toward zero, sacrificing unbiasedness for stability. The bias-variance trade-off has a sweet spot: a small amount of shrinkage reduces variance much faster than it adds bias, so the mean squared prediction error decreases.

The geometric picture is the most useful one. The OLS solution is the unconstrained minimum of the elliptical cost surface $\Vert \mathbf{y}-\mathbf{X}\mathbf{\beta }{\Vert }^2$, which has level sets that are ellipses around ${\hat{\mathbf{\beta }}}_{\text{OLS}}$. The regularized solution adds the constraint that $\mathbf{\beta }$ lies in a ball: an $L_2$ ball $\Vert \mathbf{\beta }{\Vert }_2\le t$ for ridge, an $L_1$ ball (a diamond in two dimensions) $\Vert \mathbf{\beta }{\Vert }_1\le t$ for lasso. The penalized estimate is where the cost ellipse first touches the ball. Because the $L_1$ ball has corners on the coordinate axes, the touching point often lands on a corner, where some coordinates are exactly zero. The $L_2$ ball is smooth, so the touching point generically has all coordinates non-zero but shrunken. That single geometric difference, smooth ball versus corners, is why lasso produces sparse solutions and ridge does not.

The Bayesian interpretation makes this even cleaner. Ridge is the maximum-a-posteriori estimate under a Gaussian prior $\mathbf{\beta }\sim \mathcal{N}(0,{\tau }^2\mathbf{I})$ with $\lambda ={\sigma }^2/{\tau }^2$, and lasso is the MAP under a Laplace prior $\mathbf{\beta }\sim \text{Laplace}(0,\tau )$. Larger $\lambda$ corresponds to a tighter prior around zero. So choosing $\lambda$ by cross-validation amounts to learning how much to trust the prior versus the data.

Properties

(Ridge Closed Form) The $L_2$-penalized objective has a unique solution computable in one step.

Writing the augmented objective as $\Vert \mathbf{y}-\mathbf{X}\mathbf{\beta }{\Vert }_2^2+\lambda \Vert \mathbf{\beta }{\Vert }_2^2$ and setting the gradient to zero gives the modified normal equations $({\mathbf{X}}^{\top }\mathbf{X}+\lambda \mathbf{I})\mathbf{\beta }={\mathbf{X}}^{\top }\mathbf{y}$, with the explicit solution
${\hat{\mathbf{\beta }}}_{\text{ridge}}=({\mathbf{X}}^{\top }\mathbf{X}+\lambda \mathbf{I}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}.$
Because ${\mathbf{X}}^{\top }\mathbf{X}+\lambda \mathbf{I}$ is positive definite for any $\lambda >0$, the inverse always exists, even when ${\mathbf{X}}^{\top }\mathbf{X}$ alone is singular (when $p>n$ or when columns are exactly collinear). Diagonalizing through the singular value decomposition $\mathbf{X}=\mathbf{UD}{\mathbf{V}}^{\top }$ writes the solution as ${\hat{\mathbf{\beta }}}_{\text{ridge}}=\mathbf{V}\text{diag}\left(\frac{d_j}{d_j^2+\lambda }\right){\mathbf{U}}^{\top }\mathbf{y}$, showing that ridge shrinks each principal component of $\mathbf{X}$ by the factor $d_j^2/(d_j^2+\lambda )$. Components with small singular values, the ones OLS amplifies most, are shrunk most aggressively.

(Lasso Has No Closed Form But Solves Efficiently) The $L_1$ penalty is non-differentiable at zero, but the objective is convex and decomposes coordinate-wise.

The lasso objective $\Vert \mathbf{y}-\mathbf{X}\mathbf{\beta }{\Vert }_2^2+\lambda \Vert \mathbf{\beta }{\Vert }_1$ is convex in $\mathbf{\beta }$, so a global minimum exists, but the absolute value’s kink prevents a closed-form expression. The standard solver is coordinate descent: cycle through coordinates $j=1,\dots ,p$ and update each ${\beta }_j$ by holding the others fixed, which reduces to a one-dimensional problem with the closed-form soft-threshold solution
${\hat{\beta }}_j=S_{\lambda }\left({\mathbf{x}}_j^{\top }{\mathbf{r}}_{(-j)}\right)/\Vert {\mathbf{x}}_j{\Vert }_2^2,S_{\lambda }(z)=\text{sign}(z)\cdot \max (|z|-\lambda ,0),$
where ${\mathbf{r}}_{(-j)}=\mathbf{y}-{\sum }_{k\ne j}{\mathbf{x}}_k{\hat{\beta }}_k$ is the partial residual. The soft-threshold function leaves large values alone, shrinks moderate ones, and crushes small ones to exactly zero. Iterating to convergence gives the lasso solution; LARS and proximal gradient methods are alternatives for large-scale problems.

(Sparsity) Lasso produces solutions with many exactly-zero coefficients; ridge does not.

The geometric reason is that the $L_1$ ball $\{\mathbf{\beta }:\Vert \mathbf{\beta }{\Vert }_1\le t\}$ has corners on the coordinate axes. When the OLS cost ellipse expands outward from ${\hat{\mathbf{\beta }}}_{\text{OLS}}$ until it first contacts the ball, the contact point is generically at a corner because corners stick out the furthest in any given direction. A corner where the $j$th coordinate axis touches has ${\beta }_j=0$ for all the other coordinates. The $L_2$ ball, by contrast, is perfectly smooth, so the contact point generically lies in the relative interior of a face and has no coordinates exactly zero. In high dimensions the lasso typically selects a number of active coefficients close to $\min (n,p)$; ridge keeps all $p$ coefficients non-zero. Sparsity is what makes lasso useful for variable selection: the non-zero coefficients identify a subset of “active” predictors, providing a built-in feature-selection mechanism that ridge lacks.

(Bias-Variance Trade-off) Both estimators are biased, but their variance is much smaller than OLS for moderate $\lambda$.

Decomposing the prediction MSE as $\mathbb{E}[(\hat{y}-y{)}^2]={\text{Bias}}^2+\text{Variance}+{\sigma }^2$, OLS sits at the bias-zero, high-variance extreme. Adding a small $\lambda$ introduces a small bias proportional to $\lambda \mathbf{\beta }$ but reduces the variance by a much larger amount, lowering the total MSE. The improvement is most dramatic when predictors are highly collinear or when $p$ is comparable to $n$, the regimes where OLS variance is worst. As $\lambda$ grows past its optimum the bias term begins to dominate and MSE rises again. Cross-validation finds the sweet spot empirically.

(Effective Degrees of Freedom) Regularized fits use fewer degrees of freedom than OLS, and the savings can be quantified.

For ridge, the effective degrees of freedom is
$\text{df}(\lambda )={\sum }_{j=1}^p\frac{d_j^2}{d_j^2+\lambda },$
where $d_j$ are the singular values of $\mathbf{X}$. At $\lambda =0$ this equals $p$ (full OLS); as $\lambda \to \infty$ it shrinks toward zero. For lasso, the effective degrees of freedom equals the number of non-zero coefficients in the fit (Zou-Hastie-Tibshirani, 2007). These quantities replace the parameter count $p$ in information criteria like AIC and BIC, allowing fair model comparison between fits at different $\lambda$.

(Scale Equivariance) Ridge and lasso are not scale-invariant; predictors must be standardized before fitting.

OLS is invariant to rescaling: multiplying column $j$ of $\mathbf{X}$ by 100 simply divides ${\hat{\beta }}_j$ by 100, leaving fitted values unchanged. Ridge and lasso are not, because the penalty ${\sum }_j{\beta }_j^2$ or ${\sum }_j|{\beta }_j|$ treats every coefficient on equal footing. A coefficient on a small-scale predictor (already large in magnitude to compensate for its small scale) is penalized more heavily than one on a large-scale predictor. The fix is to standardize: center each column of $\mathbf{X}$ to mean zero and scale to unit variance before fitting, leave the intercept unpenalized (subtract $\bar{y}$ from $\mathbf{y}$ for the same reason), and rescale coefficients back to the original units after fitting. This is what every production library does by default.

(Elastic Net) Combines ridge stability with lasso sparsity.

When predictors are highly correlated, lasso has a quirk: it tends to pick one representative from each correlated group and zero the rest, somewhat arbitrarily. Ridge keeps all of them shrunken together. The elastic net penalty
$P_{\text{EN}}(\mathbf{\beta })=\alpha \Vert \mathbf{\beta }{\Vert }_1+(1-\alpha )\Vert \mathbf{\beta }{\Vert }_2^2$
combines the two, controlled by a mixing parameter $\alpha \in [0,1]$. The $L_2$ component encourages correlated predictors to enter the model together; the $L_1$ component still produces sparsity. For most applied problems with correlated predictors, elastic net outperforms either pure ridge or pure lasso, with $\alpha$ chosen by cross-validation alongside $\lambda$.

(Bayesian Interpretation) Ridge and lasso are MAP estimates under Gaussian and Laplace priors.

Under the Gaussian likelihood $\mathbf{y}\mid \mathbf{\beta }\sim \mathcal{N}(\mathbf{X}\mathbf{\beta },{\sigma }^2\mathbf{I})$, placing a Gaussian prior $\mathbf{\beta }\sim \mathcal{N}(0,{\tau }^2\mathbf{I})$ produces a posterior whose mode is the ridge estimate with $\lambda ={\sigma }^2/{\tau }^2$. A Laplace prior $\mathbf{\beta }\sim \text{Laplace}(0,\tau )$ produces lasso with $\lambda =2{\sigma }^2/\tau$. The interpretation is that the regularizer encodes a prior belief that coefficients are small, and the strength $\lambda$ encodes how confident that belief is. This is also why deep-learning practitioners call $L_2$ regularization weight decay: each gradient step shrinks weights by a factor of $1-\eta \lambda$ before applying the data gradient, mirroring the pull of a Gaussian prior toward zero.

Examples

Two correlated predictors

Two predictors $x_1,x_2$ are nearly identical: $x_2=x_1+0.01\epsilon$, with the true model $y=2x_1+\text{noise}$. Compare OLS, ridge, and lasso fits.

What each estimator does $({\hat{\beta }}_1,{\hat{\beta }}_2)=(50,-48)$ on one resample and $(-30,32)$ on another, since the data cannot distinguish "all the weight on $x_1$" from "split between them." The fitted values are similar but the individual coefficients have huge standard errors. Ridge with moderate $\lambda$ produces a stable answer like $({\hat{\beta }}_1,{\hat{\beta }}_2)=(1,1)$, splitting the credit evenly between the two collinear predictors. Lasso typically picks one of them: $({\hat{\beta }}_1,{\hat{\beta }}_2)=(2,0)$ or $(0,2)$, depending on tiebreaks. Elastic net interpolates: it identifies that both predictors are active and gives them similar but not identical weights. The right choice depends on whether you want stability (ridge), sparsity (lasso), or both (elastic net).

OLS gives wildly unstable estimates: maybe

$p>n$ regime

A genomic study has $n=100$ patients and $p=20,000$ gene expression measurements as predictors of a clinical outcome. OLS is undefined because ${\mathbf{X}}^{\top }\mathbf{X}$ is singular.

Approach ${\mathbf{X}}^{\top }\mathbf{X}+\lambda \mathbf{I}$ is invertible, and the prediction is reasonable but uses every gene. Lasso also produces a unique solution and selects at most $\min (n,p)=100$ non-zero coefficients, identifying a sparse subset of genes that explain the outcome; this set is interpretable for downstream biological investigation. In practice, elastic net is preferred over pure lasso here, because correlated genes (e.g., from the same pathway) often need to enter the model together rather than have one chosen arbitrarily. Cross-validation picks $\lambda$ and $\alpha$ jointly. The selected gene list is typically reported as candidates for follow-up rather than as definitive biomarkers, since the lasso's selection is sensitive to noise in $p>n$ regimes.

Ridge produces a unique solution because

Polynomial regression

Fit $y={\sum }_{k=0}^d{\beta }_k{x}^k+\epsilon$ with $d=20$ to a smooth function observed with noise.

Without and with regularization $\lambda$ smooths the fit dramatically: high-degree coefficients shrink toward zero, and the resulting curve approximates the true smooth function. Lasso would set most high-degree coefficients to exactly zero, effectively performing automatic degree selection by keeping only the polynomial terms the data justify. This is the canonical demonstration that regularization controls overfitting even when the model is grossly over-parameterized.

Without regularization, OLS fits a wildly oscillating polynomial that interpolates the training points almost exactly but predicts terribly between them, the classic Runge phenomenon. The high-degree coefficients are huge and oscillate in sign. Ridge with a CV-chosen

See Also

• 1.2. Linear Regression: the unregularized OLS estimator that ridge and lasso shrink toward zero.
• 1.2a. Multiple Linear Regression: the matrix form ridge and lasso are built on top of.
• 1.4. Nonlinear Regression: regularization extends to nonlinear least squares with the same penalty terms.
• 1.6. Cross-Validation: how $\lambda$ (and $\alpha$ for elastic net) are chosen in practice.
• 3.8.1 Bayesian Linear Regression: ridge and lasso as MAP estimates under Gaussian and Laplace priors.
• 2.4 Time Series Regression: how these regularizers are applied to wide time-series designs (lagged predictors, Fourier bases, VAR models).