Regression Models

Regression predicts a numeric response from one or more inputs. The notes in this folder cover the canonical machine-learning treatment: ordinary least squares, its matrix-form extension to many predictors, nonlinear models, robust estimation, regularized estimators, and the cross-validation machinery that selects between them. Classification (predicting a categorical response) lives in the next folder; the regression side here is the foundation it builds on.

Contents

• 1.1. Regression: what regression is, the supervised-learning framing, and the bias-variance setup.
• 1.2. Linear Regression: simple linear regression by OLS; the closed-form ${\hat{\beta }}_1=\widehat{\text{Cov}}(x,y)/\widehat{\text{Var}}(x)$, Gauss-Markov, and Gaussian-noise MLE.
• 1.2a. Multiple Linear Regression: matrix-form OLS with $\hat{\mathbf{\beta }}=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$, the hat matrix, residual variance, and adjusted $R^2$.
• 1.3. RANSAC: robust regression by random consensus, for designs corrupted by outliers.
• 1.4. Nonlinear Regression: nonlinear least squares, Gauss-Newton, Levenberg-Marquardt, asymptotic inference.
• 1.5. Regularization: ridge, lasso, elastic net; geometric picture, bias-variance trade-off, scale-equivariance.
• 1.6. Cross-Validation: $k$-fold, leave-one-out, train-test, stratified and nested variants, with rolling-origin reserved for the time-series adaptation.

Cross-domain applications

• 2.4 Time Series Regression applies this machinery in the time-indexed setting (trend + seasonal + lag predictors), with rolling-origin CV replacing random $k$-fold.
• 3.8.1 Bayesian Linear Regression gives the Bayesian counterpart: a prior on $\mathbf{\beta }$ and a posterior distribution instead of a point estimate.