Time Series Regression

Time series regression applies the linear-model machinery to the setting where the predictors are functions of time (trends, seasonal indicators, sinusoids, lagged values of the response, exogenous time-indexed covariates) and the residuals are almost never independent. The mechanics of fitting the regression are the standard ones from linear and multiple linear regression; what changes is which predictors appear in the design, how to handle the serially correlated residuals, and how to validate the fit without leaking future information into past training data. The basic specification is
$y_t={\beta }_0+{\sum }_{j=1}^{p-1}{\beta }_j{x}_{tj}+w_t,t=1,\dots ,n,$
where $\{x_{tj}\}$ are the time-indexed predictors and $\{w_t\}$ is the noise process. Common choices for the predictors include a polynomial in $t$ (trend), seasonal dummies or low-frequency Fourier basis (seasonality), lagged response values $y_{t-1},y_{t-2},\dots$ (autoregressive structure), exogenous inputs $x_t^{\text{ext}}$, and lagged versions of those exogenous inputs (distributed-lag).

The single time-series-specific topic that needs its own treatment in this folder is the sinusoidal regression model, in which $y_t$ is fit by a sum of sines and cosines at one or more frequencies. With known frequencies the model is linear in $\mathbf{\beta }$ and the OLS solution applies directly; with unknown frequencies it becomes nonlinear in the parameters and is usually fit by profile least squares over a frequency grid. The model also obeys the Nyquist limit: at sampling frequency $f_s$, only frequencies $f\in [0,f_s/2]$ are identifiable, with higher frequencies aliasing into this range.

Three further practical points reshape the regression workflow when applied to time series. First, OLS standard errors are wrong under serial correlation, even though the point estimates remain unbiased; the right substitutes are HAC (Newey-West) standard errors or generalized least squares (GLS) with an estimated noise covariance. Second, regressing one nonstationary series on another can produce spurious regression: a high $R^2$ and significant coefficients even when the series are independent, driven by their shared stochastic trends. The remedy is to difference the series, test for cointegration, or include explicit trend terms. Third, cross-validation by random folds breaks chronology and is replaced by rolling-origin CV.

Use Time-Series Cross-Validation $k$-fold CV mixes future observations into past training sets and produces wildly optimistic error estimates. Always use rolling-origin CV (TimeSeriesSplit in scikit-learn) for any model selection or hyperparameter tuning on time-series data, including for choosing the ridge $\lambda$ or comparing ARIMA orders.

Random

Intuition

A time series is not a bag of independent samples that happen to be indexed by time; the index matters. Two facts follow. The predictors a time-series regression cares about are usually time itself (trends), periodic functions of time (seasonal cycles), and lagged values of the response or of exogenous inputs (memory). The residuals from such a regression almost never look like white noise on the first try; they carry the leftover serial structure the model failed to capture, and the analyst’s job is to inspect them and refine the model until they do.

The general regression machinery, OLS, MLE, ridge, lasso, cross-validation, lives in the machine-learning regression folder and is not re-derived here. This folder covers only the pieces that are time-series-specific: the sinusoidal regression model and the Nyquist limit (this note), distributed-lag identification of input-output couplings (2.4.3 Time-Lagged Regression), and rolling-origin cross-validation (2.4.5 Time-Series Cross-Validation).

Sinusoidal Regression and the Nyquist Limit

A pure sinusoid observed in noise can be written
$y_t={\beta }_0+R\cos (2\pi ft+\varphi )+{\epsilon }_t={\beta }_0+{\beta }_1\cos (2\pi ft)+{\beta }_2\sin (2\pi ft)+{\epsilon }_t,$
using the trigonometric identity $R\cos (2\pi ft+\varphi )=R\cos \varphi \cos (2\pi ft)-R\sin \varphi \sin (2\pi ft)$, which gives ${\beta }_1=R\cos \varphi$ and ${\beta }_2=-R\sin \varphi$. Recovering the amplitude and phase from the fitted coefficients uses $R=\sqrt{{\beta }_1^2+{\beta }_2^2}$ and $\varphi =\arctan (-{\beta }_2/{\beta }_1)$.

(Known Frequency) If $f$ is known, sinusoidal regression is ordinary linear regression.

The design matrix is
${\mathbf{X}}_f=\left(\begin{array}{ccc}1& \cos (2\pi f\cdot 1)& \sin (2\pi f\cdot 1)\\ 1& \cos (2\pi f\cdot 2)& \sin (2\pi f\cdot 2)\\ ⋮& ⋮& ⋮\\ 1& \cos (2\pi f\cdot n)& \sin (2\pi f\cdot n)\end{array}\right),$
and the OLS solution is $\hat{\mathbf{\beta }}=({\mathbf{X}}_f^{\top }{\mathbf{X}}_f{)}^{-1}{\mathbf{X}}_f^{\top }\mathbf{y}$. Multiple known frequencies can be stacked into one design with three columns per frequency (the constant and the two trig basis), recovering the harmonic-regression model. This is what produces the Fourier-basis representation of seasonality in classical time-series regression.

(Nyquist Limit) Only frequencies $f\in [0,f_s/2]$ are identifiable from data sampled at rate $f_s$.

A signal at frequency $f>f_s/2$ aliases into the range $[0,f_s/2]$: it is indistinguishable from a signal at frequency $|f-n{f}_s|$ for some integer $n$. The cutoff $f_{\max }=f_s/2$ is the Nyquist frequency, and $1/(2f_s)$ is the Nyquist period, the shortest period that can be resolved at sampling rate $f_s$. For monthly data ($f_s=1$ cycle/month) the Nyquist limit is $f_{\max }=0.5$ cycles/month, so the shortest detectable period is $1/0.5=2$ months. Any oscillation faster than that aliases into the resolvable range. Practically: a frequency grid for sinusoidal regression should never extend past $f_s/2$, and a sampling rate should be chosen to lie above twice the highest frequency of interest in the underlying signal.

(Unknown Frequency) If $f$ must be estimated, the model is nonlinear and fit by profile least squares over a grid.

Parameterizing the unknown $f$ explicitly turns the model into a nonlinear regression. The clean way to fit it exploits that the model is linear in $({\beta }_0,{\beta }_1,{\beta }_2)$ for any fixed $f$: pick a fine grid $\mathcal{G}\subset [0,f_s/2]$, fit OLS at every grid point to get ${\hat{\mathbf{\beta }}}_f$ and the residual sum of squares $\text{RSS}(f)=\Vert \mathbf{y}-{\mathbf{X}}_f{\hat{\mathbf{\beta }}}_f{\Vert }^2$, then select $\hat{f}=\arg {\min }_{f\in \mathcal{G}}\text{RSS}(f)$ and the corresponding ${\hat{\mathbf{\beta }}}_{\hat{f}}$. This profile least squares trick reduces the nonlinear problem to a sequence of linear regressions, one per grid point, and is what every periodogram-based frequency-detection routine is doing under the hood. The resulting $\text{RSS}(f)$ as a function of $f$ is, up to a constant, the negative of the periodogram.

(Multiple Frequencies) Either pre-specify a set of candidates or use sparse regularization.

When several frequencies may be present, the design grows to one constant plus two trigonometric columns per candidate frequency. Two strategies emerge. If the candidate frequencies are known (e.g., monthly seasonal harmonics at $f=1/12,2/12,\dots ,6/12$), fit the full design by OLS. If the candidate set is large and only a few are truly active, lasso or group lasso is the right tool: it sets the inactive frequency pairs to exactly zero and recovers a sparse representation. This is the regression-side counterpart to peak-picking on the periodogram and gives more stable estimates of the active frequencies’ amplitudes.

Contents

• 2.4.3 Time-Lagged Regression: distributed-lag (FIR) regression for identifying impulse responses between two series, with applications to neuroscience (STRF), economics, and audio.
• 2.4.5 Time-Series Cross-Validation: rolling-origin and fixed-window CV schemes that respect chronology, the right replacement for random $k$-fold on time-series data.

The general linear-regression machinery (OLS solution, residual variance, $R^2$, Gauss-Markov, ridge, lasso, ordinary cross-validation) lives in the machine-learning regression folder: 1.2. Linear Regression, 1.2a. Multiple Linear Regression, 1.4. Nonlinear Regression, 1.5. Regularization, 1.6. Cross-Validation. Reach for those notes when the question is about regression mechanics; reach for the notes in this folder when the question is about time-series-specific features (sinusoidal models, Nyquist, distributed lags, rolling-origin CV).

Examples

Sinusoidal fit at an unknown frequency

A signal $y_t$ is observed for $n=200$ time points and is believed to contain a single sinusoidal component at unknown frequency $f\in (0,0.5)$.

Estimation procedure $\mathcal{G}=\{0.005,0.010,\dots ,0.495\}$. For each $f\in \mathcal{G}$: build the design matrix ${\mathbf{X}}_f$, compute ${\hat{\mathbf{\beta }}}_f=({\mathbf{X}}_f^{\top }{\mathbf{X}}_f{)}^{-1}{\mathbf{X}}_f^{\top }\mathbf{y}$ and $\text{RSS}(f)=\Vert \mathbf{y}-{\mathbf{X}}_f{\hat{\mathbf{\beta }}}_f{\Vert }^2$. Plot $\text{RSS}(f)$ versus $f$; the minimum is the estimated frequency. Recover $\hat{R}=\sqrt{{\hat{\beta }}_1^2+{\hat{\beta }}_2^2}$ and $\hat{\varphi }=\arctan (-{\hat{\beta }}_2/{\hat{\beta }}_1)$. The grid spacing controls how finely $\hat{f}$ is resolved; refine the grid near the minimum for higher precision. The whole procedure is mathematically equivalent to peak-picking on the periodogram.

Set up a frequency grid

Trend plus seasonal dummies

Monthly airline passenger counts $y_t$ over $n$ years exhibit both an upward trend and strong yearly seasonality.

Model specification $y_t={\beta }_0+{\beta }_{1}t+{\sum }_{j=2}^{12}{\alpha }_{j}1\{m(t)=j\}+w_t,$ with $m(t)$ the month of observation $t$ and the constraint that one month is the baseline (absorbed into ${\beta }_0$). Fit by OLS using multiple linear regression. The fitted values ${\hat{y}}_t$ capture trend and seasonality; the residuals ${\hat{w}}_t$ are the seasonally adjusted series. Inspect their ACF: persistent serial correlation usually means the residuals carry an ARIMA-style structure that needs modeling separately. Use HAC standard errors (Newey-West) or refit by GLS with the estimated noise covariance for valid inference on ${\hat{\beta }}_1$.

Spurious regression on two random walks

Two independent random walks $X_t=X_{t-1}+{\epsilon }_t$ and $Y_t=Y_{t-1}+{\eta }_t$ are simulated. A linear regression of $Y$ on $X$ is fit.

What goes wrong $R^2$ (often $>0.5$) and a slope coefficient with apparently small $p$-value, suggesting strong predictive power. The series are statistically independent: there is no real relationship. The pathology is that both series are nonstationary (they wander), and any pair of wandering series will appear correlated by chance over a finite window, with the appearance growing more striking as $n$ increases (the opposite of the usual law of large numbers). The remedies are to difference both series before regressing ($\Delta Y_t$ on $\Delta X_t$ becomes a sensible regression of stationary noise on stationary noise) or to test the regression residuals for stationarity (the Engle-Granger cointegration test) before trusting the coefficient.

The fit typically gives a large

See Also

• 1.2. Linear Regression: the simple-regression cheat sheet, including OLS estimators and Gauss-Markov.
• 1.2a. Multiple Linear Regression: matrix-form OLS with all the details on the design matrix, hat matrix, and residual variance.
• 1.4. Nonlinear Regression: Gauss-Newton and profile-least-squares for models nonlinear in the parameters.
• 1.5. Regularization: ridge, lasso, and elastic net for wide designs (including many candidate frequencies or many lags).
• 1.6. Cross-Validation: the i.i.d.-data CV machinery that this folder’s 2.4.5 adapts to chronological order.
• 2.3.2 Periodogram: the spectral counterpart of sinusoidal regression at every Fourier frequency.
• 2.4.3 Time-Lagged Regression: the distributed-lag specialization for input-output regression on two time series.
• 2.4.5 Time-Series Cross-Validation: rolling-origin CV that replaces random $k$-fold on time-series data.
• 2.5 ARIMA Models: the dynamical-system counterpart to time-series regression.