Time-Series Cross-Validation

Time-Series Cross-Validation

Time-series cross-validation is the family of model-selection schemes that respect the temporal ordering of the data: every validation split must put the validation observations after the training observations, so that the model never gets to “see the future” when scoring its predictions. The standard scheme is the rolling-origin or expanding-window split: train on observations $\{1,2,\dots ,t\}$, evaluate the forecast on the next $h$ observations $\{t+1,\dots ,t+h\}$, then advance the cut $t\to t+1$ and repeat. The cross-validated error is the average forecast error across the resulting $T-t_0$ rolling windows. This is the time-series-respecting variant of the standard i.i.d. $k$-fold CV from the machine-learning toolbox.

Random k-fold cross-validation does not work for time series. Shuffling observations across time mixes future data into the training set and produces wildly optimistic CV scores: a model that “knows” what comes after a held-out validation point can use that information to fit it almost perfectly. The CV error then no longer estimates real forecast performance. The pathology is most extreme when the series has a strong trend (the future training observations encode the trend’s location) but applies to any serially correlated series.

Two practical variants of the scheme:

The expanding-window version keeps every past observation in the training set as $t$ advances, so the training set grows from $t_0$ up to $T-h$. This is the right default when the data-generating process is stable.

The fixed-window (or sliding-window) version drops old observations as new ones come in, keeping a constant training window of length $L$. This is the right default when the data-generating process drifts slowly over time and old observations would mislead the model.

Both schemes are implemented out of the box: sklearn.model_selection.TimeSeriesSplit does expanding-window splits with a configurable test size, and a few lines of manual indexing produces the fixed-window variant. The same scheme drives hyperparameter selection (e.g., the ridge $\lambda$ in regularized regression) and model comparison (e.g., choosing among ARIMA orders).

The Random-Shuffle Trap cross_val_score(model, X, y, cv=5) with default settings randomly shuffles observations into folds. On a trended series this produces CV errors that look excellent but predict nothing about real forecast performance: the model has effectively memorized the trend's level using future data. The fix is mechanical: pass cv=TimeSeriesSplit(5) instead. Make this substitution in every CV call on time-series data, including for choosing $\lambda$ in ridge or lasso fits, choosing $(p,d,q)$ in ARIMA selection, or comparing forecast methods.

Calling

Intuition

The point of cross-validation is to estimate how well a model will perform on data it has not seen. For i.i.d. data, randomly partitioning observations into folds works because shuffling the data does not change anything about the underlying problem: every observation is exchangeable with every other. For time-series data, observations are explicitly not exchangeable: an observation at time $t$ is more similar to its neighbors at $t-1$ and $t+1$ than to a randomly chosen observation, and a model fit on observations after $t$ has access to information that would not be available at the moment of forecasting $t$. Both of these break the exchangeability assumption that random k-fold relies on.

The fix is to enforce that every validation split mirrors the actual forecasting situation: train on the past, predict the future. Rolling-origin CV simulates this exactly. The “validation error” produced by averaging across many such splits is then a real forecast error estimate, not the inflated number that random shuffling gives. The cost is that the splits are no longer independent (successive expanding-window training sets share most of their observations), so the variance of the CV estimate is larger and harder to characterize than in the i.i.d. case. But the bias is correct, and that matters more than the variance for model selection.

Properties

(Expanding-Window Default) Train on $\{1,\dots ,t\}$, validate on $\{t+1,\dots ,t+h\}$, advance $t\to t+1$.

The full procedure produces $T-t_0-h+1$ training-validation pairs, where $t_0$ is the smallest training-set size deemed adequate. The validation horizon $h$ is set to match the actual forecast horizon of interest: choose $h=1$ for one-step-ahead model evaluation, $h=12$ for monthly data forecasted a year out, and so on. The CV error is the average of the validation errors across the windows. Because the training sets grow over time, the later windows have more training data than the earlier ones, which is the right behavior when more data is genuinely better, but a problem when the underlying process has changed (then the early windows give better models than the late ones).

(Fixed-Window Variant) Keep the training set at constant length $L$ instead of letting it expand.

The training set rolls forward as a window of length $L$: train on $\{t-L+1,\dots ,t\}$, validate on $\{t+1,\dots ,t+h\}$, advance. Older observations are dropped, so the model is always fit on the most recent $L$ points. Use this when the data-generating process drifts (e.g., financial volatility regimes, structural breaks in macroeconomic data) and stale observations would harm the fit more than they help. For stable processes, the expanding-window version uses more data and gives lower-variance estimates.

(Direct Multi-Step vs. Recursive) Two ways to evaluate a forecast at horizon $h>1$.

The direct approach trains a model to predict $h$ steps ahead in one shot, so the validation target is $X_{t+h}$ given $\{X_1,\dots ,X_t\}$. The recursive approach trains a one-step-ahead model and applies it $h$ times, plugging each forecast back in as the next input. Direct is simpler to evaluate but requires retraining for each horizon; recursive uses a single model but accumulates compounding error. Both are valid CV schemes; for ARMA models the recursive route is the default because the model itself is a one-step-ahead recursion.

(Choosing the Window Count) Trade between training data and validation reliability.

Each rolling window’s validation error is noisy (often just $h$ observations), so the CV estimate’s reliability comes from averaging many windows. The standard recipe: pick $t_0$ large enough to fit the model honestly (typically $\ge 10$ times the number of model parameters), pick $h$ to match the operational forecast horizon, and let the rolling produce as many windows as the data permit. With $T=1000$ daily observations and $t_0=200,h=30$, the procedure yields about 770 windows, plenty for a stable estimate.

(Independence of Successive Windows is Violated) Training sets in adjacent windows differ by one observation, so the validation errors are highly correlated.

The CV estimate is still unbiased for the average forecast error, but its standard error is not what i.i.d. CV would give. Constructing honest confidence intervals for the CV estimate requires the moving-block bootstrap or similar dependence-aware resampling. In practice most analysts skip the confidence interval and just report the CV mean, then check robustness by varying the rolling-window settings (different $t_0$, different $h$, expanding vs. fixed).

(Compatible with Information Criteria) AIC and BIC remain useful but should not replace CV.

For nested model families (e.g., choosing ARIMA orders), AIC and BIC give a quick analytic alternative to CV. They are based on in-sample log-likelihood penalized by parameter count and do not require holding any data out. They work well when the model assumptions hold (Gaussian residuals, correctly specified family). When the assumptions might fail (non-Gaussian errors, non-nested models, regularized fits with effective degrees of freedom), CV is the safer choice because it makes no assumption about the noise distribution. Many production pipelines use both and flag disagreement.

Examples

Choosing $\lambda$ for a ridge regression on a trended series

You’re fitting a ridge regression on a wide design matrix (many lagged predictors), and you want to choose the regularization strength $\lambda$ by cross-validation.

The right and the wrong way Wrong: RidgeCV(alphas=alphas, cv=5) defaults to k-fold, randomly shuffles observations, and gives an overly small $\hat{\lambda }$.

Right: RidgeCV(alphas=alphas, cv=TimeSeriesSplit(5)) uses expanding-window splits, gives an honest estimate of forecast error at each $\lambda$, and picks a $\hat{\lambda }$ that actually generalizes. The two choices can differ by an order of magnitude on a trended series, and the random-fold estimate produces forecasts that look great in CV but fail catastrophically out of sample.

Comparing ARIMA orders on weekly gas prices

You have $T=500$ weeks of gas-price data and want to compare ARIMA(1, 0, 0), ARIMA(2, 0, 0), and ARIMA(1, 0, 1) for one-step-ahead forecasting.

Procedure $t_0=100$ and $h=1$, giving $400$ rolling windows. For each window: fit each candidate model on the first $t$ observations, forecast ${\hat{X}}_{t+1}$, record the squared error $(X_{t+1}-{\hat{X}}_{t+1}{)}^2$. Average across all windows to get the CV mean squared error for each model. Pick the model with the smallest CV MSE; report the standard error across windows for sanity. If two models tie within standard error, prefer the simpler one. This is the same workflow used in the lecture's gas-price example, and it generalizes immediately to longer forecast horizons by setting $h$ accordingly.

Set

When fixed-window beats expanding-window

You’re modeling daily electricity demand, and you suspect the consumption pattern has changed since smart-meter rollout three years ago.

Why fixed-window is better here $L=730$ days (two years) drops the oldest observations as new ones come in, so the model is always fit on data from the current regime. The CV estimate then reflects forecast performance under the current generating process, not a blend of current and historical. The trade-off: with $L=730$ training points instead of growing to $L=1095$, model variance is slightly larger. For a slowly drifting process the bias reduction is worth the variance cost; for a stable process expanding-window wins.

An expanding-window CV would include the pre-rollout years in every training set, dragging the model toward an outdated pattern. A fixed window of length