2.1. Time Series

Time Series

A time series is an ordered sequence of observations indexed by time, modeled as a stochastic process
$\{X_t{\}}_{t\in T},T\subseteq \mathbb{Z}\text{ or }\mathbb{R},$
where each realization produces observed data $x_1,x_2,\dots ,x_T$. Two features distinguish the analysis from cross-sectional statistics: the temporal ordering carries information that shuffling would destroy, and the observations are not assumed independent. Both features mean that the analyst typically observes only one realization of the process, so the entire inference depends on assumptions linking different time points to each other.

The two primary objects characterizing the process are the mean function ${\mu }_t=\mathbb{E}[X_t]$ and the autocovariance function $\gamma (s,t)=\mathrm{Cov}(X_s,X_t)$. The mean function captures any systematic time-varying level (a trend), and the autocovariance captures dependence across time points. Many real series additionally exhibit seasonality, systematic repetition at fixed period $\tau$ ($X_t\approx X_{t+\tau }$ in distribution), which adds a periodic component on top of the trend.

The single most important assumption in time-series analysis is stationarity: that the probabilistic structure of the process does not depend on absolute time. Without some form of stationarity, every time point would have its own mean and dependence structure, and one observed path would not be enough to estimate anything. The applied default is weak stationarity (constant mean and lag-only autocovariance), which is the minimal assumption needed for the ACF and sample autocovariance to be well-defined. The fully general definition (and the strict-vs-weak distinction) lives in the probability treatment.

The standard non-stationary baseline is the random walk $x_t=x_{t-1}+w_t$, whose variance grows linearly with $t$ and whose autocovariance depends on $\min (s,t)$ rather than only on the lag. Differencing the random walk once recovers white noise. This is the algebraic core of the “I” in ARIMA and the reason non-stationary series in practice are usually handled by differencing them until what remains is stationary.

Intuition

A time series is fundamentally different from a cross-sectional sample because the ordering matters. Shuffling the rows of a regression dataset changes nothing; shuffling a time series destroys its structure. The core challenge is that we typically observe only one realization of the underlying stochastic process: one path out of the infinitely many that could have occurred. To learn anything about the process from a single path, the analyst needs assumptions that link different time points to each other so that observations at one $t$ can be used to estimate something about the process at another. Stationarity (the rules of the process are the same at every $t$) and ergodicity (the time average over one path equals the ensemble average) are the two assumptions that do that work.

The structure that makes a time series interesting comes from departures from independence. Trend makes the mean drift over time. Seasonality makes the mean (or variance) repeat on a known period. Serial dependence makes consecutive values correlated even after trend and seasonality are removed. Most of time-series analysis is the disciplined business of decomposing an observed series into these three layers and modeling each one: fitting a deterministic or differencing-based handle for the trend, sinusoids or seasonal differencing for the seasonality, and an ARMA-type or state-space model for the residual dependence.

Properties

(White Noise) The simplest stationary process (uncorrelated, mean-zero, constant-variance) and the building block for richer models.

A time series $\{W_t\}\sim \mathrm{WN}(0,{\sigma }^2)$ is white noise when its values are uncorrelated, mean-zero, and constant variance:
$\mathbb{E}[W_t]=0,\mathrm{Var}(W_t)={\sigma }^2,{\gamma }_W(h)=\{\begin{array}{ll}{\sigma }^2& h=0\\ 0& h\ne 0.\end{array}$
White noise has no memory at all: knowing $W_{t-1}$ tells you nothing about $W_t$. It is the canonical innovation driving more structured processes: an MA process is a sliding window over white noise, an AR process feeds white noise through a recursive filter, and ARMA does both. Any stationary process with a rational spectral density admits an ARMA representation driven by white noise, which is why so much of classical time-series modeling reduces to “find the right linear filter on white noise.”

(Ergodicity) Time averages converge to ensemble averages.

A stationary process $\{X_t\}$ is ergodic in the mean when the running time average converges in probability to the unconditional mean,
${\bar{X}}_T=\frac{1}{T}{\sum }_{t=1}^T{X}_t\stackrel{p}{\to }\mu =\mathbb{E}[X_t],$
as $T\to \infty$. A sufficient condition is that the autocovariance decays to zero, $\gamma (h)\to 0$, which holds for every causal ARMA process. Without ergodicity, the single observed realization cannot in general be used to estimate the population mean (the realization could persist forever in some atypical region), so ergodicity is what bridges one path to one population in time-series inference. The detailed treatment of stationarity (weak vs. strict, when each applies) lives at 4.7 Stationarity.

(Trend and Seasonality as Decomposable Components) Most real series can be written as $X_t={\mu }_t+s_t+Y_t$, where ${\mu }_t$ is a slow trend, $s_t$ is a periodic seasonal, and $Y_t$ is the stationary residual.

The decomposition is not unique without additional assumptions, but the workflow is standardized. Estimate the trend by smoothing or polynomial fitting; estimate the seasonal by averaging over the period; subtract both to leave $Y_t$, then model $Y_t$ as ARMA, ARIMA, or a state-space process. Alternatively, differencing removes both trend and seasonality in one step: $\nabla$ kills polynomial trends and ${\nabla }_{\tau }=1-B^{\tau }$ kills period-$\tau$ seasonality.

(One Realization, Many Time Points) The fundamental statistical predicament of time-series analysis.

Unlike cross-sectional data, where a sample of size $n$ comes from $n$ independent draws from the underlying distribution, time-series data comes as one path of length $T$. Estimating the mean, autocovariance, or any other population quantity requires assumptions (stationarity, ergodicity) that allow time-averaging to substitute for ensemble-averaging. This is the conceptual reason every later piece of the framework, from sample autocovariance estimators, to bootstrap variants for time series, to the entire time-series cross-validation machinery, looks different from its i.i.d. counterpart.

Examples

Random walk vs. white noise: a stationarity stress test

You observe a series and want to know whether to model it directly or to difference it first. What’s the diagnostic?

Plot the series, the sample ACF, and the differenced sample ACF. A stationary process (e.g., white noise or an ARMA fit) gives a sample ACF that decays quickly to inside the $\pm 1.96/\sqrt{T}$ band. A random walk-like process gives an ACF that stays close to one for many lags before slowly trailing off, the signature of an integrated process. After one differencing, the random walk becomes white noise (flat ACF), confirming the model choice. If the differenced ACF still shows structure, the original series is more complex than a pure random walk and may need ARIMA(p, 1, q) with non-trivial $p$ or $q$.

Trend plus stationary noise

Annual global temperature anomalies $y_t$ are modeled as a linear trend plus stationary residual: $y_t={\beta }_0+{\beta }_{1}t+{\epsilon }_t$ with ${\epsilon }_t$ a stationary AR(1).

The mean function ${\mu }_t={\beta }_0+{\beta }_{1}t$ depends on $t$, so $\{y_t\}$ is not stationary. Two equivalent paths to a stationary residual: regress $y_t$ on $t$ (recovering the slope ${\beta }_1$) and model the residuals as AR(1), or difference once ($\nabla y_t={\beta }_1+\nabla {\epsilon }_t$) and fit ARIMA(1, 1, 0) with drift. Both give the same long-run forecast trajectory; they differ in their handling of the trend uncertainty (the regression treats the slope as deterministic; the differencing treats it as a constant drift in a stochastic-trend model).