AR(p) Process

AR(p) Process

An autoregressive process of order $p$ describes a time series $\{X_t\}$ whose value at any time is a linear regression of its own $p$ most recent past values plus a random shock. The name is literal: auto (self) plus regression (linear prediction), so an AR model regresses the series on lagged copies of itself. In equation form,
$X_t={\varphi }_1{X}_{t-1}+{\varphi }_2{X}_{t-2}+\cdots +{\varphi }_p{X}_{t-p}+W_t,$
where each coefficient ${\varphi }_j$ is a fixed real constant, ${\varphi }_p\ne 0$ (otherwise the order would actually be smaller than $p$), and $\{W_t\}\sim \text{WN}(0,{\sigma }_w^2)$ is a white-noise shock: an unpredictable disturbance, uncorrelated across time and with the past of $X$. The same equation written through the backshift operator becomes the more compact $\varphi (B)X_t=W_t$ with $\varphi (B)=1-{\varphi }_{1}B-\cdots -{\varphi }_p{B}^p$.

Whether or not the equation describes a sensible (stationary) time series depends on the coefficients. The condition is that all roots of the polynomial $\varphi (z)=1-{\varphi }_{1}z-\cdots -{\varphi }_p{z}^p$ must lie strictly outside the unit circle in the complex plane; on this condition $\{X_t\}$ has constant mean, finite variance, and an autocovariance that depends only on lag, and the recursion can be unrolled into a convergent expression in past shocks (see 2.5.5 Causality and Invertibility).

The first-order case AR(1), $X_t=\varphi X_{t-1}+W_t$, is the cleanest example. With $|\varphi |<1$ and zero mean, the variance, autocovariance, and autocorrelation collapse to $\gamma (0)={\sigma }_w^2/(1-{\varphi }^2)$, $\gamma (h)={\varphi }^{|h|}\gamma (0)$, and $\rho (h)={\varphi }^{|h|}$. The variance grows toward infinity as $|\varphi |$ approaches $1$, which is the algebraic form of the stationarity condition $|\varphi |<1$. Once $\varphi$ reaches $\pm 1$ the formula breaks down because the process accumulates past shocks indefinitely instead of forgetting them.

The data signature of an AR($p$) is unmistakable: the autocorrelation decays smoothly toward zero (geometrically when the roots are real, with damped oscillation when complex), and the partial autocorrelation is exactly zero past lag $p$. This combination, a slow tail in the ACF and a clean cutoff at lag $p$ in the PACF, is what allows the order to be read straight off two diagnostic plots.

Intuition

An AR process is a feedback loop. Each new value is a re-scaled echo of recent values plus a fresh, unforecastable shock. The coefficient $\varphi$ in the AR(1) controls how loud that echo is. When $\varphi$ is positive and close to $1$, today’s value is almost the same as yesterday’s, with only a small new shock added: the series drifts slowly, takes long excursions away from its mean, and “remembers” past disturbances for many time steps. As $\varphi$ shrinks toward $0$, the echo fades and the series collapses toward white noise: the past tells you nothing, and each value is just the current shock.

The phrase mean-reverting describes what happens between any two consecutive steps when the long-run mean is zero and $0<\varphi <1$. Take the conditional expectation $\mathbb{E}[X_t\mid X_{t-1}]=\varphi X_{t-1}$. If $X_{t-1}$ sits a distance $d$ above zero, the expected next value is only $\varphi d$ above zero, closer to the mean by a factor of $1-\varphi$. Each step pulls the series partway back toward its long-run level, which is why values that wander far from the mean tend to walk back. With $\varphi$ negative the same shrinkage happens, but with a sign flip, so the series alternates above and below zero in a saw-tooth pattern.

Once $p\ge 2$, the dynamics split into two regimes determined by the roots of the polynomial $\varphi (z)$. When the roots are real and outside the unit circle, the autocovariance is a sum of decaying exponentials and the process slides smoothly back to its mean (overdamped in the language of mechanical oscillators). When the roots are complex conjugates, the autocovariance is a product of an exponential decay and a cosine, and realizations ring at a characteristic frequency before dying out. The location of the roots in the complex plane therefore translates directly into observable behavior: how persistent the memory is, whether and at what rate the series oscillates, how sharply concentrated the energy is in any one frequency band.

Properties

(Stationarity Condition) All roots of $\varphi (z)$ outside the unit circle.

The series has a constant mean and a lag-only autocovariance (the two conditions for weak stationarity) exactly when every solution $z$ of $1-{\varphi }_{1}z-\cdots -{\varphi }_p{z}^p=0$ satisfies $|z|>1$. For AR(1), $\varphi (z)=1-\varphi z$ has the single root $z=1/\varphi$, so the condition reduces to $|\varphi |<1$. For AR(2), the same condition translates to the parameter region ${\varphi }_2+{\varphi }_1<1$, ${\varphi }_2-{\varphi }_1<1$, and $|{\varphi }_2|<1$, an open triangle in the $({\varphi }_1,{\varphi }_2)$-plane on whose boundary the variance and autocovariance formulas blow up.

(MA($\infty$) Representation) A stationary AR($p$) writes today’s value as a convergent infinite sum of past shocks.

Inverting the operator polynomial gives $X_t=\varphi (B{)}^{-1}{W}_t={\sum }_{j=0}^{\infty }{\psi }_j{W}_{t-j}$, where the weights ${\psi }_j$ are determined by the algebraic identity $\varphi (B)\psi (B)=1$ and satisfy ${\sum }_j|{\psi }_j|<\infty$. For AR(1) the weights work out to the simple geometric series ${\psi }_j={\varphi }^j$, so $X_t=W_t+\varphi W_{t-1}+{\varphi }^2{W}_{t-2}+\cdots$. This expression makes explicit that today’s value is an exponentially weighted average of all past shocks, and that the influence of any single shock decays geometrically with rate $\varphi$, which is the algebraic source of the autocorrelation $\rho (h)={\varphi }^{|h|}$.

(Yule-Walker Equations) The autocovariances satisfy a linear recursion at every lag.

Multiplying the defining equation $\varphi (B)X_t=W_t$ by $X_{t-h}$ and taking expectations gives
$\gamma (h)-{\varphi }_1\gamma (h-1)-\cdots -{\varphi }_p\gamma (h-p)=\{\begin{array}{ll}{\sigma }_w^2& h=0\\ 0& h\ge 1,\end{array}$
because $W_t$ is uncorrelated with everything in the past. The equations at $h=0,1,\dots ,p$ form a linear system in the unknowns $\gamma (0),\gamma (1),\dots ,\gamma (p)$ and ${\sigma }_w^2$, and solving the system gives the autocovariance directly from the AR coefficients. Reading the system in the other direction (substituting sample autocovariances on the left and solving for the ${\varphi }_j$) produces the Yule-Walker estimator, the method-of-moments fit that is often used as a starting point for maximum-likelihood estimation.

(ACF Tails Off) The autocorrelation $\rho (h)$ never reaches exact zero.

By the Yule-Walker recursion, $\rho (h)$ at large lags is a fixed linear combination of $\rho (h-1),\dots ,\rho (h-p)$ with the same coefficients as the AR equation itself. The general solution is a sum of terms of the form $z_i^{-h}$ where $z_i$ are the AR roots, so the tail decays geometrically (real roots) or as a decaying cosine (complex conjugate roots), but is non-zero at every finite lag. This continual non-vanishing of $\rho (h)$ is the structural opposite of the moving-average case, whose autocorrelation reaches exact zero past lag $q$.

(PACF Cuts Off at Lag $p$) The partial autocorrelation ${\varphi }_{hh}$ is exactly zero for every $h>p$.

The partial autocorrelation at lag $h$ measures the correlation between $X_t$ and $X_{t-h}$ after the linear effect of the intermediate values $X_{t-1},\dots ,X_{t-h+1}$ has been removed. Because the AR($p$) equation says exactly that those intermediate values, plus the shock $W_t$, are everything that matters for $X_t$, no additional past observation can add predictive information beyond lag $p$, and the corresponding partial correlation vanishes. This sharp cutoff is the diagnostic that identifies the AR order from data: a sample PACF whose only significant entries are at lags $1,\dots ,p$ points unambiguously at AR($p$).

(Spectral Density) The frequency-domain shape of an AR process is the reciprocal of a polynomial.

By the linear-filter identity (see 2.3.4 Linear Filters and Frequency Response), the spectral density is
$f_X(\omega )=\frac{{\sigma }_w^2}{|\varphi (e^{-2\pi i\omega }){|}^2},$
the variance of the white-noise driver divided by the squared magnitude of the AR polynomial evaluated on the unit circle. This rational form is what gives AR spectra their characteristic peaks: wherever $\varphi (e^{-2\pi i\omega })$ comes close to zero (a pole of the spectrum), the density spikes upward, and the closer the corresponding root is to the unit circle, the sharper the spike. For AR(1) with positive $\varphi$ the peak sits at $\omega =0$, capturing the slow, long-memory drift; with negative $\varphi$ it sits at $\omega =1/2$, capturing the rapid alternation. For AR(2) with complex roots at angle $\theta$, the peak sits near $\omega =\theta /(2\pi )$, the natural ringing frequency.

Examples

AR(1) with $\varphi =0.9$

What does the variance, the lag-5 autocorrelation, and a typical realization look like?

Solution $\gamma (0)={\sigma }_w^2/(1-0.81)\approx 5.26{\sigma }_w^2$, more than five times the variance of the white-noise shocks alone. The interpretation is that today's value is essentially the sum of the recent few dozen shocks (the geometric tail ${\varphi }^j$ has half-life around $\log 2/|\log \varphi |\approx 6.6$ steps), so the standard deviation of the series is much larger than that of one shock. The lag-5 autocorrelation is $\rho (5)=0.9^5\approx 0.59$, still substantial: even five steps apart the values share most of their variation. Realizations look like long, gentle drifts above and below zero with no preferred direction, qualitatively similar to a random walk over short stretches but eventually returning toward the mean.

The variance is

AR(2) with complex roots

For $X_t=0.6X_{t-1}-0.5X_{t-2}+W_t$, locate the roots of the AR polynomial, decide whether the process is stationary, and estimate the natural ringing frequency.

Solution $1-0.6z+0.5z^2=0$ has roots $z=(0.6\pm \sqrt{0.36-2})/1=0.6\pm \sqrt{-1.64}/1=0.6\pm 1.28i$ from the quadratic formula. The modulus is $|z|=\sqrt{0.36+1.64}=\sqrt{2}\approx 1.41$, comfortably greater than $1$, so all roots lie outside the unit circle and the process is stationary. The argument of the root is $\arg z=\arctan (1.28/0.6)\approx 1.13$ radians, and the autocovariance therefore oscillates with period $2\pi /1.13\approx 5.6$ time steps. Realizations should show a faintly oscillatory pattern at that period, with the oscillation amplitude decaying at a rate set by the modulus: closer to $1$ would mean longer-lasting ringing, closer to infinity (impossible here) would mean immediate damping.

The polynomial

Yule-Walker estimation of AR(2)

Given sample autocovariances $\hat{\gamma }(0),\hat{\gamma }(1),\hat{\gamma }(2)$ from data believed to be AR(2), recover the coefficients ${\varphi }_1,{\varphi }_2$.

Solution $h=1$ and $h=2$ gives the linear system $\hat{\gamma }(1)={\varphi }_1\hat{\gamma }(0)+{\varphi }_2\hat{\gamma }(1),\hat{\gamma }(2)={\varphi }_1\hat{\gamma }(1)+{\varphi }_2\hat{\gamma }(0),$ which can be stacked into the matrix form $\left(\begin{array}{cc}\hat{\gamma }(0)& \hat{\gamma }(1)\\ \hat{\gamma }(1)& \hat{\gamma }(0)\end{array}\right)\left(\begin{array}{c}{\varphi }_1\\ {\varphi }_2\end{array}\right)=\left(\begin{array}{c}\hat{\gamma }(1)\\ \hat{\gamma }(2)\end{array}\right).$ Inverting the $2\times 2$ Toeplitz matrix on the left yields explicit formulas for the estimates ${\hat{\varphi }}_1,{\hat{\varphi }}_2$. These are the method-of-moments coefficients; software (R arima, statsmodels) typically uses them as starting values for a subsequent maximum-likelihood refinement, which slightly corrects for the small-sample bias of the Yule-Walker estimates.

Substituting into the Yule-Walker equations at lags