ARMA(p,q) Process

ARMA(p,q) Process

An autoregressive moving-average process of orders $(p,q)$ combines the two memory mechanisms of AR and MA into a single model: today’s value is a linear combination of the $p$ most recent past values and the present and $q$ most recent shocks. The “AR” piece feeds past observations forward through a feedback loop, producing the long, geometrically decaying memory typical of autoregressive dynamics; the “MA” piece slides a fixed-length window over the recent shocks, adding a short-range, finite-memory component on top. In equation form,
$X_t={\varphi }_1{X}_{t-1}+\cdots +{\varphi }_p{X}_{t-p}+W_t+{\theta }_1{W}_{t-1}+\cdots +{\theta }_q{W}_{t-q},$
with $\{W_t\}\sim \text{WN}(0,{\sigma }_w^2)$ and the coefficients ${\varphi }_p,{\theta }_q\ne 0$ so that the orders are not secretly smaller than $p$ and $q$. Through the backshift operator the model collapses to the compact form $\varphi (B)X_t=\theta (B)W_t$ with $\varphi (B)=1-{\varphi }_{1}B-\cdots -{\varphi }_p{B}^p$ on the AR side and $\theta (B)=1+{\theta }_{1}B+\cdots +{\theta }_q{B}^q$ on the MA side. If a non-zero mean $\mu$ is desired, the equation is shifted to $\varphi (B)(X_t-\mu )=\theta (B)W_t$, equivalent to including an intercept $\alpha =\mu (1-{\varphi }_1-\cdots -{\varphi }_p)$ on the right.

The two boundary cases are the families ARMA reduces to. With $q=0$ only the AR side survives, recovering AR(p). With $p=0$ only the MA side survives, recovering MA(q). ARMA fills the gap between the two, useful when neither pure family alone fits the observed correlation pattern.

For the model to define a sensible (stationary, invertible) process, both polynomials must satisfy the unit-circle root conditions: roots of $\varphi (z)$ outside the unit circle for stationarity, roots of $\theta (z)$ outside for invertibility. The detailed reasoning behind both conditions lives in 2.5.5 Causality and Invertibility; software fitting routines automatically enforce both.

The data signature of ARMA, contrasted against pure AR and pure MA, is that the autocorrelation and the partial autocorrelation both tail off without exact zero. AR alone would give a clean PACF cutoff at lag $p$ and an ACF tail; MA alone would give a clean ACF cutoff at lag $q$ and a PACF tail. When both diagnostic functions tail off rather than cutting off cleanly, ARMA is the natural candidate, and the orders $(p,q)$ are typically chosen by trying small candidate values and comparing AIC or BIC across fits.

Common Factors and Parameter Redundancy $(1-\alpha B)$, dividing both sides of $\varphi (B)X_t=\theta (B)W_t$ by that factor reduces the apparent ARMA(p, q) to an ARMA(p-1, q-1) describing exactly the same process. The original parameterization is then unidentifiable: the data are consistent with a whole family of $(\varphi ,\theta )$ pairs that share the same reduced polynomials, and the likelihood surface is flat along that family. Standard software (arima in R, statsmodels.tsa.ARIMA in Python) flags near-cancellation between roots of $\varphi$ and $\theta$ as a warning; always inspect the fitted polynomial roots to make sure no two are nearly equal.

If the AR and MA polynomials share a common factor

Intuition

The reason ARMA, rather than pure AR or pure MA, is the workhorse of classical linear time-series modeling is parsimony. Wold’s decomposition guarantees that every stationary process can be written as an MA($\infty$) driven by white noise, but estimating infinitely many MA coefficients from finite data is impossible. ARMA approximates an MA($\infty$) by a rational function $\theta (z)/\varphi (z)$ of the backshift operator, which means a small handful of parameters can mimic the autocorrelation and spectrum of an infinite-order MA. An AR($p$) alone can also approximate any spectrum but typically needs much higher orders than an ARMA fit, because each AR pole adds a peak whereas each MA zero adds a notch; having both available makes the fit much more flexible.

Conceptually, the AR side gives the process its long-range structure: a single shock takes many time steps to propagate through the feedback loop, so its influence decays geometrically rather than disappearing all at once. The MA side adds a short-range, finite-memory bump on top: the shock’s direct contribution to $X_t,X_{t+1},\dots ,X_{t+q}$ rides above the geometric AR tail. The combination produces autocorrelation patterns that are more flexible than what either ingredient alone can manage, while still being controlled by only $p+q+1$ parameters.

Properties

(Equivalent MA($\infty$) and AR($\infty$) Representations) A causal, invertible ARMA(p,q) is equivalent to two convergent infinite-order representations.

Inverting the operator equation in two ways gives the dual forms
$X_t={\sum }_{j=0}^{\infty }{\psi }_j{W}_{t-j}\text{and}{W}_t=X_t-{\sum }_{j=1}^{\infty }{\pi }_j{X}_{t-j},$
the first an MA($\infty$) expressing today’s value entirely in terms of past white-noise shocks, the second an AR($\infty$) expressing today’s shock entirely in terms of past observations. The weights are determined by the polynomial identities $\theta (B)=\varphi (B)\psi (B)$ and $\varphi (B)=\theta (B)\pi (B)$, both of which can be solved recursively. The MA($\infty$) form is what makes the forecast easy to write down (one-step-ahead predictions and their variances reduce to finite truncations of the $\psi$-series), and the AR($\infty$) form is what makes innovations recoverable from past observations, which in turn powers the residual diagnostics.

(ACF and PACF Both Tail Off) For a causal, invertible ARMA(p, q), neither the autocorrelation nor the partial autocorrelation cuts off.

The MA($\infty$) representation means the autocorrelation depends on infinitely many shocks, so $\rho (h)$ never reaches exact zero. The AR($\infty$) representation means infinitely many past observations contribute meaningfully to the regression of $X_t$ on its past, so the partial autocorrelation also never cuts off. The structural consequence, and the reason the diagnostic table for the ARMA family always says “both tail off,” is that order identification has to fall back on information criteria rather than visual inspection of a cutoff lag.

(Mean Equation) The unconditional mean and the intercept are related by the AR coefficients.

If a constant $\alpha$ appears on the right side of the equation, taking expectations on both sides and using the constancy of the mean of a stationary process gives
$\mathbb{E}[X_t]=\mu =\frac{\alpha }{1-{\varphi }_1-\cdots -{\varphi }_p},$
provided the AR polynomial is invertible at $z=1$ (which the stationarity condition guarantees). The practical caveat is that many software libraries fit the centered equation $\varphi (B)(X_t-\bar{X})=\theta (B)W_t$ with $\bar{X}$ the sample mean, and the reported intercept is the centered one, not $\alpha$. Converting back to the unconditional mean requires the formula above.

(Spectral Density) The frequency-domain shape of an ARMA process is a rational function: a polynomial divided by a polynomial.

Combining the MA spectrum (a polynomial in $\cos$, contributing notches at MA zeros) with the AR spectrum (the reciprocal of a polynomial in $\cos$, contributing peaks at AR poles) gives the spectral density
$f_X(\omega )={\sigma }_w^2\frac{|\theta (e^{-2\pi i\omega }){|}^2}{|\varphi (e^{-2\pi i\omega }){|}^2}.$
The denominator’s polynomial-magnitude term creates peaks wherever $\varphi$ has roots near the unit circle, sharper as the roots approach the circle; the numerator’s polynomial-magnitude term creates notches wherever $\theta$ has roots near the unit circle. The resulting rational shape can mimic almost any spectrum with very few parameters, which is why estimating an ARMA model on a stationary series is closely related to estimating the locations of a small number of poles and zeros of the underlying spectrum.

(Yule-Walker Holds Past Lag $q$) The autocovariance satisfies a homogeneous AR-style recursion at lags larger than $q$.

Multiplying $\varphi (B)X_t=\theta (B)W_t$ by $X_{t-h}$ for $h\ge q+1$ and taking expectations kills every cross-product on the right (because the noise at times $t,t-1,\dots ,t-q$ is uncorrelated with $X_{t-h}$ for $h$ that large), leaving the homogeneous recursion
$\gamma (h)-{\varphi }_1\gamma (h-1)-\cdots -{\varphi }_p\gamma (h-p)=0,h\ge q+1.$
The implication is that the tail of the ARMA autocovariance, past lag $q$, behaves identically to the tail of a pure AR(p) autocovariance: the geometric decay or damped oscillation comes entirely from the AR roots. The MA part shows up only in the autocovariances at lags $0,1,\dots ,q$, where the cross-products with the noise are non-zero.

Examples

ARMA(1,1) explicit form

Write $X_t=0.5X_{t-1}+W_t+0.4W_{t-1}$ in operator form, verify that it is stationary and invertible, and find the first few weights of its MA($\infty$) representation.

Solution $(1-0.5B)X_t=(1+0.4B)W_t$, with AR root at $z=1/0.5=2$ (outside the unit circle, so stationary) and MA root at $z=-1/0.4=-2.5$ (also outside, so invertible). The MA($\infty$) weights satisfy $\theta (B)=\varphi (B)\psi (B)$, i.e. $1+0.4B=(1-0.5B)({\psi }_0+{\psi }_{1}B+{\psi }_2{B}^2+\cdots )$. Matching coefficients of $B^0$ gives ${\psi }_0=1$; matching $B^1$ gives $0.4={\psi }_1-0.5{\psi }_0$, so ${\psi }_1=0.9$; for $j\ge 2$ matching $B^j$ gives $0={\psi }_j-0.5{\psi }_{j-1}$, so ${\psi }_j=0.5{\psi }_{j-1}=0.9\cdot 0.5^{j-1}$. The full representation is therefore $X_t=W_t+{\sum }_{j\ge 1}0.9\cdot 0.5^{j-1}{W}_{t-j}$, with weights that decay geometrically at rate $0.5$ (set by the AR root) but with a one-time bump in size from ${\psi }_0=1$ to ${\psi }_1=0.9$ that reflects the MA contribution.

The operator form is

When ACF and PACF both tail off

The sample ACF of a length-500 series tails off geometrically with a slight oscillation, and the sample PACF also tails off with significant entries at lags 1 and 2 and weak ones at lags 3 and 4.

What model do you fit? $q$), and the PACF's continued non-trivial values past lag 2 suggest that an MA contribution is also present (a pure AR(p) would force the PACF to zero at lag $p+1$ and beyond). Reasonable starting candidates are ARMA(1, 1), ARMA(2, 1), and ARMA(1, 2). Each fit produces a log-likelihood; comparing AIC or BIC across the three usually narrows the choice to one. After fitting, a Ljung-Box test on the residuals checks whether the chosen $(p,q)$ is high enough; significant residual autocorrelation means a higher order is needed.

Both diagnostic functions tail off rather than cutting off cleanly, so neither a pure AR nor a pure MA is a good fit. The ACF’s geometric tail is the structural sign of an AR contribution (the autocovariance follows the homogeneous recursion past lag

Common-factor pathology

What happens if you try to fit $X_t=0.5X_{t-1}+W_t-0.5W_{t-1}$?

Solution $(1-0.5B)X_t=(1-0.5B)W_t$, in which the same factor $(1-0.5B)$ appears on both sides and cancels, leaving $X_t=W_t$. So the apparent "ARMA(1, 1)" is simply white noise, with the parameters $\varphi =0.5$ and $\theta =-0.5$ artificially manufactured but unidentifiable from any data this process produces: the same observations are consistent with $\varphi =0.4$ and $\theta =-0.4$, or $\varphi =0$ and $\theta =0$, or any other matched pair. A software fit will return wildly varying coefficient estimates with huge standard errors; the inverted operator polynomial roots will be almost equal, the diagnostic that flags this redundancy. The fix is always to inspect the polynomial roots after fitting and, if two roots nearly cancel, to refit with a smaller order.

The operator form is