ARIMA(p,d,q) Process

ARIMA(p,d,q) Process

An autoregressive integrated moving-average process of orders $(p,d,q)$ is the natural extension of ARMA(p, q) to time series whose mean drifts over time and whose first or second differences are needed to bring them to stationarity. The “I” stands for integrated, meaning that the original series is the $d$-fold cumulative sum (the discrete integral) of an underlying stationary ARMA process. In practice, fitting goes the other way around, by differencing the observed series $d$ times to produce a stationary residual that is then modeled as ordinary ARMA. A series $\{X_t\}$ is therefore ARIMA(p, d, q) when its $d$-th difference is stationary ARMA(p, q),
${\nabla }^d{X}_t=(1-B{)}^d{X}_t\sim \text{ARMA}(p,q),$
or equivalently $\varphi (B)(1-B{)}^d{X}_t=\theta (B)W_t$, where $\varphi$ and $\theta$ satisfy the usual stationarity-of-differenced-process and invertibility root conditions and $\{W_t\}\sim \text{WN}(0,{\sigma }_w^2)$. If the differenced series is allowed to have a non-zero mean $\mu$, a drift term $\delta =\mu (1-{\varphi }_1-\cdots -{\varphi }_p)$ is added to give $\varphi (B)(1-B{)}^d{X}_t=\delta +\theta (B)W_t$.

The differencing order $d$ is almost always either $1$ or $2$. One difference removes a linear trend or a single random walk; two differences remove a quadratic trend or a random walk whose own slope is itself a random walk. Differencing more than twice is essentially never needed in practice and is generally a sign of over-differencing: differencing a series that was already stationary creates spurious correlation and leaves the fit struggling with a near-unit MA root, so $d$ should be chosen no larger than necessary. Standard practice is to test for non-stationarity with the Augmented Dickey-Fuller (ADF) test or the KPSS test before deciding on $d$.

Forecasting an ARIMA model is done in two stages: the differenced series is forecast by ordinary ARMA techniques, and the resulting forecasts are then integrated back to the original scale by repeated cumulative summation, propagating both the point forecasts and their variances. Because the differencing filter $\nabla =1-B$ has zero gain at frequency $\omega =0$ (see 2.3.4 Linear Filters and Frequency Response), it irreversibly removes any constant level from the series, and the absolute level of the forecast has to be reconstructed from the last observed value through the integration step rather than being predicted directly by the model.

Over-Differencing $X_t=W_t$ is white noise, then $\nabla X_t=W_t-W_{t-1}$ is an MA(1) whose polynomial $1-B$ has a root at exactly $z=1$, on the unit circle and therefore not invertible. A fitted ARIMA(0, 1, 1) on white noise will return $\hat{\theta }\approx -1$ with vanishing standard error, and the inverted recursion needed for forecasting will fail to converge. The rule is to difference until the series looks stationary in mean and variance and then stop, supported by an ADF or KPSS test. Default to $d=1$; only use $d=2$ when there is clear diagnostic support for a doubly-integrated trend.

Differencing a series that is already stationary adds spurious correlation and produces a non-invertible MA component. Concretely, if

Intuition

The conceptual move from ARMA to ARIMA is to handle non-stationarity by removing it before modeling. Instead of asking how to fit a non-stationary series directly, the analyst differences the series (subtracts each value from the next) until the result no longer drifts, and then fits an ordinary ARMA to the differenced version. The “integrated” name reflects the inverse step: undoing the differencing means cumulatively summing, the discrete analogue of integration. A pure random walk $X_t=X_{t-1}+W_t$, which has no preferred level and whose variance grows linearly with time, is the simplest example: differencing once recovers white noise, so the random walk is ARIMA(0, 1, 0), with no AR or MA structure beyond the integration itself.

The cost of differencing is a partial loss of low-frequency information. The differencing filter passes high-frequency variation through unchanged but kills the constant component entirely: its frequency response is $|1-e^{-2\pi i\omega }{|}^2=4{\sin }^2(\pi \omega )$, which is zero at $\omega =0$ and reaches its maximum of $4$ at $\omega =1/2$. This is exactly the desired behavior when removing a trend, but it also means the differenced model has nothing to say about the absolute level of the series, only about its changes. Forecasting back to absolute levels requires the integration step, and forecast variance grows much faster for ARIMA than for stationary ARMA: the variance of the $h$-step random-walk forecast grows linearly in $h$, while for a pure stationary ARMA it converges to a finite limit.

Properties

(Differencing Removes Polynomial Trends) For deterministic polynomial trends of degree $k$, $k$ differences make the series stationary.

If $X_t={\mu }_t+Y_t$ with ${\mu }_t={\sum }_{j=0}^k{\beta }_j{t}^j$ a degree-$k$ polynomial trend and $Y_t$ a stationary process, then applying the operator ${\nabla }^k=(1-B{)}^k$ annihilates polynomials of degree below $k$ entirely and reduces the degree-$k$ polynomial to a constant. Combined with the differencing of the stationary part (which remains stationary because differencing is a fixed linear filter on a stationary input), the result ${\nabla }^k{X}_t$ is stationary. The leading coefficient ${\beta }_k$ shows up as the constant mean of the differenced series, a drift term in the ARIMA fit.

(Differencing Removes Stochastic Trends) For random-walk trends, one difference suffices; for nested random walks, two differences are needed.

If $X_t={\mu }_t+Y_t$ where ${\mu }_t$ is itself a random walk (${\mu }_t={\mu }_{t-1}+V_t$ with $V_t$ stationary), then $\nabla X_t=V_t+\nabla Y_t$ is stationary because both pieces are. If instead ${\mu }_t$ is itself an integrated random walk (a random walk whose increments $V_t$ are themselves a random walk), then differencing once gives $\nabla X_t=V_t+\nabla Y_t$ in which $V_t$ is still non-stationary, and a second difference is required. Many real series, such as GDP, populations, and asset prices, behave more like random walks than like deterministic trends, and the same differencing operator handles both deterministic and stochastic trends without needing to commit to which type the series follows.

(Random Walk = ARIMA(0, 1, 0)) The simplest non-trivial ARIMA model is the random walk, and it is the benchmark every ARIMA fit should beat.

The random walk $X_t=X_{t-1}+W_t$ becomes white noise after one differencing, so it is ARIMA(0, 1, 0): no AR or MA structure beyond the integration. Its $h$-step-ahead forecast is just the last observed value, ${\hat{X}}_{t+h\mid t}=X_t$ for every $h$, with forecast variance $h{\sigma }_w^2$ growing linearly in the horizon. This is the “no information about the future” benchmark: any ARIMA model with $p>0$ or $q>0$ has to demonstrate out-of-sample accuracy beyond the random walk to justify its added parameters. For many financial-return series, the random walk turns out to be hard to beat.

(ARIMA(0, 1, 1) = Single Exponential Smoothing) The simplest ARIMA model with a non-trivial MA component is mathematically equivalent to single exponential smoothing.

The model $\nabla X_t=W_t+\theta W_{t-1}$, that is, ARIMA(0, 1, 1) with no AR component, turns out, after working through the recursion, to give one-step-ahead forecasts of the form ${\hat{X}}_{t+1}=\alpha X_t+(1-\alpha ){\hat{X}}_t$ with smoothing parameter $\alpha =1+\theta$ (and $-1<\theta <0$ corresponds to the standard $0<\alpha <1$ range). This connects classical exponential smoothing (Holt’s method, Brown’s method, and their seasonal extensions) to the ARIMA framework, and means that the family of seasonal ARIMA models like the famous “airline model” ARIMA$(0,1,1)\times (0,1,1{)}_{12}$ are the modern, general-purpose generalizations of the exponential-smoothing tools that dominated business forecasting before Box-Jenkins.

(Forecast Variance Grows with Horizon) The widening forecast cone is the structural cost of integration.

For a stationary ARMA model, the $h$-step forecast variance converges as $h\to \infty$ to the unconditional variance of the process, capping the size of the prediction interval at a finite value. For a non-stationary ARIMA model with $d\ge 1$, no such cap exists: the variance grows polynomially in the horizon. For ARIMA(0, 1, 0) it grows linearly as $h{\sigma }_w^2$; for general ARIMA(p, 1, q) it grows asymptotically as $Ch$ for some constant $C$ that depends on the AR and MA coefficients; for ARIMA(p, 2, q) it grows much faster, asymptotically as $C{h}^3/3$. The widening forecast cone reflects the fundamental forecastability limit of integrated processes: the further out you predict, the wider your uncertainty becomes, and beyond some horizon the forecast stops being informative.

(Over-Differencing Creates Non-Invertible MA Roots) The diagnostic signature of over-differencing is an MA(1) coefficient pinned at the boundary $-1$.

If $\{X_t\}$ is stationary and you difference it anyway, the resulting series $(1-B)X_t$ has the factor $(1-B)$ in its MA polynomial, which has root $z=1$ exactly on the unit circle. Software then fights to keep the MA coefficient at the boundary of invertibility, producing huge standard errors and warning messages. The simple diagnostic is to fit an MA(1) to the residual after the alleged differencing and check whether $\hat{\theta }\approx -1$; if it is, you have over-differenced, and the model should be refit with one fewer difference.

Examples

Linear-trend example

Show that $\nabla X_t$ is stationary when $X_t={\beta }_0+{\beta }_{1}t+Y_t$ with $Y_t$ a stationary AR(1).

Solution $\nabla X_t=X_t-X_{t-1}=({\beta }_0+{\beta }_{1}t+Y_t)-({\beta }_0+{\beta }_1(t-1)+Y_{t-1})={\beta }_1+(Y_t-Y_{t-1})={\beta }_1+\nabla Y_t.$ The constant ${\beta }_1$ is the slope of the trend, which becomes the drift of the differenced series; $\nabla Y_t$ is the differencing of a stationary process, which is itself stationary because differencing is a fixed linear filter. Fitting an ARIMA(p, 1, q) with drift to $X_t$ recovers the slope ${\beta }_1$ as the drift coefficient and the AR(1) parameter of $Y$ as the AR(1) parameter of the differenced series.

Computing the difference directly,

Two nested random walks need $d=2$

For ${\mu }_t={\mu }_{t-1}+V_t$ where $V_t$ itself follows $V_t=V_{t-1}+e_t$ with $e_t$ stationary, show that $\nabla {\mu }_t$ is non-stationary but ${\nabla }^2{\mu }_t$ is stationary.

Solution ${\mu }_t={\mu }_{t-1}+V_t$ comes $\nabla {\mu }_t=V_t$. But $V_t$ is itself a random walk, since $V_t=V_0+{\sum }_{s=1}^t{e}_s$, with variance $V(0)+t{\sigma }_e^2$ that grows linearly in $t$, so it is non-stationary. So one differencing leaves a series whose variance still depends on time. Differencing again, ${\nabla }^2{\mu }_t=\nabla V_t=e_t$, which is stationary by assumption. So ${\mu }_t$ is "integrated of order 2," conventionally written $I(2)$, and an ARIMA fit needs $d=2$ to bring it to stationarity.

From

Over-differencing white noise

What happens when you fit an ARIMA(0, 1, 1) to a series that is in fact pure white noise?

Solution $\nabla X_t=W_t-W_{t-1}$, with autocovariance $\gamma (0)=2,\gamma (1)=-1$, and zero at higher lags. This is an MA(1) whose autocorrelation $\rho (1)=-1/2$ implies a $\theta$ satisfying $\theta /(1+{\theta }^2)=-1/2$, which gives $\theta =-1$, exactly on the invertibility boundary. The MA polynomial $1-B$ has its root at $z=1$, on the unit circle. Fitting an ARIMA(0, 1, 1) to white noise will return $\hat{\theta }$ very close to $-1$ with a standard error close to zero, and any forecasting routine that requires invertibility will refuse to converge or will produce nonsensical forecasts. The correct model is ARIMA(0, 0, 0), no differencing at all.

Differencing white noise gives

Forecasting an ARIMA(0, 1, 1) model

For $\nabla X_t=W_t+0.5W_{t-1}$ with ${\sigma }_w^2=1$, with $X_T=100$ and the last computed shock ${\hat{W}}_T=0.3$, compute the one-step and two-step forecasts.

Solution $\nabla X_{T+1}=X_{T+1}-X_T=W_{T+1}+0.5W_T$, take expectation given the past: ${\hat{X}}_{T+1\mid T}=X_T+{\hat{W}}_{T+1\mid T}+0.5{\hat{W}}_T=100+0+0.5\cdot 0.3=100.15$. The two-step forecast does the same trick from the now-known ${\hat{X}}_{T+1}$: ${\hat{X}}_{T+2\mid T}={\hat{X}}_{T+1\mid T}+{\hat{W}}_{T+2\mid T}+0.5{\hat{W}}_{T+1\mid T}=100.15+0+0=100.15$. The MA shocks past the latest observation are unforecastable (their conditional expectation is zero), and the AR side is just the random-walk integration, so all multi-step forecasts collapse to the one-step value. The forecast variance, on the other hand, grows: at horizon 2 it is ${\sigma }_w^2(1+(1+0.5{)}^2)=1+2.25=3.25$, and continues to grow linearly in $h$ thereafter.

The one-step forecast unrolls