2.2.2 Autocovariance Function

Autocovariance Function

The autocovariance function of a time series $\{X_t\}$ measures the linear dependence between observations at two time points $s$ and $t$, defined as
${\gamma }_X(s,t)=\text{Cov}(X_s,X_t)=\mathbb{E}[(X_s-{\mu }_s)(X_t-{\mu }_t)],$
where ${\mu }_s=\mathbb{E}[X_s]$ and ${\mu }_t=\mathbb{E}[X_t]$ are the values of the mean function at times $s$ and $t$. This is the time series analogue of covariance between two random variables.

When the process is (weakly) stationary, the autocovariance depends only on the lag $h=|s-t|$, not on the absolute times, and is written
${\gamma }_X(h)=\text{Cov}(X_t,X_{t+h})\text{for all }t.$

Note that ${\gamma }_X(t,t)={\gamma }_X(0)=\text{Var}(X_t)$ is the variance of the process.

Autocovariance is Not Scale-Free ${\gamma }_X(h)$ has units equal to the square of the units of $X_t$, making it difficult to compare across series measured on different scales. The autocorrelation function ${\rho }_X(h)={\gamma }_X(h)/{\gamma }_X(0)$ normalizes this to a unitless quantity in $[-1,1]$.

The autocovariance

Intuition

The autocovariance function captures how the memory of a process decays over time. A large $|{\gamma }_X(h)|$ at lag $h$ means that knowing the value at time $t$ gives useful information about the value at time $t+h$. If ${\gamma }_X(h)>0$, values tend to stay on the same side of the mean (persistence); if ${\gamma }_X(h)<0$, they tend to alternate (mean-reversion). If ${\gamma }_X(h)\approx 0$ for all $h\ne 0$, the series has no linear memory: it behaves like white noise.

The rate at which ${\gamma }_X(h)\to 0$ as $h$ grows characterizes how quickly the process “forgets” its past. Short-memory processes (e.g., ARMA) have autocovariances that decay exponentially; long-memory processes have autocovariances that decay polynomially.

Properties

(Symmetry) ${\gamma }_X(h)={\gamma }_X(-h)$ for all $h$. Dependence between $X_t$ and $X_{t+h}$ is the same regardless of direction in time.

(Positive Semi-Definiteness) For any $n$, any times $t_1,\dots ,t_n$, and any constants $a_1,\dots ,a_n$,

${\sum }_{i=1}^n{\sum }_{j=1}^n{a}_i{a}_j{\gamma }_X(t_i,t_j)\ge 0.$
This ensures that the variance of any linear combination $\sum a_i{X}_{t_i}$ is nonnegative.

(Variance at Lag Zero) ${\gamma }_X(0)=\text{Var}(X_t)\ge 0$.

(Cauchy–Schwarz Bound) $|{\gamma }_X(h)|\le {\gamma }_X(0)$ for all $h$. The autocovariance at any lag cannot exceed the variance.

(White Noise) If $\{W_t\}\sim \text{WN}(0,{\sigma }^2)$, then ${\gamma }_W(h)=\{\begin{array}{ll}{\sigma }^2& h=0,\\ 0& h\ne 0.\end{array}$

(Linearity) If $Y_t=a{X}_t+b$, then ${\gamma }_Y(h)=a^2{\gamma }_X(h)$.

Examples

MA(1) Process

Let $X_t=W_t+\theta W_{t-1}$, where $\{W_t\}\sim \text{WN}(0,{\sigma }^2)$ and $\theta \in \mathbb{R}$.

Derive the autocovariance function. $h=0$: ${\gamma }_X(0)=\text{Var}(W_t+\theta W_{t-1})={\sigma }^2+{\theta }^2{\sigma }^2=(1+{\theta }^2){\sigma }^2.$ For lag $h=1$: ${\gamma }_X(1)=\text{Cov}(W_t+\theta W_{t-1},W_{t+1}+\theta W_t)=\theta {\sigma }^2,$ since the only shared term is $W_t$ appearing as $\theta W_t$ in both expressions. For $|h|\ge 2$: ${\gamma }_X(h)=0$, because $X_t$ and $X_{t+h}$ share no common white noise terms. So ${\gamma }_X(h)=\{\begin{array}{ll}(1+{\theta }^2){\sigma }^2& h=0,\\ \theta {\sigma }^2& |h|=1,\\ 0& |h|\ge 2.\end{array}$ The MA(1) has memory of exactly one lag: it forgets its past after a single step.

For lag

Random Walk

Let $X_t={\sum }_{j=1}^t{W}_j$ with $\{W_j\}\sim \text{WN}(0,{\sigma }^2)$.

Show that the autocovariance depends on absolute time, not just the lag. $s\le t$: ${\gamma }_X(s,t)=\text{Cov}\left({\sum }_{i=1}^s{W}_i,{\sum }_{j=1}^t{W}_j\right)={\sum }_{i=1}^s\text{Var}(W_i)=s{\sigma }^2=\min (s,t){\sigma }^2,$ since only the first $s$ terms overlap. Because ${\gamma }_X(s,t)=\min (s,t){\sigma }^2$ depends on $s$ and $t$ individually, not just on $|s-t|$, the random walk is not stationary.

For