Bayesian AutoRegression

Bayesian AR(1) and AR(p)

The Bayesian AR(1) model assumes a scalar time series $\{y_t{\}}_{t=1}^n$ with
$y_t\mid y_{t-1},\varphi ,{\sigma }^2\sim N(\varphi y_{t-1},{\sigma }^2),t=2,\dots ,n,$
and an initial observation $y_1$ (often treated as known or given its stationary distribution). The AR(p) extension has
$y_t\mid {\varphi }_1,\dots ,{\varphi }_p,{\sigma }^2\sim N\left({\sum }_{k=1}^p{\varphi }_k{y}_{t-k},{\sigma }^2\right).$
Written in regression form with $\mathbf{\varphi }=({\varphi }_1,\dots ,{\varphi }_p{)}^{\top }$ and design matrix $\mathbf{X}$ whose rows are lagged windows $(y_{t-1},\dots ,y_{t-p})$, the conditional likelihood is
$p(\mathbf{y}\mid \mathbf{\varphi },{\sigma }^2)=N(\mathbf{X}\mathbf{\varphi },{\sigma }^2\mathbf{I}),$
identical to a Bayesian linear regression with the design built from lags. Under a Normal-Inverse-Gamma prior $\mathbf{\varphi }\mid {\sigma }^2\sim N({\mathbf{\varphi }}_0,{\sigma }^2{\mathbf{\Lambda }}_0^{-1})$ and ${\sigma }^2\sim \text{InvGamma}(a_0,b_0)$, the posterior is again NIG, with the same closed-form updates.

Stationarity constraint. AR(1) is stationary iff $|\varphi |<1$. For AR(p), stationarity requires all roots of the characteristic polynomial $1-{\varphi }_{1}z-\cdots -{\varphi }_p{z}^p=0$ to lie outside the unit circle. A Bayesian analysis with unrestricted Gaussian prior on $\mathbf{\varphi }$ puts nonzero mass on non-stationary parameters. Truncating the prior to the stationarity region (or reparametrizing to partial autocorrelations) is standard practice when the series is assumed stationary.

Why AR lives in the Bayesian-regression folder. The likelihood factorizes by the Markov property, $p(\mathbf{y}\mid \mathbf{\varphi },{\sigma }^2)=p(y_1){\prod }_{t=2}^{n}p(y_t\mid y_{t-1},\mathbf{\varphi },{\sigma }^2)$, so conditional on the initial observation(s), AR is a linear regression of each $y_t$ on its lags. All the conjugate machinery of 3.8.1 Bayesian Linear Regression applies.

Why this is in the course. AR is a canonical illustration of the likelihood principle. A frequentist analyst must specify whether $n$ was fixed in advance or whether sampling continued until a particular event (e.g., first observation exceeding a threshold). These stopping rules change the frequentist sampling distribution and therefore the frequentist confidence interval. The Bayesian posterior depends only on the observed likelihood — the stopping rule does not enter. The Bayesian is invariant to the stopping rule.

Prerequisites

• 3.1 Bayesian Framework — Bayesian mechanics including likelihood principle.
• 3.8.1 Bayesian Linear Regression — the regression machinery applies directly.
• 3.5.2 Normal-Normal Model — univariate conjugate Gaussian analysis.

Intuition

An AR(1) process $y_t=\varphi y_{t-1}+{\epsilon }_t$ with i.i.d. Gaussian ${\epsilon }_t$ has the property that conditional on the previous observation, the current one is Gaussian with mean $\varphi y_{t-1}$ — just a linear regression whose predictor happens to be a lagged copy of the response. The conditional likelihood factorizes the same way (product over $t$), and the Gaussian-Gaussian conjugacy goes through unchanged. Bayesian inference for $\varphi$ under a Gaussian prior is therefore identical to Bayesian inference for a regression coefficient.

The stopping-rule invariance of Bayesian AR analysis is the most-taught consequence of the likelihood principle. Suppose an analyst plans to observe an AR(1) series until time $n$ or until the first time the series crosses some threshold $c$, whichever comes first. Frequentist analysis must track which stopping rule produced the data: different rules produce different sampling distributions for the MLE, different confidence intervals, different $p$-values. The Bayesian posterior is the same regardless — provided the stopping rule depends only on observed quantities, the likelihood factor from the stopping rule multiplies into both the numerator and denominator of Bayes’ rule and cancels. The AR(1) walk-through in STAT 238 makes this unmistakable: plot two posteriors from datasets drawn under different stopping rules, and they coincide; plot the matching frequentist CIs and they diverge.

The stationarity question is where Bayesian and frequentist analyses share a genuine difficulty. Near the stationarity boundary $|\varphi |=1$ (unit root), posterior distributions and frequentist sampling distributions are both highly non-Gaussian and sensitive to prior choice or normalization convention. This is where priors on $\varphi$ matter: a flat prior on $\varphi$ puts extensive mass near 1, which biases inference toward unit-root behavior. A prior concentrated in $(-1,1)$ — or reparametrization to the partial autocorrelation in $(-1,1)$ — handles this cleanly.

Properties

(Likelihood-Principle Invariance) The posterior depends only on the observed likelihood and is invariant to the stopping rule.

If data $\mathbf{y}$ is collected under any stopping rule $\tau$ whose decision at each step depends only on $y_{1:t}$ (not on $\mathbf{\varphi }$ or ${\sigma }^2$), then the likelihood factors as
$p(\mathbf{y},\tau =n\mid \mathbf{\varphi },{\sigma }^2)=p(\mathbf{y}\mid \mathbf{\varphi },{\sigma }^2)\cdot p(\tau =n\mid \mathbf{y}),$
and the stopping-rule factor does not depend on the parameters. It cancels in Bayes’ rule:
$p(\mathbf{\varphi },{\sigma }^2\mid \mathbf{y},\tau =n)=\frac{p(\mathbf{\varphi },{\sigma }^2)p(\mathbf{y}\mid \mathbf{\varphi },{\sigma }^2)}{p(\mathbf{y})}=p(\mathbf{\varphi },{\sigma }^2\mid \mathbf{y}).$
Frequentist inference, by contrast, integrates over the sampling distribution of $(\mathbf{y},\tau )$, which differs by stopping rule even for identical observed data. This is the clearest inferential consequence of the likelihood principle; see 3.1 Bayesian Framework.

(Conjugate NIG Posterior via Regression Representation) AR(p) inherits the closed-form NIG posterior of Bayesian linear regression.

With $\mathbf{X}$ the design matrix of lagged values and $\mathbf{y}$ the response vector (for $t=p+1,\dots ,n$),
${\mathbf{\varphi }}_n={\mathbf{\Lambda }}_n^{-1}({\mathbf{X}}^{\top }\mathbf{y}+{\mathbf{\Lambda }}_0{\mathbf{\varphi }}_0),{\mathbf{\Lambda }}_n={\mathbf{X}}^{\top }\mathbf{X}+{\mathbf{\Lambda }}_0,$
with posterior marginal $\mathbf{\varphi }\mid \mathbf{y}\sim t_{2a_n}({\mathbf{\varphi }}_n,(b_n/a_n){\mathbf{\Lambda }}_n^{-1})$. All 3.8.1 Bayesian Linear Regression results carry over unchanged. The conditional likelihood treats the initial $p$ observations as given; exact inclusion (via the stationary-distribution density) modifies $b_n$ only.

(Stationarity Constraint) Unrestricted Gaussian prior on $\mathbf{\varphi }$ places mass on non-stationary parameters.

For AR(1), the stationary region is $|\varphi |<1$; for AR(p), the roots of $1-{\varphi }_{1}z-\cdots -{\varphi }_p{z}^p$ must lie outside the unit disk. Default Gaussian priors are not supported on the stationarity region. Options: (a) truncate the prior to the stationarity region and renormalize (Gibbs-friendly); (b) reparametrize to partial autocorrelations ${\rho }_1,\dots ,{\rho }_p\in (-1,1)$ and put independent Beta-style priors on each (Barndorff-Nielsen-Schou transform); (c) accept that the posterior may place mass on non-stationary parameters and interpret accordingly.

(Posterior Predictive via Markov Sampling) Future values are forecast by cascading posterior samples through the AR recursion.

The $h$-step-ahead posterior predictive for AR(1) is
${\tilde{y}}_{n+h}\mid \mathbf{y}={\varphi }^h{y}_n+{\sum }_{k=0}^{h-1}{\varphi }^k{\epsilon }_{n+h-k},\epsilon \sim N(0,{\sigma }^2),$
marginalized over the posterior of $(\varphi ,{\sigma }^2)$. Closed form is not generally available, but Monte Carlo is direct: sample $({\varphi }^{(s)},{\sigma }^{2,(s)})$ from the posterior, then cascade the recursion forward to draw ${\tilde{y}}_{n+1}^{(s)},\dots ,{\tilde{y}}_{n+h}^{(s)}$. The empirical distribution of the samples is the posterior predictive.

(Reference Prior and Bayesian Bridge to Frequentist AR) The improper prior $p(\mathbf{\varphi },{\sigma }^2)\propto 1/{\sigma }^2$ reproduces Yule-Walker / least-squares inference.

Under the reference prior limit, ${\mathbf{\varphi }}_n=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$ equals the conditional-MLE AR coefficient (also the Yule-Walker estimate up to a finite-sample correction), and the marginal posterior is Student-$t$ around it. Frequentist confidence intervals and Bayesian credible intervals numerically coincide in this limit.

(AR as a Gaussian-Process Kernel) The AR(1) process is a GP with exponential kernel (discrete-time analog).

The AR(1) autocovariance is $\text{Cov}(y_s,y_t)={\sigma }^2{\varphi }^{|s-t|}/(1-{\varphi }^2)$ in stationarity. This is the discrete-time version of the Ornstein-Uhlenbeck kernel used in GP regression: $k(s,t)=\exp (-|s-t|/\ell )$ with $\varphi =\exp (-1/\ell )$. Bayesian AR is therefore a GP with a specific kernel and equi-spaced observations; the posterior Student-$t$ for $\mathbf{\varphi }$ and the GP posterior over functions are two windows onto the same mathematical object.

Examples

Example 1: Stopping-Rule Invariance on AR(1)

Two analysts observe the same AR(1) process, but with different stopping rules:

Analyst A: Plans to observe $n=100$ data points regardless.
Analyst B: Plans to stop at the first time $t$ such that $y_t>3$, up to a maximum of $n=100$.

Both collect the same initial 100 observations before the threshold is crossed (say at $t=100$, just barely). The observed likelihood is identical:
$p(\mathbf{y}\mid \varphi ,{\sigma }^2)={\prod }_{t=2}^{100}N(y_t;\varphi y_{t-1},{\sigma }^2).$

Bayesian posterior: identical for A and B under any common prior.
Frequentist MLE sampling distribution: different — B’s sampling distribution conditions on $y_{100}>3$ and all earlier $y_t\le 3$, which is not the same as A’s unconditional distribution. Frequentist confidence intervals differ.

The Bayesian analyst simply reports $\varphi \mid \mathbf{y}$ and is done. The frequentist must explain why the analysis depends on what the experimenter was thinking about stopping — a philosophical discomfort often cited as the single most concrete reason to prefer Bayesian methods for sequential data collection.

Example 2: Closed-Form NIG Posterior for AR(1)

Observed series $y_1,\dots ,y_n$ with weakly informative prior $\varphi \sim N(0,10)$, ${\sigma }^2\sim \text{InvGamma}(2,1)$ (prior mean ${\sigma }^2=1$, fairly vague).

Design quantities: $S_{xx}={\sum }_{t=2}^n{y}_{t-1}^2$, $S_{xy}={\sum }_{t=2}^n{y}_{t-1}{y}_t$, $S_{yy}={\sum }_{t=2}^n{y}_t^2$.
Posterior mean: ${\varphi }_n=S_{xy}/(S_{xx}+0.1)$ — the ridge-regularized AR coefficient.
Posterior scale: $a_n=2+(n-1)/2$, $b_n=1+(S_{yy}-{\varphi }_n{S}_{xy})/2+0.1{\varphi }_n^2/2$.
Marginal posterior for $\varphi$: $t_{2a_n}({\varphi }_n,b_n/(a_n(S_{xx}+0.1)))$.

For a series of $n=200$ observations with sample autocorrelation ${\hat{\rho }}_1\approx 0.7$, the posterior concentrates sharply near 0.7 with a narrow credible interval. Under the prior $|\varphi |<1$ truncation, the posterior tail at $\varphi =0.99$ is negligible and no correction is needed.

Example 3: Posterior Predictive Forecast for AR(1)

Given posterior $\varphi \mid \mathbf{y}\sim t_{200}(0.7,0.01)$, ${\sigma }^2\mid \mathbf{y}\sim \text{InvGamma}(100,100)$ (so ${\hat{\sigma }}^2\approx 1$), and final observation $y_n=2.0$, forecast $y_{n+1},\dots ,y_{n+5}$.

Monte Carlo procedure: draw $S=10,000$ posterior samples $({\varphi }^{(s)},{\sigma }^{2,(s)})$, then iterate the recursion for each sample, drawing ${\epsilon }_{n+h}^{(s)}\sim N(0,{\sigma }^{2,(s)})$.

One-step forecast: mean $\approx 0.7\cdot 2.0=1.4$, 95% credible interval $[1.4-1.96,1.4+1.96]\approx [-0.57,3.36]$ (widening by $\sqrt{{\hat{\sigma }}^2}=1$).
Five-step forecast: mean $\approx 0.7^5\cdot 2.0=0.336$, with cumulative variance ${\sigma }^2(1+{\varphi }^2+{\varphi }^4+{\varphi }^6+{\varphi }^8)\approx 2.45$; 95% CI $\approx [-2.73,3.40]$.

Forecast uncertainty grows as the horizon extends; in the limit, the posterior predictive approaches the stationary distribution $N(0,{\sigma }^2/(1-{\varphi }^2))\approx N(0,1.96)$.

Derivation

Goal: show that the stopping-rule factor cancels from the AR(1) posterior.

Suppose $\tau$ is a stopping rule whose decision $\{\tau \le t\}$ is $\sigma (y_1,\dots ,y_t)$-measurable, i.e. depends only on data observed so far. The joint density of the observed data and the stopping event is
$p(\mathbf{y},\tau =n\mid \mathbf{\varphi },{\sigma }^2)=p(y_1\mid \mathbf{\varphi },{\sigma }^2){\prod }_{t=2}^{n}p(y_t\mid y_{t-1},\mathbf{\varphi },{\sigma }^2)\cdot 1[\tau =n\mid y_{1:n}].$
The stopping-rule indicator depends on $\mathbf{y}$ but not on the parameters — call this factor $S(\mathbf{y})$. Then
$p(\mathbf{\varphi },{\sigma }^2\mid \mathbf{y},\tau =n)=\frac{p(\mathbf{\varphi },{\sigma }^2)p(\mathbf{y}\mid \mathbf{\varphi },{\sigma }^2)S(\mathbf{y})}{\int p(\mathbf{\varphi },{\sigma }^2)p(\mathbf{y}\mid \mathbf{\varphi },{\sigma }^2)S(\mathbf{y})d\mathbf{\varphi }d{\sigma }^2}.$
The $S(\mathbf{y})$ factor appears identically in numerator and denominator and cancels, leaving
$p(\mathbf{\varphi },{\sigma }^2\mid \mathbf{y},\tau =n)=\frac{p(\mathbf{\varphi },{\sigma }^2)p(\mathbf{y}\mid \mathbf{\varphi },{\sigma }^2)}{\int p(\mathbf{\varphi },{\sigma }^2)p(\mathbf{y}\mid \mathbf{\varphi },{\sigma }^2)d\mathbf{\varphi }d{\sigma }^2}=p(\mathbf{\varphi },{\sigma }^2\mid \mathbf{y}).$
The posterior is the same as if $n$ had been fixed in advance. This is exactly the (Likelihood Principle) property of 3.1 Bayesian Framework applied to AR models.

See Also

• 3.8 Bayesian Regression — parent index.
• 3.8.1 Bayesian Linear Regression — conjugate machinery applied here.
• 3.1 Bayesian Framework — likelihood principle and stopping-rule invariance.
• 3.10 Gaussian Processes — AR(1) as a discrete-time GP with exponential kernel.
• 3.11.2 Gibbs Sampling — sampler when the stationarity constraint or hierarchical extensions break conjugacy.
• 2.4 Time Series Regression — frequentist time-series regression for comparison.