Credible Intervals

Credible Interval

A $(1-\alpha )$ credible interval for a parameter $\theta$ under posterior $p(\theta \mid y)$ is any set $C\subseteq \Theta$ satisfying
$P(\theta \in C\mid y)={\int }_{C}p(\theta \mid y)d\theta =1-\alpha .$
Unlike a frequentist confidence interval, the probability statement is a direct posterior probability: given the observed data (and the prior), there is a $1-\alpha$ probability that $\theta$ lies in $C$.

Two conventional choices of $C$ satisfy this constraint:

• Equal-tail interval: $C=[q_{\alpha /2},q_{1-\alpha /2}]$, where $q_{\gamma }$ is the $\gamma$-quantile of the posterior. Trivial to compute from posterior samples.
• Highest posterior density (HPD) interval: $C=\{\theta :p(\theta \mid y)\ge k\}$, where $k$ is chosen so $P(\theta \in C\mid y)=1-\alpha$. This is the shortest credible set for unimodal posteriors.

The two coincide when the posterior is symmetric and unimodal (e.g., Gaussian); they can differ meaningfully for skewed posteriors, where the equal-tail interval can exclude the mode or include low-density tail regions.

Contrast with confidence intervals. A frequentist $(1-\alpha )$ confidence interval $[L(Y),U(Y)]$ satisfies $P_{\theta }(L(Y)\le \theta \le U(Y))=1-\alpha$ for every fixed $\theta$: the randomness is in the data, the parameter is fixed, and the coverage statement is over hypothetical repetitions of the experiment. A credible interval reverses this: the randomness is in $\theta$ (as a posterior random variable), the data are fixed at their observed value, and the probability statement is directly about the parameter. The two can coincide numerically under flat priors and regular models (e.g., Normal mean under a flat prior gives the same interval either way), but they answer different questions.

Intuition

Credible intervals summarize posterior uncertainty in a single “plausible range.” They are the Bayesian counterpart of confidence intervals but carry the interpretation that every student wishes a confidence interval had — a direct probability statement about where the parameter is. This conceptual cleanness is the main practical appeal of Bayesian inference: if the posterior has been correctly computed, the $95\%$ credible interval is the range containing $95\%$ of the posterior mass, full stop.

The choice between equal-tail and HPD comes down to skew. For a symmetric posterior, the two coincide and reporting either is fine. For a heavily skewed posterior (common with ratio parameters, variance components, or small-sample binomial proportions) the equal-tail interval can be noticeably wider than the HPD and can exclude the posterior mode. The HPD is always the shortest set with the required mass but may be disconnected for multi-modal posteriors — which is itself a useful signal that a single interval is the wrong summary.

Properties

(Direct Probability Interpretation) A $95\%$ credible interval has $0.95$ posterior probability of containing $\theta$.

The statement "$P(\theta \in [L,U]\mid y)=0.95$" is literal and direct, not a statement about the long-run frequency of repeated intervals. This matches most users’ intuition for what a $95\%$ interval “should” mean. The cost is that the probability statement is only as good as the prior: if the prior is misspecified, the posterior is miscalibrated and the credible interval can have poor frequentist coverage. For reasonable priors and regular models, Bernstein-von Mises asymptotics guarantee that the credible interval’s frequentist coverage approaches $1-\alpha$ as $n\to \infty$, so the two notions converge in the large-data limit.

(HPD Minimizes Length) Among all sets with posterior probability $1-\alpha$, the HPD set has minimum total length.

Formally, the HPD region $C^{\ast }=\{\theta :p(\theta \mid y)\ge k_{\alpha }\}$ with $k_{\alpha }$ chosen so $P(\theta \in C^{\ast }\mid y)=1-\alpha$ minimizes $\text{Leb}(C)$ over all sets $C$ with $P(\theta \in C\mid y)=1-\alpha$. The proof is a swap argument: any set $C$ with the same mass but different shape must include some low-density point not in $C^{\ast }$ and exclude some high-density point in $C^{\ast }$, so swapping shortens the total length. For unimodal posteriors $C^{\ast }$ is an interval; for multi-modal posteriors it can be a union of disconnected intervals, which is either a feature (exposes the multi-modality) or a flag that the posterior summary is inadequate.

(Equal-Tail Is Transformation-Invariant) Equal-tail credible intervals are preserved under monotone reparameterization; HPD intervals are not.

If $\eta =h(\theta )$ for a monotone increasing $h$, the equal-tail posterior quantiles transform as $q_{\gamma }(\eta )=h(q_{\gamma }(\theta ))$. The interval $[h(q_{\alpha /2}),h(q_{1-\alpha /2})]$ is the equal-tail credible interval for $\eta$. The HPD interval for $\eta$ is not in general $[h(L_{\text{HPD}}(\theta )),h(U_{\text{HPD}}(\theta ))]$ — the Jacobian of $h$ reshapes the posterior density and moves the level-set boundary. For scale parameters (where “log-$\theta$ flat” is natural), this matters: the HPD interval for ${\sigma }^2$ and the one for $\log {\sigma }^2$ generally do not agree after back-transformation, while the equal-tail intervals do.

(Frequentist Coverage) A Bayesian credible interval has approximate frequentist coverage $1-\alpha$ under regularity and diffuse priors.

For smooth parametric models with interior true parameter, the posterior concentrates around the MLE as $N(\hat{\theta },\mathcal{I}(\hat{\theta }{)}^{-1}/n)$ (Bernstein-von Mises), so the credible interval converges to the frequentist Wald interval. For small $n$ or informative priors, the coverage can be systematically above or below $1-\alpha$ — this is a feature of Bayesian inference, not a bug, because the coverage is conditional on the prior being correct. Diagnosing coverage requires simulation studies (fixed-$\theta$ frequentist audits of Bayesian intervals) and is standard practice for validated Bayesian workflows.

(Computation from Samples) Both interval types are trivial from MCMC output.

With $N$ posterior samples ${\theta }^{(1)},\dots ,{\theta }^{(N)}$, the equal-tail interval is $[{\theta }_{(\lfloor N\alpha /2\rfloor )},{\theta }_{(\lceil N(1-\alpha /2)\rceil )}]$ — just the empirical $\alpha /2$ and $1-\alpha /2$ quantiles. The HPD interval requires locating the minimum-width window containing $1-\alpha$ of the samples: sort ${\theta }^{(i)}$, consider every contiguous block of $\lceil (1-\alpha )N\rceil$ samples, and take the one with smallest spread. This is $O(N\log N)$ and gives the HPD interval to Monte Carlo precision. For multivariate posteriors, HPD regions require kernel density estimation or level-set methods and are rarely reported — marginal equal-tail intervals and joint credible ellipsoids from the posterior covariance are the standard substitutes.

(Credible Interval $\ne$ Confidence Interval) The two answer different questions and can give numerically different intervals for the same data.

A confidence interval quantifies the reliability of a procedure — “intervals built this way capture the truth $95\%$ of the time over repeated experiments.” A credible interval quantifies belief given observed data — “given what we saw, there is a $95\%$ probability the parameter is here.” For a Normal mean under a flat prior the two are numerically identical; for a Binomial proportion under a uniform prior the Bayesian $\text{Beta}(y+1,n-y+1)$ credible interval differs from the Wilson confidence interval, and both differ from the Wald interval. In small samples near the boundary (e.g., $y=0$ or $y=n$), the Bayesian interval is well-defined and stays strictly inside $(0,1)$ while the Wald interval collapses to a single point — a practical advantage of the Bayesian approach.

Examples

Beta posterior credible intervals

Observe $y=3$ heads in $n=10$ flips with a $\text{Beta}(1,1)$ (uniform) prior. The posterior is $\theta \mid y\sim \text{Beta}(4,8)$.

The $95\%$ equal-tail credible interval uses the posterior quantiles:
$[q_{0.025}(\text{Beta}(4,8)),q_{0.975}(\text{Beta}(4,8))]\approx [0.109,0.610].$

The $95\%$ HPD interval locates the shortest window containing $0.95$ posterior mass. For $\text{Beta}(4,8)$ (which is right-skewed because $\alpha <\beta$), the HPD is shifted slightly left of the equal-tail:
$[L_{\text{HPD}},U_{\text{HPD}}]\approx [0.094,0.586].$

The HPD is roughly $0.49$ wide vs. the equal-tail’s $0.50$ — a $2\%$ savings from recognizing the skew. For a near-symmetric $\text{Beta}(50,50)$ posterior the two intervals would be indistinguishable.

Compare to a $95\%$ Wald confidence interval: $\hat{\theta }\pm 1.96\sqrt{\hat{\theta }(1-\hat{\theta })/n}=0.3\pm 0.284=[0.016,0.584]$, which is narrower but has known undercoverage issues for small $n$ and extreme $\hat{\theta }$. The Bayesian interval is automatically bounded inside $[0,1]$ and remains sensible when $y=0$ or $y=n$.

Student- $t$ posterior for a Normal mean

With $y_1,\dots ,y_n\sim N(\mu ,{\sigma }^2)$ and a flat prior on $(\mu ,\log {\sigma }^2)$, the marginal posterior for $\mu$ is a location-scale Student-$t$:
$\mu \mid y\sim t_{n-1}\left(\bar{y},\frac{s^2}{n}\right),$
where $s^2=\frac{1}{n-1}\sum (y_i-\bar{y}{)}^2$. The $95\%$ equal-tail credible interval is
$\bar{y}\pm t_{n-1,0.025}\frac{s}{\sqrt{n}},$
which is numerically identical to the frequentist $t$-confidence interval. The interpretation differs: the Bayesian reads this as a direct posterior probability, the frequentist as a repeated-sampling guarantee. This coincidence is a special case of the Bernstein-von Mises theorem at work in the Gaussian setting; it does not generalize to all flat-prior problems.

Credible interval from MCMC output

A Metropolis-Hastings chain on a Bayesian logistic regression produces $N=9,000$ post-burn-in samples of a coefficient ${\beta }_j$. To report a $95\%$ credible interval:

Equal-tail: sort the samples and take the $225$th and $8,775$th values (indices $\lceil 0.025\cdot 9000\rceil$ and $\lfloor 0.975\cdot 9000\rfloor$). For example, $[-0.21,0.67]$.

HPD: sort the samples ${\beta }_j^{(1)}\le \cdots \le {\beta }_j^{(9000)}$. Compute the $8,549$-wide windows $[{\beta }_j^{(i)},{\beta }_j^{(i+8549)}]$ for $i=1,\dots ,451$ and return the window of smallest width. For a skewed posterior the HPD is typically shifted toward the mode; for a near-symmetric posterior the two intervals differ only in Monte Carlo noise.

In both cases, the Monte Carlo error of the interval endpoints is $O(1/\sqrt{N_{\text{eff}}})$, where $N_{\text{eff}}$ is the effective sample size. Running the chain long enough that $N_{\text{eff}}\gg 1000$ ensures two-decimal-place precision on the interval.

See Also

• 3.1 Bayesian Framework — prior, likelihood, posterior; credible intervals are posterior summaries.
• 1.3.2 Confidence Intervals — frequentist counterpart; coincides numerically in regular flat-prior models.
• 3. Bayesian Statistics — parent index.
• 3.11 MCMC — how to compute credible intervals from posterior samples.
• 1.2.2 Maximum a Posteriori Estimation — posterior mode; often reported alongside a credible interval.