Bayesian Bootstrap

The Bayesian bootstrap (Rubin, 1981) replaces the equal-weight resampling of the classical bootstrap with random Dirichlet-distributed weights on the observed data points. Given a sample $x = (x_{1}, \dots, x_{n})$ , each Bayesian bootstrap replicate is generated by:

Drawing a random weight vector $w^{*} = (w_{1}^{*}, \dots, w_{n}^{*})$ from a $Dir (1, 1, \dots, 1)$ distribution (i.e., uniform on the $(n - 1)$ -simplex).

Computing the statistic using these weights: $T_{n}^{*} = g (x; w^{*})$ .

For the weighted mean, this gives $\overset{ˉ}{X}^{*} = \sum_{i = 1}^{n} w_{i}^{*} x_{i}$ . The distribution of $T_{n}^{*}$ across many draws of $w^{*}$ approximates a posterior distribution over the parameter of interest.

The Bayesian bootstrap arises from placing a noninformative prior on the space of all distributions. If the data-generating distribution $F$ is modeled nonparametrically via a Dirichlet process prior $F \sim DP (α)$ with $α \to 0$ (the limiting noninformative case), then the posterior over $F$ given the data assigns random weights to the observed values according to $Dir (1, \dots, 1)$ —which is exactly the Bayesian bootstrap.

Classical vs. Bayesian Bootstrap integer counts for each observation (some appear $0, 1, 2, \dots$ times). The Bayesian bootstrap assigns continuous weights drawn from a Dirichlet, so every observation contributes to every replicate—just with varying importance. The classical bootstrap estimates the sampling distribution of a statistic; the Bayesian bootstrap estimates a posterior distribution over the parameter. Despite this philosophical difference, the two methods produce similar intervals in practice for smooth functionals.

The classical bootstrap resamples with replacement, producing

Intuition

The classical bootstrap answers: “how would my estimate vary across repeated samples from $\hat{F}_{n}$ ?” The Bayesian bootstrap answers a different question: “given the data I observed, what is my posterior belief about the parameter?”

The key insight is the connection to the Dirichlet distribution. In a nonparametric Bayesian framework, the unknown distribution $F$ is modeled as a random object. After observing data, the posterior over $F$ concentrates on distributions supported on the observed values, with random weights. Drawing these weights from $Dir (1, \dots, 1)$ —the uniform distribution over all ways to allocate probability mass to $n$ points—corresponds to the most “uninformative” prior belief about $F$ .

Each Bayesian bootstrap replicate represents one plausible version of $F$ drawn from the posterior. Computing the statistic under each such $F$ traces out the posterior distribution of the parameter, providing a fully Bayesian uncertainty quantification without requiring any parametric model.

Properties

(Dirichlet Weights) The weight vector $w^{} \sim Dir (1, \dots, 1)$ has $E [w_{i}^{}] = 1/ n$ and $Var (w_{i}^{}) = \frac{n - 1}{n ^{2} ( n + 1 )}$ . By the Gamma construction, $w_{i}^{} = G_{i} / \sum_{j} G_{j}$ where $G_{i} \sim Exp (1)$ are independent.

(Posterior Mean) The Bayesian bootstrap mean satisfies $E [\overset{ˉ}{X}^{} ∣ x] = \overset{x}{ˉ}$ (the sample mean) and $Var (\overset{ˉ}{X}^{} ∣ x) = \frac{1}{n + 1} \overset{σ}{^}^{2}$ , where $\overset{σ}{^}^{2} = \frac{1}{n} \sum_{i = 1}^{n} (x_{i} - \overset{x}{ˉ})^{2}$ . This is close to but not identical to the classical bootstrap variance $\overset{σ}{^}^{2} / n$ .

(Asymptotic Equivalence) For smooth functionals (means, quantiles, regression coefficients), the classical and Bayesian bootstrap are asymptotically equivalent—both approximate the same limiting distribution as $n \to \infty$ .

(Credible Intervals) The $(1 - α)$ Bayesian bootstrap interval is the $[α /2, 1 - α /2]$ quantile range of $T_{n}^{*}$ over the Dirichlet replicates. Unlike frequentist confidence intervals, these can be interpreted directly as posterior probability statements (under the nonparametric prior).

(Weighted Likelihood) For parametric models, the Bayesian bootstrap generalizes to weighted likelihood bootstrap: maximize $\sum_{i = 1}^{n} w_{i}^{} lo g f (x_{i}; θ)$ for each draw of $w^{}$ . The resulting distribution of $\hat{θ}^{*}$ approximates the Bayesian posterior under a flat prior.

Examples

Bayesian Bootstrap of the Mean

A sample of $n = 5$ observations is $x = (2.1, 3.4, 2.8, 4.0, 3.2)$ with sample mean $\overset{x}{ˉ} = 3.1$ .

Procedure and Posterior $B = 10, 000$ replicates, draw $(w_{1}^{*}, \dots, w_{5}^{*}) \sim Dir (1, 1, 1, 1, 1)$ and compute $\overset{ˉ}{X}^{*} = \sum_{i = 1}^{5} w_{i}^{*} x_{i}$ . The posterior mean is $E [\overset{ˉ}{X}^{*}] = 3.1$ and the posterior variance is $Var (\overset{ˉ}{X}^{*} ∣ x) = \frac{1}{6} \cdot \frac{1}{5} \sum_{i = 1}^{5} (x_{i} - 3.1)^{2} = \frac{1}{6} \cdot 0.412 \approx 0.069,$ giving a posterior standard deviation of $\approx 0.263$ . A $95%$ credible interval from the Dirichlet replicates might be approximately $[2.55, 3.60]$ .

For each of

Comparing Classical and Bayesian Bootstrap

Given $n = 100$ observations from an unknown distribution with sample mean $\overset{x}{ˉ} = 50$ and sample standard deviation $s = 10$ .

Variance Comparison $\overset{ˉ}{X}^{*}$ is approximately $s^{2} / n = 100/100 = 1.0$ . The Bayesian bootstrap variance is $\overset{σ}{^}^{2} / (n + 1) = 100/101 \approx 0.99$ . For $n = 100$ , the two are nearly identical. The practical difference only matters for very small samples, where the Bayesian bootstrap produces slightly narrower intervals reflecting the Dirichlet prior's regularization.

The classical bootstrap variance of

Project Alexandria

Explorer

1.5.2 Bayesian Bootstrap

Intuition

Properties

(Dirichlet Weights) The weight vector $w^{} \sim Dir (1, \dots, 1)$ has $E [w_{i}^{}] = 1/ n$ and $Var (w_{i}^{}) = \frac{n - 1}{n ^{2} ( n + 1 )}$ . By the Gamma construction, $w_{i}^{} = G_{i} / \sum_{j} G_{j}$ where $G_{i} \sim Exp (1)$ are independent.

(Asymptotic Equivalence) For smooth functionals (means, quantiles, regression coefficients), the classical and Bayesian bootstrap are asymptotically equivalent—both approximate the same limiting distribution as $n \to \infty$ .

(Credible Intervals) The $(1 - α)$ Bayesian bootstrap interval is the $[α /2, 1 - α /2]$ quantile range of $T_{n}^{*}$ over the Dirichlet replicates. Unlike frequentist confidence intervals, these can be interpreted directly as posterior probability statements (under the nonparametric prior).

(Weighted Likelihood) For parametric models, the Bayesian bootstrap generalizes to weighted likelihood bootstrap: maximize $\sum_{i = 1}^{n} w_{i}^{} lo g f (x_{i}; θ)$ for each draw of $w^{}$ . The resulting distribution of $\hat{θ}^{*}$ approximates the Bayesian posterior under a flat prior.

Examples

Graph View

Table of Contents

Backlinks

Project Alexandria

Explorer

1.5.2 Bayesian Bootstrap

Intuition

Properties

(Dirichlet Weights) The weight vector w∗∼Dir(1,…,1) has E[wi∗​]=1/n and Var(wi∗​)=n2(n+1)n−1​. By the Gamma construction, wi∗​=Gi​/∑j​Gj​ where Gi​∼Exp(1) are independent.

(Posterior Mean) The Bayesian bootstrap mean satisfies E[Xˉ∗∣x]=xˉ (the sample mean) and Var(Xˉ∗∣x)=n+11​σ^2, where σ^2=n1​∑i=1n​(xi​−xˉ)2. This is close to but not identical to the classical bootstrap variance σ^2/n.

(Asymptotic Equivalence) For smooth functionals (means, quantiles, regression coefficients), the classical and Bayesian bootstrap are asymptotically equivalent—both approximate the same limiting distribution as n→∞.

(Credible Intervals) The (1−α) Bayesian bootstrap interval is the [α/2,1−α/2] quantile range of Tn∗​ over the Dirichlet replicates. Unlike frequentist confidence intervals, these can be interpreted directly as posterior probability statements (under the nonparametric prior).

(Weighted Likelihood) For parametric models, the Bayesian bootstrap generalizes to weighted likelihood bootstrap: maximize ∑i=1n​wi∗​logf(xi​;θ) for each draw of w∗. The resulting distribution of θ^∗ approximates the Bayesian posterior under a flat prior.

Examples

Graph View

Table of Contents

Backlinks

(Dirichlet Weights) The weight vector $w^{} \sim Dir (1, \dots, 1)$ has $E [w_{i}^{}] = 1/ n$ and $Var (w_{i}^{}) = \frac{n - 1}{n ^{2} ( n + 1 )}$ . By the Gamma construction, $w_{i}^{} = G_{i} / \sum_{j} G_{j}$ where $G_{i} \sim Exp (1)$ are independent.

(Asymptotic Equivalence) For smooth functionals (means, quantiles, regression coefficients), the classical and Bayesian bootstrap are asymptotically equivalent—both approximate the same limiting distribution as $n \to \infty$ .

(Credible Intervals) The $(1 - α)$ Bayesian bootstrap interval is the $[α /2, 1 - α /2]$ quantile range of $T_{n}^{*}$ over the Dirichlet replicates. Unlike frequentist confidence intervals, these can be interpreted directly as posterior probability statements (under the nonparametric prior).

(Weighted Likelihood) For parametric models, the Bayesian bootstrap generalizes to weighted likelihood bootstrap: maximize $\sum_{i = 1}^{n} w_{i}^{} lo g f (x_{i}; θ)$ for each draw of $w^{}$ . The resulting distribution of $\hat{θ}^{*}$ approximates the Bayesian posterior under a flat prior.