1.5.1 Classical Bootstrap

Classical Bootstrap

The classical (nonparametric) bootstrap is a resampling method for approximating the sampling distribution of a statistic $T_n=g(X_1,\dots ,X_n)$ without making parametric assumptions about the population. Given an observed sample $\mathbf{x}=(x_1,\dots ,x_n)$, the bootstrap proceeds by:

1. Defining the empirical distribution ${\hat{F}}_n$, which places mass $1/n$ on each observed value $x_i$.
2. Drawing $B$ independent bootstrap samples ${\mathbf{x}}^{\ast 1},\dots ,{\mathbf{x}}^{\ast B}$, each of size $n$, by sampling with replacement from ${\hat{F}}_n$.
3. Computing the statistic on each bootstrap sample: $T_n^{\ast b}=g({\mathbf{x}}^{\ast b})$ for $b=1,\dots ,B$.

The distribution of $T_n^{\ast 1},\dots ,T_n^{\ast B}$ approximates the true sampling distribution of $T_n$. From this, one can estimate:

• Standard error: ${\widehat{\text{SE}}}_{\text{boot}}=\text{SD}(T_n^{\ast 1},\dots ,T_n^{\ast B})$.
• Bias: ${\widehat{\text{bias}}}_{\text{boot}}={\bar{T}}_n^{\ast }-T_n(\mathbf{x})$.
• Confidence intervals: via the percentile method, $[T_n^{\ast (\alpha /2)},T_n^{\ast (1-\alpha /2)}]$.

The bootstrap is justified by the plug-in principle: the empirical distribution ${\hat{F}}_n$ converges to the true distribution $F$ as $n\to \infty$ (by the Glivenko-Cantelli theorem), so resampling from ${\hat{F}}_n$ mimics sampling from $F$.

When the Bootstrap Fails $T_n$ to depend smoothly on the underlying distribution. It can fail for extreme-value statistics (e.g., ${\max }_i{X}_i$), when the population has infinite variance, or when the data exhibits strong dependence (e.g., time series) unless modified (block bootstrap). It also requires $n$ to be large enough that ${\hat{F}}_n$ adequately represents $F$.

The bootstrap requires the statistic

Intuition

The core idea of the bootstrap is deceptively simple: treat the observed sample as if it were the population, and resample from it. Since we cannot draw new samples from the unknown population $F$, we instead draw from our best approximation of it—the empirical distribution ${\hat{F}}_n$, which assigns equal probability to each observed data point.

By resampling with replacement, each bootstrap sample is a plausible “alternative dataset” that we might have observed. Some original observations will appear multiple times, others not at all—this variability across bootstrap samples mirrors the variability across real samples from $F$. Computing the statistic on many such resamples reveals how much the statistic fluctuates, giving us the standard error and confidence intervals without needing to know $F$ or to derive formulas analytically.

The bootstrap replaces mathematical derivation with computation: instead of proving that $\hat{\theta }$ is asymptotically normal and deriving its variance, we simply observe the spread of ${\hat{\theta }}^{\ast }$ across resamples.

Properties

(Consistency) Under regularity conditions, the bootstrap distribution of $\sqrt{n}(T_n^{\ast }-T_n)$ converges to the true sampling distribution of $\sqrt{n}(T_n-\theta )$ as $n\to \infty$. That is, the bootstrap is consistent—its approximation improves with sample size.

(Percentile Interval) The $(1-\alpha )$ bootstrap percentile confidence interval is $[T_n^{\ast (\alpha /2)},T_n^{\ast (1-\alpha /2)}]$, where $T_n^{\ast (q)}$ denotes the $q$-th quantile of the bootstrap distribution. This interval is transformation-invariant: if $\varphi =h(\theta )$ for a monotone $h$, the interval for $\varphi$ is $[h(T_n^{\ast (\alpha /2)}),h(T_n^{\ast (1-\alpha /2)})]$.

(Bootstrap-$t$ Interval) A refinement that studentizes the bootstrap: compute $Z^{\ast b}=(T_n^{\ast b}-T_n)/{\widehat{\text{SE}}}^{\ast b}$ for each resample, then set the interval as $T_n\pm {\hat{q}}_{1-\alpha /2}\cdot \widehat{\text{SE}}(T_n)$. This achieves second-order accuracy: coverage error is $O(n^{-1})$ rather than $O(n^{-1/2})$.

(Number of Resamples) For standard error estimation, $B=200$–$500$ resamples typically suffice. For confidence intervals, $B\ge 1000$ (often $B=10,000$) is recommended to reduce Monte Carlo error in the tail quantiles.

(Parametric Bootstrap) If a parametric model $F_{\theta }$ is assumed, one can estimate $\hat{\theta }$ from the data and resample from $F_{\hat{\theta }}$ instead of ${\hat{F}}_n$. This is more efficient when the model is correct but less robust to misspecification.

Examples

Bootstrap Standard Error of the Median

A sample of $n=50$ observations has sample median $\tilde{x}=3.7$. Unlike the mean, the sampling distribution of the median has no simple closed-form standard error.

Bootstrap Procedure $B=10,000$ bootstrap samples of size $50$ with replacement from the observed data. Compute the median ${\tilde{x}}^{\ast b}$ of each. The bootstrap standard error is ${\widehat{\text{SE}}}_{\text{boot}}=\sqrt{\frac{1}{B-1}{\sum }_{b=1}^B({\tilde{x}}^{\ast b}-{\overline{\stackrel{~}{x}}}^{\ast }{)}^2},$ where ${\overline{\stackrel{~}{x}}}^{\ast }=\frac{1}{B}{\sum }_{b=1}^B{\tilde{x}}^{\ast b}$. If the resulting SE is $0.42$, an approximate $95\%$ confidence interval for the population median is $3.7\pm 1.96\times 0.42=[2.88,4.52]$.

Draw

Bootstrap Confidence Interval for Correlation

Given $n=30$ paired observations $(X_i,Y_i)$, the sample correlation is $\hat{\rho }=0.65$. Construct a $95\%$ bootstrap percentile interval for $\rho$.

Procedure $B=10,000$ resamples, draw $30$ pairs with replacement from the observed data and compute ${\hat{\rho }}^{\ast b}$. Sort the bootstrap correlations and take the $2.5$-th and $97.5$-th percentiles. If these are $0.38$ and $0.82$, the $95\%$ percentile interval is $[0.38,0.82]$. Note this interval is not symmetric about $\hat{\rho }=0.65$, reflecting the skewness of the correlation's sampling distribution near the boundary.

For each of