1.3.1 Standard Error

Standard Error

The standard error of an estimator ${\hat{\Theta }}_n=g(X_1,\dots ,X_n)$ is the standard deviation of its sampling distribution:
$\text{SE}({\hat{\Theta }}_n)=\sqrt{\text{Var}({\hat{\Theta }}_n)}.$
For the sample mean ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$ of i.i.d. observations with population standard deviation $\sigma$, this simplifies to
$\text{SE}({\bar{X}}_n)=\frac{\sigma }{\sqrt{n}}.$
When $\sigma$ is unknown (the typical case), it is replaced by the sample standard deviation $s$, giving the estimated standard error $\widehat{\text{SE}}({\bar{X}}_n)=\frac{s}{\sqrt{n}}$.

When sampling without replacement from a finite population of size $N$, the standard error includes the finite population correction:
$\text{SE}({\bar{X}}_n)=\frac{\sigma }{\sqrt{n}}\sqrt{\frac{N-n}{N-1}}.$
This factor is negligible when $n\ll N$ but becomes significant when the sampling fraction $n/N$ is large.

Standard Error vs. Standard Deviation $\sigma$ measures the spread of individual observations in the population. The standard error measures the spread of a statistic (like $\bar{X}$) across repeated samples. As $n$ grows, $\sigma$ stays fixed but $\text{SE}\to 0$, reflecting that larger samples yield more precise estimates.

The standard deviation

Intuition

The standard error quantifies how much an estimate would vary if we repeated the entire experiment. Imagine drawing a sample of size $n$, computing $\bar{X}$, then drawing a new sample and computing $\bar{X}$ again, and so on. The standard error is the standard deviation of all these $\bar{X}$ values.

The $1/\sqrt{n}$ dependence is the key insight: doubling precision requires quadrupling the sample size. This follows directly from the variance of a sum of independent random variables—each observation contributes ${\sigma }^2$ to the variance of the total, so $\text{Var}(\bar{X})={\sigma }^2/n$.

Properties

(Rate of Decay) $\text{SE}({\bar{X}}_n)=O(1/\sqrt{n})$, so the standard error halves when the sample size is quadrupled.

(CLT Connection) By the central limit theorem, $\frac{{\bar{X}}_n-\mu }{\text{SE}({\bar{X}}_n)}\stackrel{d}{\to }\mathcal{N}(0,1)$ as $n\to \infty$, which justifies using normal-based confidence intervals.

(General Estimators) For estimators other than the sample mean—proportions, regression coefficients, medians—the standard error takes different forms but always measures sampling variability. When no closed-form expression exists, the bootstrap provides a nonparametric estimate of SE.

Examples

Poll Margin of Error

A poll of $n=1000$ voters finds $\hat{p}=0.52$ support for a candidate. The estimated standard error of the proportion is $\widehat{\text{SE}}(\hat{p})=\sqrt{\hat{p}(1-\hat{p})/n}$.

Calculation $\widehat{\text{SE}}(\hat{p})=\sqrt{\frac{0.52\times 0.48}{1000}}=\sqrt{\frac{0.2496}{1000}}\approx \mathrm{0.0158.}$ The "margin of error" reported by polls at the $95\%$ level is $1.96\times 0.0158\approx 0.031$, or about $\pm 3.1$ percentage points. This means the true proportion plausibly lies in $[0.489,0.551]$—the race is too close to call.