Hypothesis Testing

A hypothesis test is a formal decision procedure for choosing between two competing statements about a data-generating distribution:

• Null hypothesis ($H_0$): a description of the chance process that generates the data, typically asserting that nothing unusual is happening (e.g., a parameter equals a specific value, or two variables are independent).
• Alternative hypothesis ($H_1$ or $H_A$): the assertion that a mechanism other than $H_0$ generated the data.

A test constructs a test statistic $T(X)$ from the data and defines a rejection region $R$ such that $H_0$ is rejected whenever $T(X)\in R$. The test is subject to two types of error:

• Type I error (false positive): rejecting $H_0$ when it is in fact true. The probability of a Type I error is the significance level,
  $\alpha =\mathbb{P}(\text{reject }{H}_0\mid H_0\text{ true}).$
• Type II error (false negative): failing to reject $H_0$ when $H_1$ is true. Its probability is denoted $\beta$, and $1-\beta$ is the power of the test—the probability of correctly detecting a true effect.

The p-value is the probability, under $H_0$, of observing a test statistic as extreme as or more extreme than the one actually observed:
$p={\mathbb{P}}_{H_0}(T(X)\ge T(x_{\text{obs}})).$
A small p-value indicates that the observed data would be unusual under $H_0$. The null is rejected when $p\le \alpha$.

Interpretation of the P-Value not the probability that $H_0$ is true. It is the probability of the observed (or more extreme) data assuming $H_0$ is true. A p-value of $0.03$ means that if $H_0$ were true, data this extreme would occur only $3\%$ of the time—not that there is a $3\%$ chance $H_0$ holds.

The p-value is

Intuition

Hypothesis testing formalizes the everyday reasoning of “is this result surprising enough to rule out coincidence?” The null hypothesis represents the skeptic’s position—nothing interesting is happening, any apparent effect is due to chance. The test asks: if the skeptic is right, how likely are we to see data this extreme?

The significance level $\alpha$ controls the false alarm rate: how often we incorrectly declare an effect. The power $1-\beta$ controls the detection rate: how often we correctly identify a real effect. These two errors trade off—lowering $\alpha$ (being more conservative) increases $\beta$ (missing real effects), and vice versa. The art of experimental design lies in choosing $\alpha$, the sample size, and the test statistic to balance these risks.

Properties

(Neyman-Pearson Framework) The classical approach fixes $\alpha$ in advance and seeks the test with maximum power. The Neyman-Pearson lemma shows that the likelihood ratio test is the most powerful test at any given significance level for simple (fully specified) hypotheses.

(Rejection Region) For a given $\alpha$, the rejection region $R$ is the set of test statistic values for which $H_0$ is rejected. Equivalently, $R=\{x:p(x)\le \alpha \}$.

(One-Sided vs. Two-Sided) A one-sided test has $H_1:\theta >{\theta }_0$ (or $\theta <{\theta }_0$), concentrating the rejection region in one tail. A two-sided test has $H_1:\theta \ne {\theta }_0$, splitting $\alpha$ across both tails.

(Connection to Confidence Intervals) Rejecting $H_0:\theta ={\theta }_0$ at level $\alpha$ is equivalent to ${\theta }_0$ falling outside a $(1-\alpha )$ confidence interval for $\theta$. This duality links hypothesis testing and interval estimation.

Examples

Coin Bias Test

A coin is flipped $n=100$ times, yielding $X=62$ heads. Test $H_0:p=0.5$ against $H_1:p>0.5$ at significance level $\alpha =0.05$.

Solution $H_0$, $X\sim \text{Bin}(100,0.5)$ with $\mathbb{E}[X]=50$ and $\text{SD}(X)=5$. The test statistic is $Z=\frac{X-n{p}_0}{\sqrt{n{p}_0(1-p_0)}}=\frac{62-50}{5}=\mathrm{2.4.}$ The p-value is $\mathbb{P}(Z\ge 2.4)\approx 0.0082$. Since $0.0082<0.05$, we reject $H_0$ and conclude there is significant evidence of bias toward heads.

Under

Two-Sample Permutation Test

Two groups of patients receive treatments A and B. Group A ($n_A=30$) has mean recovery time ${\bar{x}}_A=12.3$ days; Group B ($n_B=30$) has ${\bar{x}}_B=14.1$ days. Test $H_0$: the treatment has no effect (recovery times are independent of group assignment).

Approach $H_0$, the group labels are exchangeable. The observed test statistic is $T_{\text{obs}}={\bar{x}}_B-{\bar{x}}_A=1.8$. By randomly permuting the group labels many times and recomputing $T$ each time, we build the null distribution of $T$. The p-value is the proportion of permuted statistics $\ge 1.8$. If this proportion falls below $\alpha$, we reject $H_0$.

Under