Bayesian statistics treats unknown parameters as random variables with probability distributions and uses Bayes’ rule to combine a prior distribution with a likelihood to yield a posterior. This differs from frequentist inference, which treats parameters as fixed unknowns and reasons about the distribution of estimators over hypothetical repeated sampling. The payoff is a single coherent object — the posterior — that answers every inferential question (point estimates, credible intervals, model comparisons, predictions) by taking expectations or quantiles.

The challenge is computation. Closed-form posteriors exist only for conjugate models; most real problems require numerical methods, which drives the second half of the folder: MCMC, Gaussian processes, and Laplace approximations.

Prerequisites

• 1. Probability and 2. Random Variables — the probability machinery Bayes’ rule operates on.
• 1.2.1 Maximum Likelihood Estimation — the frequentist counterpart Bayesian inference is often compared against.
• 5.1 Markov Chains — background for MCMC.

Contents

Foundations

• 3.1 Bayesian Framework — prior, likelihood, posterior, Bayes’ rule, and the likelihood principle (stopping-rule invariance).
• 3.2 Credible Intervals — posterior-probability intervals and contrast with frequentist confidence intervals.
• 3.3 Bayes Factors — marginal-likelihood ratio for model comparison; Jeffreys-Lindley paradox.
• 3.4 Cox-Jaynes Derivation — axiomatic derivation of probability from plausibility.

Conjugate Analysis

• 3.5 Conjugate Families — subfolder index over conjugate prior-likelihood pairs.
• 3.6 Hierarchical Models — two-level hierarchies, shrinkage, James-Stein, empirical Bayes.
• 3.7 Marginal Likelihood — evidence computation, digamma, hyperparameter tuning.
• 1.5.2 Bayesian Bootstrap — Dirichlet(1, …, 1) posterior over weights (lives under 1. Statistical Inference/1.5 Bootstrapping/).

Bayesian Regression

• 3.8 Bayesian Regression — subfolder index over linear, autoregressive, logistic, and Poisson models.

Approximate Inference

• 3.9 Laplace Approximation — Gaussian posterior around the MAP via the Hessian.
• 3.10 Gaussian Processes — subfolder over kernel-based priors on functions.

MCMC

• 3.11 MCMC — subfolder over sampling-based posterior inference, including Metropolis-Hastings, Gibbs, MALA, and HMC.

Recommended Path

A reader with no prior Bayesian background should start with 3.1 Bayesian Framework and 3.5.1 Beta-Binomial Model to see the machinery on the simplest nontrivial example, then 3.2 Credible Intervals to understand how to summarize a posterior, and finally 3.11 MCMC to see how the same framework extends to models where the posterior is not analytically tractable.