2.1. Language Models

Lanugage Models

A language model is a probabilistic model of a token sequence, assigning a probability to text by factorizing a joint distribution over tokens. Given a sequence $x_{1:T}=(x_1,\dots ,x_T)$, an autoregressive language model defines the chain rule
$\mathbb{P}(x_1,x_2,\dots ,x_T)={\prod }_{t=1}^T\mathbb{P}(x_t\mid x_{1:t-1}).$
Equivalently, it models the next-token distribution $\mathbb{P}(x_{t+1}\mid x_{1:t})$, which can be used both to evaluate text (via likelihood) and to generate text (by sampling tokens sequentially).

In practice, directly representing the full joint distribution over all sequences is intractable in both memory and compute. Model architectures therefore impose structure – most commonly a factorization or conditional-independence assumptions – to approximate the true distribution.

decoding