Large Language Model
A large language model (LLM) is a decoder-only transformer trained at scale to model the probability distribution of natural language. LLMs implement an autoregressive factorization of text, learning
where the model predicts each token using only preceding tokens. This constraint is implemented through masked self-attention, which blocks all attention scores to positions when predicting token . During training, the objective is maximum likelihood:
making LLMs next-token predictors.Modern LLMs differ from standard decoders not in architectural complexity but in scale. They contain dozens to hundreds of layers, hidden dimensions in the thousands, and vocabularies in the tens of thousands. Pre-training uses corpora containing trillions of tokens. Scaling all three axes—model size, dataset size, and compute—yields predictable improvements in performance, a phenomenon formalized by scaling laws.
LLMs are trained in a purely self-supervised pre-training phase using next-token prediction alone, without instructions or task-specific labels. This broad objective induces the model to learn linguistic structure, factual associations, semantic regularities, and long-range dependencies. Subsequent stages—such as supervised fine-tuning on instructions or preference optimization (RLHF, DPO)—shape behavior but do not alter the underlying model class.
At inference time, an LLM generates text autoregressively. Given a prefix x_{1:n}, the model samples or selects the next token from p_\theta(\cdot \mid x_{<n}), appends it, and repeats. This iterative process allows the model to produce sentences, paragraphs, and extended reasoning chains, with capabilities arising from the interaction of architectural simplicity and large-scale training.