9.4. Alignment Methods

Alignment

Large language models are pre-trained with next-token prediction, but this objective alone does not guarantee behavior that is helpful, safe, or aligned with human intent. To shape an LLM’s responses toward desirable outputs, modern practice relies on alignment methods, which adjust the model using supervised demonstrations or human preference signals. These techniques do not change the underlying transformer architecture; instead, they modify the training objective to shift the model’s output distribution toward human-desired behavior.

There are multiple alignment methods in LLM post-training:

• Supervised Fine-Tuning (SFT) adapts a pretrained LLM to follow instructions by training on curated input-output pairs $(x,y)$. The decoder minimizes a conditional next-token likelihood:
  ${\mathcal{L}}_{\text{SFT}}=-{\sum }_{t=1}^{|y|}\log p_{\theta }(y_t\mid y_{<t},x).$
  SFT improves instruction-following and structured responses, but it does not incorporate explicit human preferences; it simply encourages imitation of high-quality demonstrations. It typically serves as the initialization for PEFT methods.
  ‎
• Synthetic Data Augmentation uses model-generated or model-transformed examples to expand the alignment dataset, allowing broader behavioral shaping without extensive human labeling. A pretrained or partially aligned model produces synthetic demonstrations $(x,y_{\text{synthetic}})$, which are then filtered, ranked, or curated (sometimes with additional models) before being used as supervision using the dataset ${\mathcal{D}}_{\text{aug}}=\{(x_i,y_i^{\text{synthetic}}){\}}_{i=1}^M$. Training then proceeds with standard supervised objectives:
  ${\mathcal{L}}_{\text{aug}}=-{\sum }_{(x,y)\in {\mathcal{D}}_{\text{aug}}}\log p_{\theta }(y\mid x).$
  Synthetic augmentation is widely used in modern LLM pipelines (e.g., self-instruct, bootstrapping, teacher–student distillation) because it amplifies the alignment signal without additional human labor. It does not enforce preferences directly but reshapes the model’s instruction-following behavior by enriching the supervised training distribution.
  ‎
• Reinforcement Learning from Human Feedback (RLHF) encourages outputs that humans prefer while constraining the model to remain close to its SFT behavior through a two-step procedure,
  1. Reward model training: given ranked pairs $(y^{+},y^{-})$ for a prompt $x$, train a scalar reward model $R_{\varphi }$ by maximizing
     $\mathbb{P}(y^{+}\succ y^{-})=\sigma (R_{\varphi }(x,y^{+})-R_{\varphi }(x,y^{-})),$
     where $\sigma$ is the logistic function.
  2. Policy optimization: fine-tune the LLM (policy ${\pi }_{\theta }$) with reinforcement learning to maximize expected reward using PPO or related methods:
     ${\max }_{\theta }{\mathbb{E}}_{y\sim {\pi }_{\theta }(\cdot \mid x)}[R_{\varphi }(x,y)]-\beta \mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{SFT}}).$
     ‎
• Direct Preference Optimization (DPO) eliminates RLHF’s reward model and RL loop by directly contrasting preferred and dispreferred responses. Given preference pairs $(y^{+},y^{-})$, DPO minimizes:
  ${\mathcal{L}}_{\text{DPO}}=-\log \frac{\exp \left[\beta \log {\pi }_{\theta }(y^{+}\mid x)-\log {\pi }_{\text{SFT}}(y^{+}\mid x)\right]}{\exp \left[\beta \log {\pi }_{\theta }(y^{+}\mid x)-\log {\pi }_{\text{SFT}}(y^{+}\mid x)\right]+\exp \left[\beta \log {\pi }_{\theta }(y^{-}\mid x)-\log {\pi }_{\text{SFT}}(y^{-}\mid x)\right]}.$