8.2.3. Parameter-Efficient Tuning

• Low Rank Approximation (LoRA) injects low-rank trainable matrices into selected weight projections while freezing the original parameters. For a weight matrix $W\in {\mathbb{R}}^{d\times k}$, LoRA decomposes the update into a rank-$r$ factorization:

$W^{\prime }=W+\Delta W,\Delta W=BA,B\in {\mathbb{R}}^{d\times r},A\in {\mathbb{R}}^{r\times k},$
with $r\ll \min (d,k)$. Only $A$ and $B$ are optimized, yielding a parameter cost of $O(r(d+k))$, a dramatic reduction compared to full fine-tuning. LoRA preserves the forward pass semantics of the pretrained model while enabling efficient task adaptation. It is inserted in attention projections (typically $W_Q,W_V$), or MLP layers.