7.1. Transformers

Transformer

A transformer is a deep neural network built around self-attention, a content-based operation that lets each token compute weighted interactions with every other token. Unlike RNNs, transformers process sequences in parallel, and unlike CNNs, they use global rather than local receptive fields. This makes them highly effective for modeling long-range dependencies across text, images, audio, and multimodal data.

Given an input sequence of token embeddings $X=[x_1,x_2,\dots ,x_T]\in {\mathbb{R}}^{T\times d},$ a transformer layer first linearly projects each token to queries, keys, and values:
$Q=X{W}_Q,K=X{W}_K,V=X{W}_V,$
with parameter matrices $W_Q,W_K\in {\mathbb{R}}^{d\times d_k}$ and $W_V\in {\mathbb{R}}^{d\times d_v}$. Having both $Q$ and $K$ allows for asymmetric relevance relationships, whereas $V$ contains the actual content to aggregate from each token.

The scaled dot-product attention computes, for all positions at once,
$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}\right)V,$
where the matrix $Q{K}^{\top }\in {\mathbb{R}}^{T\times T}$ contains pairwise similarity scores between tokens, scaled by $\sqrt{d_k}$ to improve training by ensuring that the inner product variance stays close to $1$. The row-wise softmax creates a probability distribution over tokens, producing a weighted average of their value vectors.

To capture multiple types of relationships in parallel, transformers apply multi-head attention, running several attention modules in parallel and concatenating their outputs:
$\mathrm{MultiHead}(X)=\left[{\mathrm{head}}_1\Vert \cdots \Vert {\mathrm{head}}_H\right]W_O,$
with $W_O\in {\mathbb{R}}^{(H{d}_v)\times d}$. Different heads specialize to different structures (syntax, coreference, long-range links) giving layers richer representational capacity. Large models often use grouped-query attention (GQA) to share keys and values across heads to reduce memory and compute.

Each transformer layer then alternates multi-head attention with a position-wise feedforward network (FFN) and residual connections:
$\begin{array}{rl}Z& =X+\mathrm{MultiHead}(X),\\ Y& =Z+\mathrm{FFN}(Z),\end{array}$
with normalization applied before and after each block. Since self-attention is permutation-invariant, positional encodings are added to preserve sequence order.

Quadratic Complexity $Q{K}^{\top }\in {\mathbb{R}}^{T\times T}$ requires $O(T^2)$ computation and memory in the sequence length $T$. This becomes a bottleneck for long sequences—both slow and memory-intensive. Transformers also cannot generalize beyond a pre-defined maximum sequence length (determined by the positional encoding scheme), unlike RNNs which process arbitrary-length inputs sequentially.

The attention matrix

Intuition

Self-attention can be understood as a soft, differentiable lookup table: the query says “here is what I’m looking for,” each key says “here is what I have,” and the attention weights determine how much each value contributes to the output. Unlike a hard lookup (which returns a single match), soft attention returns a weighted average over all values. In the RNN seq2seq setting, queries come from the decoder hidden state and keys/values from the encoder; in a transformer’s self-attention, all three come from the same sequence, letting every token attend to every other token simultaneously.

The FFN is critical because self-attention on its own is a linear operation over value vectors: stacking more self-attention layers without nonlinearities would just re-average the same values. The FFN introduces element-wise nonlinearity, giving the network the capacity to transform token representations beyond what weighted averaging can achieve. In practice, the FFN accounts for the majority of a transformer’s parameters and FLOPs—at GPT-3 scale (~175B parameters), roughly 80% of floating-point operations occur in the FFN layers.

Residual connections ensure an uninterrupted gradient flow through the full depth of the network (analogous to the cell state highway in LSTMs), while layer normalization keeps activations on a stable scale. Together with multi-head attention, these components form a clean, modular architecture that is highly parallelizable on GPUs—the key advantage over recurrent architectures.

Properties

Input Embedding

Given a sequence of tokens $x_1,\dots ,x_n$ from a vocabulary of size $V$, the input layer produces $d$-dimensional vectors:
${\tilde{\mathbf{x}}}_i={\mathbf{E}}_{x_i}+\mathrm{PE}(i)\in {\mathbb{R}}^d,$
where $\mathbf{E}\in {\mathbb{R}}^{V\times d}$ is a learned embedding lookup table and $\mathrm{PE}(i)$ is the positional encoding for position $i$.

Feedforward Network (FFN)

(FFN Formula) The position-wise feedforward network applies a two-layer MLP independently to each token:

$\mathrm{FFN}({\mathbf{x}}_i)=\mathrm{ReLU}({\mathbf{x}}_i{\mathbf{W}}_1+{\mathbf{b}}_1){\mathbf{W}}_2+{\mathbf{b}}_2,$
where ${\mathbf{W}}_1\in {\mathbb{R}}^{d\times d_{\text{ff}}}$, ${\mathbf{b}}_1\in {\mathbb{R}}^{d_{\text{ff}}}$, ${\mathbf{W}}_2\in {\mathbb{R}}^{d_{\text{ff}}\times d}$, ${\mathbf{b}}_2\in {\mathbb{R}}^d$. The inner dimension $d_{\text{ff}}$ is typically $4d$ (e.g., $d_{\text{ff}}=4096$ for $d=1024$ in the original transformer). Modern architectures replace ReLU with SwiGLU or GELU-GLU variants.

(Why FFN Is Needed) Self-attention computes a weighted sum of value vectors: ${\mathbf{z}}_i={\sum }_j{\alpha }_{i,j}{\mathbf{v}}_j$. This is a linear operation over the values. Without the FFN, stacking multiple self-attention layers would only produce different weighted averages of the same value space—no element-wise nonlinear transformation is ever applied. The FFN breaks this by applying a nonlinear transformation to each token independently, giving the network the capacity to learn complex per-token features.

(FFN Operates Per-Token) Unlike self-attention, which models dependencies across tokens, the FFN and layer normalization operate independently on each token. Self-attention is the only component in a transformer layer that mixes information between different positions.

Residual Connections and Layer Normalization

(Residual Connection) Each sub-layer (attention or FFN) is wrapped in a residual connection: $\mathrm{Residual}(x,f(x))=x+f(x)$. This prevents vanishing gradients by providing a direct additive path for gradient flow, and allows $f$ to learn the residual difference from the identity rather than the full mapping.

(Layer Normalization) After each residual addition, layer normalization is applied: $y=\frac{x-\mathbb{E}[x]}{\sqrt{\mathrm{Var}[x]+ϵ}}\times \gamma +\beta$, where $\gamma ,\beta$ are learned scale and shift parameters. This normalizes activations to a consistent range, stabilizing training.

Self-Attention Complexity

(Time and Space) For a sequence of length $T$ with model dimension $d$, a single self-attention layer requires $O(T^{2}d)$ time and $O(T^2+Td)$ memory. The $T^2$ term dominates for long sequences. By contrast, a recurrent layer is $O(T{d}^2)$ in time but $O(Td)$ in memory—cheaper in memory for long sequences but inherently sequential.

(Parallelism) Unlike RNNs where the forward pass at position $t$ depends on position $t-1$ (yielding $O(T)$ sequential operations), self-attention computes all positions simultaneously in $O(1)$ sequential steps. This is the fundamental reason transformers train much faster on modern GPU hardware.

Examples

Transformer Layer Forward Pass

Given input $X\in {\mathbb{R}}^{T\times d}$ and a single-head transformer layer with $d_k=d_v=d$:

Step-by-step computation

1. Self-attention: Compute $Q=X{W}_Q$, $K=X{W}_K$, $V=X{W}_V$, then $A=\mathrm{softmax}(Q{K}^{\top }/\sqrt{d_k})V$.
2. Residual + Norm: $Z=\mathrm{LayerNorm}(X+A)$.
3. FFN: $F=\mathrm{ReLU}(Z{\mathbf{W}}_1+{\mathbf{b}}_1){\mathbf{W}}_2+{\mathbf{b}}_2$.
4. Residual + Norm: $Y=\mathrm{LayerNorm}(Z+F)$.

The output $Y\in {\mathbb{R}}^{T\times d}$ has the same shape as the input, so layers can be stacked: the $(L+1)$-th layer takes the $L$-th layer’s output as input.