6.1.3. RNNs

Recurrent Neural Networks

A recurrent neural network (RNN) is a one-dimensional scanning network designed for sequential data: it scans across time rather than across spatial pixels, applying the same parameterized transition function at every position in the sequence. Unlike a CNN, whose local function depends only on the current input patch, an RNN’s local function takes both the current input and the network’s accumulated hidden state as arguments, so each step can carry information forward through arbitrarily many positions in principle.

Given an input sequence $(x_1,x_2,\dots ,x_T)$ with each $x_t\in {\mathbb{R}}^d$, the RNN defines a recurrent update
$h_t=\varphi (W_h{h}_{t-1}+W_x{x}_t+b),$
where $h_t\in {\mathbb{R}}^m$ is the hidden state at time $t$, $\varphi$ is a nonlinearity (typically $\tanh$ or ReLU), and the parameters $\{W_h,W_x,b\}$ are reused at every step. An optional output head produces the prediction ${\hat{y}}_t=W_y{h}_t+b_y$ from the hidden state at each step (or only at the final step, depending on the task). The recurrence runs forward, $t=1,\dots ,T$, with an initial hidden state $h_0$ that is either learned or set to zero.

The parameter sharing across time gives the RNN three properties characteristic of scanning networks:

Sparse connectivity. Each output state $h_t$ depends only on $(h_{t-1},x_t)$, not on the entire input sequence at once. The dependence on earlier inputs is mediated through the hidden state.

Shared connectivity pattern. The same parameters $W_h,W_x,b$ are used at every time step: the function $g(h_{t-1},x_t)$ is the same map regardless of $t$.

Shift equivariance in time. Applying the recurrence to a time-shifted version of the input produces the same hidden states shifted accordingly.

Vanishing and Exploding Gradients $W_h$ repeatedly, so for long sequences the gradient either decays to zero (vanishing) or grows without bound (exploding) depending on the spectral radius of $W_h$. Vanilla RNNs in practice learn dependencies only over short time spans (tens of steps). Gated variants (LSTMs and GRUs) extend the local function $g$ with multiplicative gates that control how much of the previous state to retain, which empirically allows useful gradient flow over hundreds of steps. Modern long-context architectures (transformers) sidestep the recurrence entirely.

Backpropagating through many time steps multiplies gradients by

Intuition

An RNN is a function with internal memory. At every time step it reads a new input, updates its memory based on the previous memory and the new input, and optionally produces an output. The memory (the hidden state $h_t$) is fixed-size, so the network has to learn to selectively retain whatever information from the past is relevant for whatever it will need to do later. The shared parameters mean the same memory-update rule is applied at every step, which is what makes the network recurrent (the same weights are used recurrently, in a loop) and what allows it to handle sequences of arbitrary length.

The structural connection to other scanning networks is that an RNN slides a small parameterized function over the time axis just as a CNN slides a small parameterized function over the spatial axes of an image. The two differ in what the local function takes as input: the CNN sees only a fixed-size spatial window of the input, while the RNN sees one input plus all the previous inputs summarized through the hidden state. This makes the RNN’s effective receptive field unbounded in principle (every $h_t$ can depend on every $x_s$ for $s\le t$) but bounded in practice by the gradient-flow problem that limits how far back the network can actually propagate learning signal.

Properties

(Recurrence Equation) The hidden state at time $t$ is a fixed function of the previous hidden state and the current input.

The update $h_t=\varphi (W_h{h}_{t-1}+W_x{x}_t+b)$ is the entire model: running it forward for $t=1,\dots ,T$ produces the full sequence of hidden states. The matrix $W_h\in {\mathbb{R}}^{m\times m}$ governs the temporal coupling (how the previous state influences the new state), $W_x\in {\mathbb{R}}^{m\times d}$ governs the input integration (how much each input feature contributes), and the bias $b\in {\mathbb{R}}^m$ shifts the activation. The nonlinearity $\varphi$ is essential: without it the recurrence would be linear and the network would reduce to a single matrix multiplication of all inputs.

(Shared Parameters Across Time) Reusing $\{W_h,W_x,b\}$ at every step is what makes the RNN parsimonious and shift-equivariant.

If each time step had its own parameters, the parameter count would scale with the sequence length and the network could not generalize across positions. Sharing the parameters means the network learns one update rule that has to work everywhere, mirroring how a CNN learns one filter that has to work over all image patches. Sequences of different lengths can be processed by the same network without changing parameter count.

(Backpropagation Through Time) Gradients flow backward through the recurrence by the chain rule, unfolded over $T$ steps.

To compute gradients of a loss $\mathcal{L}$ at time $T$ with respect to the parameters, the RNN is unrolled into a feedforward network of depth $T$ (one copy of the recurrent block per time step, with shared weights) and standard backpropagation is applied through the unrolled graph. The gradient of $\mathcal{L}$ with respect to $W_h$ accumulates contributions from every time step, and each contribution involves a product of $W_h$ Jacobians from time $T$ back to the contributing step. This product is the source of the vanishing/exploding gradient problem.

(Vanishing/Exploding Gradient) The gradient norm scales like the spectral radius of $W_h$ raised to the unrolling depth.

If $\rho (W_h)<1$, gradients vanish exponentially fast in $T$: earlier time steps receive negligible learning signal, and the network cannot learn long-range dependencies. If $\rho (W_h)>1$, gradients explode and training becomes unstable. The mathematical cause is the recursive multiplication by $W_h$ during backpropagation; the practical cure is gradient clipping (rescale large gradients to a fixed norm) and gated architectures.

(LSTM/GRU) Gated variants replace the simple update with a gated one that lets the network learn which information to keep and which to forget.

The LSTM introduces a separate “cell state” $c_t$ alongside the hidden state, with three gates (input, forget, output) controlling what gets written to, erased from, and read out of the cell. The GRU is a smaller variant with two gates (update, reset). Both architectures use sigmoid-gated multiplicative paths that allow the gradient to flow through the cell state without being multiplied by $W_h$ at every step, dramatically improving long-range dependency learning. LSTMs and GRUs were the dominant sequence models from the late 1990s through the mid-2010s.

(Bidirectional and Stacked Variants) Process the sequence in both directions, or stack multiple recurrent layers.

A bidirectional RNN runs one RNN forward over the sequence and another backward, concatenating the two hidden states at each time step. This gives access to both past and future context at every $t$, useful when the entire sequence is available offline (e.g., transcribing a recorded sentence). A stacked RNN uses the hidden states from one layer as the inputs to the next, building deeper representations. Both extensions are orthogonal, and bidirectional and stacked LSTMs were the workhorses of pre-transformer NLP.

Examples

Sentiment classification

An RNN reads a sentence one word at a time and outputs a single sentiment label at the end. What does each component do?

Solution $x_t\in {\mathbb{R}}^d$. The RNN scans the sentence with $h_t=\tanh (W_h{h}_{t-1}+W_x{x}_t+b)$, accumulating a representation of the sentence so far. After the last word, the final hidden state $h_T$ is fed through a linear-plus-softmax classifier to produce a probability over sentiment classes. Training uses cross-entropy loss at the final step, and gradients flow back through all $T$ unrolled steps to update $W_h,W_x,W_y$ and the embeddings. For long sentences, an LSTM or GRU is used in place of the vanilla RNN to handle the long-range dependencies (e.g., a negation at the start of the sentence flipping the sentiment of the predicate).

Each word is embedded into

Sequence-to-sequence translation

An encoder RNN reads an English sentence and a decoder RNN generates the French translation. How does information flow?

Solution $h_T^{\text{enc}}$ that summarizes the entire input. This vector seeds the initial hidden state of the decoder RNN, which then generates French tokens one at a time, conditioning each output on the previous output and the encoder summary. This encoder-decoder template is the basis of pre-transformer machine translation systems. The bottleneck (the entire sentence has to fit through the single vector $h_T^{\text{enc}}$) is what motivated attention (every decoder step looks back at all encoder states), and ultimately the transformer architecture that dropped the recurrence entirely.

The encoder RNN runs over the English tokens, producing a final hidden state