Gated Linear Unit (GLU)
A gated linear unit (GLU) is a feed-forward transformation that introduces an explicit multiplicative gate to control which features pass through a layer. Given an input vector , GLU first computes two affine projections: a value stream and a gate stream. The gate stream is passed through a squashing nonlinearity (typically sigmoid) and then multiplied elementwise with the value stream, producing a feature-wise filter:
Intuitively, each coordinate of acts like a learned “on/off” knob in that modulates the corresponding coordinate of the value projection.
- Compared to a plain MLP nonlinearity, gating can improve expressivity because it allows the layer to represent conditional feature selection (some dimensions are suppressed while others pass) rather than applying the same pointwise nonlinearity everywhere.
In Transformer feed-forward blocks, GLU-style layers are often implemented by projecting to a wider hidden dimension, splitting it into two halves (value/gate), applying the gate nonlinearity, multiplying, and then projecting back to the model dimension.