SwiGLU
SwiGLU is a GLU-family feed-forward variant where the gating nonlinearity is SiLU (also called Swish) rather than sigmoid. The typical Transformer FFN parameterization uses two “up” projections from to : one is passed through SiLU to become the gate, and the other provides the value stream. Their elementwise product is then projected back down:
Relative to sigmoid-gated GLU, SiLU gating is smooth and does not strictly saturate into a hard mask, which can yield better gradient flow and empirical performance in modern LLMs. In practice, SwiGLU is usually paired with a larger hidden dimension than the classic ReLU/GELU FFN so that the gated product retains sufficient capacity, making it a common default choice in LLaMA-style decoder blocks.