4. Activation Functions

Activation Functions

To achieve non-linearity, MLPs use activation functions that transform pre-activations $a$ into outputs $z=h(a)$:

• The Sigmoid maps $\mathbb{R}$ to $(0,1)$, so it naturally parameterizes Bernoulli probabilities:
  $\sigma (a)=\frac{1}{1+e^{-a}}.$
  It is most common in binary-classification output layers (or gating mechanisms) rather than deep hidden layers, because it saturates (i.e., its derivative becomes tiny when $a$ is very positive or very negative).
  ‎
• The Hyperbolic Tangent (tanh) maps $\mathbb{R}$ to $(-1,1)$ and is centered at zero, which often makes optimization more stable than sigmoid in settings where saturation is acceptable:
  $\tanh (a)=\frac{e^a-e^{-a}}{e^a+e^{-a}}.$
  Like sigmoid, it saturates for large $|a|$, which can slow learning in deep networks unless mitigated by normalization or careful initialization.
  ‎
• Softmax takes a vector of logits $a\in {\mathbb{R}}^K$ and returns a probability distribution over $K$ classes:
  $\mathrm{softmax}(a{)}_k=\frac{e^{a_k}}{{\sum }_{j=1}^K{e}^{a_j}}.$
  It is the standard multiclass output activation because it produces nonnegative components that sum to one, enabling training with cross-entropy loss. Numerically stable implementations use a shift by ${\max }_k{a}_k$ since adding a constant to all logits does not change the output distribution.
  ‎
• The Rectified Linear Unit (ReLU) is piecewise linear: it transmits positive values unchanged and clamps negative values to zero, such that
  $\mathrm{ReLU}(a)=\max (0,a).$
  This avoids saturation on the positive side and yields sparse activations, which often improves gradient flow and training speed. Its main failure mode is the dying ReLU problem, where units can get stuck outputting zero if their weighted sum plus bias stays negative for essentially every example.
  • ReLU then always outputs zero, and because the gradient through ReLU is zero on the negative side, backprop sends no update signal to that neuron’s weights or bias, so it never shifts back into the active (positive) region.
    ‎
• The Gaussian Error Linear Unit (GELU) is a smooth alternative to ReLU that down-weights negative inputs continuously rather than hard-thresholding them:
  $\mathrm{GELU}(a)=a\cdot \Phi (a)\approx 0.5a(1+\tanh (\sqrt{2/\pi }(a+0.044715a^3))).$
  It can be interpreted as multiplying $a$ by a soft gate $\Phi (a)$ (the standard normal CDF), which tends to provide smoother gradients near zero and is widely used in Transformer feed-forward blocks.
• The Sigmoid-Weighted Linear Unit (SiLU/Swish) is a smooth, non-monotone activation defined as the input multiplied by its sigmoid gate:
  $\mathrm{SiLU}(a)=a\cdot \sigma (a)=\frac{a}{1+e^{-a}}.$
  It behaves like a softened ReLU: negative values are not hard-clipped, and gradients change smoothly around zero. It is commonly used in modern architectures (and appears inside SwiGLU variants of feed-forward networks).