4.1. Neural Network Normalization

Normalization in Neural Networks

Normalization is an optimization method in neural networks that stabilizes training by controlling the scale and distribution of activations or inputs. It does not directly reduce model capacity (i.e., it is not a regularizer), but instead acts as a reparameterization of the network that improves gradient flow, reduces sensitivity to initialization, and accelerates convergence.

Given a vector $x$, normalization computes:
$\mu =\frac{1}{|S|}{\sum }_{i\in S}{x}_i,{\sigma }^2=\frac{1}{|S|}{\sum }_{i\in S}(x_i-\mu {)}^2,$
where the set $S$ depends on the normalization type. Then, ${\hat{x}}_i=\frac{x_i-\mu }{\sqrt{{\sigma }^2+ϵ}}$, such that the learned affine transform $y_i=\gamma {\hat{x}}_i+\beta$ restores representational flexibility.

Normalization techniques differ in where the mean and variance are computed:

• BatchNorm uses mini-batch statistics during training, and running averages at inference. It is performed across the batch dimension $B$ for each feature, such that
  ${\mu }_j=\frac{1}{B}{\sum }_{b=1}^B{x}_{bj},{\sigma }_j^2=\frac{1}{B}{\sum }_{b=1}^B(x_{bj}-{\mu }_j{)}^2.$
  Although BatchNorm speeds up training, makes MLPs and CNNs less sensitive to initialization, and adds mild regularization due to batch noise, it performs poorly in sequence models since batch dependence is undesirable.
  ‎
• LayerNorm is independent of batch size, and is executed within a single sample, such that
  ${\mu }_b=\frac{1}{d}{\sum }_{j=1}^d{x}_{bj},{\sigma }_b^2=\frac{1}{d}{\sum }_{j=1}^d(x_{bj}-{\mu }_b{)}^2.$
  It is stable for variable batch sizes and has no regularization effect, and therefore is preferred across sequence models.