Note
Weight initialization determines the starting values of a neural network’s parameters before training begins. Although learning updates weights through gradient-based optimization, the initialization strongly affects optimization stability, convergence speed, and whether deep networks can be trained at all.
A primary reason weights must be initialized randomly is to break symmetry between neurons. If all neurons in a layer start with identical weights, they compute identical outputs and receive identical gradients, so they remain indistinguishable throughout training. Random initialization ensures neurons start differently and can specialize. Bias terms are typically initialized to zero because they do not create symmetry issues in the same way weights do.
Beyond symmetry, initialization must control the scale of activations and gradients across layers. Consider a pre-activation of the form
[
a_m^{(l)} = \sum_{i=1}^{M^{(l-1)}} W_{mi}^{(l)} z_i^{(l-1)}.
]
If the variance of the weights is too large, activations (and gradients) can explode as depth increases; if too small, they can vanish. Modern initialization schemes are designed so that the variance of activations remains approximately constant across layers.A simple approach is generic random initialization, sampling weights from a zero-mean distribution such as or . While this breaks symmetry, the choice of is critical and depends on layer width, and naive choices often lead to instability in deep networks.
Xavier (Glorot) initialization improves on this by explicitly balancing variance in both the forward and backward passes. It is well suited for approximately symmetric activations such as
tanhorsigmoid. The variance is chosen using both fan-in and fan-out:
[
\mathrm{Var}(W) = \frac{2}{M^{(l-1)} + M^{(l)}}.
]
This leads to common choices such as
[
W \sim \mathcal{N}!\left(0,\frac{2}{M^{(l-1)} + M^{(l)}}\right)
\quad\text{or}\quad
W \sim \mathrm{Uniform}!\left[-\sqrt{\frac{6}{M^{(l-1)} + M^{(l)}}},\sqrt{\frac{6}{M^{(l-1)} + M^{(l)}}}\right].
]He initialization adapts this idea to ReLU-type activations. Since ReLU zeroes out roughly half of its inputs, the variance must be scaled up to compensate. He initialization preserves variance by using only the fan-in:
[
\mathrm{Var}(W) = \frac{2}{M^{(l-1)}}.
]
Typical choices are
[
W \sim \mathcal{N}!\left(0,\frac{2}{M^{(l-1)}}\right)
\quad\text{or}\quad
W \sim \mathrm{Uniform}!\left[-\sqrt{\frac{6}{M^{(l-1)}}},\sqrt{\frac{6}{M^{(l-1)}}}\right].
]
This is the standard initialization for deep ReLU networks.In summary, random initialization breaks symmetry, while Xavier and He initialization additionally preserve variance across layers. Xavier is appropriate for
tanh-like activations, He is appropriate for ReLU-like activations, and good initialization is essential for stable and effective training of deep neural networks.