5.3.3. Adaptive Learning Rates

Adaptive Learning Rates

Adaptive learning rate algorithms are extensions of gradient descent that automatically adjust the learning rate $\eta$ for each parameter during training. Instead of using a single global learning rate, they maintain individual learning rates that change over time based on gradient history in order to reduce oscillations and improve convergence by moving quickly along dimensions where gradients are small, and slowing down where they are large.

Formally, these methods still follow the same update rule,
$w_i^{(t)}=w_i^{(t-1)}-{\eta }_i^{(t)}\frac{\partial L}{\partial w_i},$
but the learning rate ${\eta }_i^{(t)}$ now depends on the parameter’s past gradients.

1. AdaGrad (Adaptive Gradient) keeps track of the cumulative sum of squared gradients, and divides the base learning rate by the square root of this accumulated quantity:
   ${\eta }_i^{(t)}\frac{\eta }{\sqrt{r_i^{(t)}+ϵ}},\text{such that}{r}_i^{(t)}=r_i^{(t-1)}+{\left(\frac{\partial L}{\partial w_i}\right)}^2.$
   This approach quickly decays learning rate for frequently updated weights, but may shrink ${\eta }_i$ too much, halting learning over time.
   ⠀
2. RMSProp modifies AdaGrad to avoid the problem of vanishing learning rates by focusing on recent rather than cumulative gradient history. It replaces the sum of squared gradients with an exponentially decaying moving average, such that
   $r_i^{(t)}=\beta r_i^{(t-1)}+(1-\beta ){\left(\frac{\partial L}{\partial w_i}\right)}^2,$
   Here,$\beta$ (typically around $0.9$) determines how much weight is given to the past. By reacting primarily to recent gradients, RMSProp maintains a stable step size even in non-stationary settings such as recurrent networks or online learning tasks.
   ⠀
3. Adam (Adaptive Moment Estimation) builds on RMSProp by incorporating momentum, effectively combining the first and second moments of the gradient. It keeps one moving average of the gradients themselves and another of their squares:
   $s_i^{(t)}={\beta }_1{s}_i^{(t-1)}+(1-{\beta }_1)\frac{\partial L}{\partial w_i},r_i^{(t)}={\beta }_2{r}_i^{(t-1)}+(1-{\beta }_2){\left(\frac{\partial L}{\partial w_i}\right)}^2.$
   Because both averages are initially biased toward zero, Adam applies a correction