Adam

Adam (KB14) (Adaptive Moment Estimation) is one of the most widely used optimization algorithms in deep learning, owing to its ability to combine the advantages of AdaGrad and RMSProp.

AdaGrad assigns a specific learning rate to each parameter, making it effective in the presence of sparse gradients. RMSProp, on the other hand, adapts the learning rate according to the gradient magnitude, making it suitable for online and nonstationary scenarios.

Adam extends these approaches by introducing an adaptive estimate of both the first-order moment (the mean of the gradients) and the second-order moment (the variance of the gradients). In particular, for each model $\boldsymbol\beta$, Adam maintains two variables:

• $m_t$: estimate of the first-order moment (mean of the gradients);
• $v_t$: estimate of the second-order moment (variance of the gradients).

These estimates are updated iteratively as follows:

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) \nabla S(\boldsymbol\beta_t)$$ (4.42)

$$v_t = \beta_2 v_{t-1} + (1 - \beta_2) \left( \nabla S(\boldsymbol\beta_t) \right)^2$$ (4.43)

where $\beta_1$ and $\beta_2$ are hyperparameters (not to be confused with $\boldsymbol\beta$, which are the model parameters) that control exponential decay (typically $\beta_1 = 0.9$, $\beta_2 = 0.999$).

Since $m_t$ and $v_t$ are initialized to zero, the first iterations are underestimated. To correct this bias, the following are computed:

$$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}$$ (4.44)

The parameters are then updated according to:

$$\boldsymbol\beta_{t+1} = \boldsymbol\beta_t - \gamma \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \varepsilon}$$ (4.45)

where $\gamma$ is the learning rate and $\varepsilon$ is a numerical-stabilization term (typically $\varepsilon = 10^{-8}$).

Adam is particularly effective in scenarios with noisy data, sparse gradients, or nonstationary objective functions. Thanks to its robustness and ease of use, it is often the default choice for optimizing deep neural networks.