Gradient descent iteratively updates model parameters by moving opposite the loss gradient, with variants like SGD, Adam, and AdamW differing in how they scale and accumulate gradients. Momentum adds a velocity term that smooths updates, helping the optimizer bypass shallow local minima and saddle points.
Step‑by‑step workflow
1. Compute the mini‑batch loss L(θ) and its gradient g = ∇θ L.
2. SGD: θ ← θ - η·g.
3. Momentum SGD: v ← μ·v - η·g; θ ← θ + v (default μ=0.9).
4. Adam: maintain first‑moment m and second‑moment v
m ← β1·m + (1-β1)·g
v ← β2·v + (1-β2)·g²
bias‑correct, then θ ← θ - η·m̂/(√v̂+ε). Typical β1=0.9, β2=0.999, ε=1e-8.
5. AdamW: same as Adam but weight decay λ applied directly: θ ← θ - η·λ·θ before the Adam update.
Quick comparison
| Optimizer | Learning‑rate schedule | Weight decay handling | Typical use |
|-----------|------------------------|-----------------------|-------------|
| SGD | constant or step LR | L2 via torch.optim.SGD(..., weight_decay=λ) | Large‑scale vision, when exact convergence matters |
| Momentum | same as SGD | same as SGD | Faster convergence on noisy gradients |
| Adam | often cosine‑anneal | L2 mixed with Adam’s update (weight_decay flag) | NLP, Transformers, quick prototyping |
| AdamW | same as Adam | decoupled decay (optim.AdamW) | State‑of‑the‑art BERT/GPT training |
PyTorch example (v2.4)
import torch, torch.nn as nn, torch.optim as optim
model = nn.Linear(784, 10)
optimizer = optim.AdamW(
model.parameters(),
lr=3e-4,
betas=(0.9, 0.999),
weight_decay=0.01,
eps=1e-8
)When to prefer each variant
- Use SGD + momentum if you need reproducible, long‑run training on massive image datasets.
- Choose Adam for rapid convergence on moderate‑size data or when per‑parameter adaptivity matters.
- Switch to AdamW for transformer‑scale models; decoupled decay prevents the learning‑rate from being implicitly reduced by L2 regularization.