DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule
DeltaMomentum introduces a direction-aware momentum update rule that replaces the fixed-rate exponential moving average (EMA) with a key-value based delta rule, where each direction is forgotten at a rate proportional to its query frequency The gradient of a linear layer naturally decomposes into input (key) and output-side error (value), enabling anisotropic forgetting without matrix inversion DeltaAdamW reaches AdamW's validation loss in up to 46.39% fewer steps at 67M parameters and 22.12% fe
Analysis
TL;DR
- DeltaMomentum introduces a direction-aware momentum update rule that replaces the fixed-rate exponential moving average (EMA) with a key-value based delta rule, where each direction is forgotten at a rate proportional to its query frequency
- The gradient of a linear layer naturally decomposes into input (key) and output-side error (value), enabling anisotropic forgetting without matrix inversion
- DeltaAdamW reaches AdamW's validation loss in up to 46.39% fewer steps at 67M parameters and 22.12% fewer steps at 370M parameters on FineWeb-Edu pretraining
- The method is a drop-in replacement for any optimizer's momentum buffer, transfers across widths under muP, and adds only 22.2%–25.0% extra compute relative to a gated-MLP block's linear cost with no persistent memory overhead
- Gains generalize across architectures (ResNet-18, ViT-Tiny on CIFAR-10) and optimizers (SGD, Muon baseline), with training diagnostics confirming healthier input directions and better gradient tracking
Why It Matters
This work directly addresses a fundamental limitation of modern optimizers—their uniform forgetting rate for all gradient directions—which is increasingly costly as models scale. By embedding anisotropic awareness into the momentum update itself rather than wrapping external processing around it, DeltaMomentum offers a principled, low-overhead improvement that could become a standard upgrade for large-scale training pipelines.
Technical Details
- Core mechanism: The gradient of a linear layer is decomposed into an input-side key and an output-side value. The momentum buffer is updated via the canonical delta rule, so frequently queried directions retain momentum longer while stale directions are cleared faster than EMA allows.
- Theoretical guarantees: Proven to be a valid momentum, to apply input-side curvature correction without matrix inversion, and to outperform EMA in clearing stale directions under both fixed and drifting optima.
- Efficiency: Extra compute is bounded between 22.2% and 25.0% of a gated-MLP block's linear cost, with no persistent memory requirement. The coefficient transfers across model widths under muP (mu-parameterization).
- Empirical results: On FineWeb-Edu pretraining, DeltaAdamW achieves AdamW's validation loss in up to 46.39 ± 4.32% fewer steps at 67M and 22.12 ± 0.80% fewer steps at 370M (three seeds). Gains persist at 1B on a Chinchilla-optimal budget. A tuned Muon baseline underperforms DeltaAdamW at both language-model scales.
- Generalization: Validated on SGD with ResNet-18 and ViT-Tiny on CIFAR-10. Training-time diagnostics confirm the predicted mechanism: better gradient tracking and healthier input directions.
Industry Insight
- DeltaMomentum's drop-in compatibility means practitioners can adopt it immediately across existing optimizer stacks (AdamW, SGD, etc.) without architectural changes, making it a low-risk, high-reward upgrade for large-scale pretraining.
- The observed speedup of up to ~46% at smaller scales suggests significant compute savings at production scale, where even marginal per-step efficiency gains translate to substantial cost reductions.
- The anisotropic forgetting principle could extend beyond language models to vision, multimodal, and reinforcement learning domains where gradient direction frequency is similarly uneven, warranting broader empirical investigation.
Disclaimer: The above content is generated by AI and is for reference only.