About this paper
FOAM is a lightweight SOAP variant that uses input-only preconditioning and a factored Adam-moment estimator to match strong second-order and Muon baselines in LLM pre-training while using half the optimizer state and running at Muon-class speed.
Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored One-sided Adam-Moment (FOAM). The path to practical impact is twofold: first, FOAM employs input-only preconditioning, dropping the output-side Kronecker factor to reduce memory and remove one eigendecomposition. The…
second-order optimizersShampooSOAPAdamWtransformer trainingoptimizer-state memoryeigendecompositioninput-only preconditioningoutput-side Kronecker factorKronecker structure of Hessian
- Upvotes
- 9
- Authors
- 4
- Published
- Nov 12, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
Factored One-sided Adam-Moment for Practical and Scalable SOAP was first seen on Hugging Face · Daily papers on Nov 13, 2026. Since then it has appeared in 36 checks on 4 different days. Its best position was #16 of 30, reached on Nov 13, 23:28 UTC. It is #17 right now, holding steady at #17 over the last 6 checks.