Trendoday

Factored One-sided Adam-Moment for Practical and Scalable SOAP

Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still rema

  • Upvotes9
#17Rank now 1
#16Peak rank
4Days on list
289Score · 7d
Nov 13, 2026First seen
Rank · 7 days

About this paper

FOAM is a lightweight SOAP variant that uses input-only preconditioning and a factored Adam-moment estimator to match strong second-order and Muon baselines in LLM pre-training while using half the optimizer state and running at Muon-class speed.

Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored One-sided Adam-Moment (FOAM). The path to practical impact is twofold: first, FOAM employs input-only preconditioning, dropping the output-side Kronecker factor to reduce memory and remove one eigendecomposition. The…

second-order optimizersShampooSOAPAdamWtransformer trainingoptimizer-state memoryeigendecompositioninput-only preconditioningoutput-side Kronecker factorKronecker structure of Hessian
Upvotes
9
Authors
4
Published
Nov 12, 2026

Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.

Trending history

Factored One-sided Adam-Moment for Practical and Scalable SOAP was first seen on Hugging Face · Daily papers on Nov 13, 2026. Since then it has appeared in 36 checks on 4 different days. Its best position was #16 of 30, reached on Nov 13, 23:28 UTC. It is #17 right now, holding steady at #17 over the last 6 checks.

Rank history

Position on Hugging Face · Daily papers at every check (lower is better).

#1#11#20#30 Nov 13Nov 14Nov 14Nov 15Nov 16 Nov 13, 07:28 UTC · #21Nov 13, 09:28 UTC · #22Nov 13, 11:28 UTC · #21Nov 13, 13:28 UTC · #21Nov 13, 15:28 UTC · #21Nov 13, 17:28 UTC · #18Nov 13, 19:28 UTC · #18Nov 13, 21:28 UTC · #17Nov 13, 23:28 UTC · #16Nov 14, 01:28 UTC · #16Nov 14, 03:28 UTC · #17Nov 14, 05:28 UTC · #17Nov 14, 07:28 UTC · #18Nov 14, 09:28 UTC · #18Nov 14, 11:28 UTC · #18Nov 14, 13:28 UTC · #18Nov 14, 15:28 UTC · #18Nov 14, 17:28 UTC · #18Nov 14, 19:28 UTC · #18Nov 14, 21:28 UTC · #18Nov 14, 23:28 UTC · #17Nov 15, 01:28 UTC · #17Nov 15, 03:28 UTC · #17Nov 15, 05:28 UTC · #17Nov 15, 07:28 UTC · #17Nov 15, 09:28 UTC · #17Nov 15, 11:28 UTC · #17Nov 15, 13:28 UTC · #17Nov 15, 15:28 UTC · #17Nov 15, 17:28 UTC · #17Nov 15, 19:28 UTC · #17Nov 15, 21:28 UTC · #17Nov 15, 23:28 UTC · #17Nov 16, 01:28 UTC · #17Nov 16, 03:28 UTC · #16Nov 16, 05:28 UTC · #17

Last 14 days

Best rank per day (UTC).

Nov 3today

Metric history

▲ as reported by Hugging Face.

05914 Nov 13Nov 14Nov 14Nov 15Nov 16 Nov 13, 07:28 UTC · 9Nov 13, 09:28 UTC · 10Nov 13, 11:28 UTC · 11Nov 13, 13:28 UTC · 12Nov 13, 15:28 UTC · 12Nov 13, 17:28 UTC · 12Nov 13, 19:28 UTC · 12Nov 13, 21:28 UTC · 12Nov 13, 23:28 UTC · 12Nov 14, 01:28 UTC · 12Nov 14, 03:28 UTC · 12Nov 14, 05:28 UTC · 12Nov 14, 07:28 UTC · 12Nov 14, 09:28 UTC · 12Nov 14, 11:28 UTC · 12Nov 14, 13:28 UTC · 12Nov 14, 15:28 UTC · 12Nov 14, 17:28 UTC · 12Nov 14, 19:28 UTC · 12Nov 14, 21:28 UTC · 12Nov 14, 23:28 UTC · 14Nov 15, 01:28 UTC · 14Nov 15, 03:28 UTC · 14Nov 15, 05:28 UTC · 14Nov 15, 07:28 UTC · 14Nov 15, 09:28 UTC · 14Nov 15, 11:28 UTC · 14Nov 15, 13:28 UTC · 14Nov 15, 15:28 UTC · 14Nov 15, 17:28 UTC · 14Nov 15, 19:28 UTC · 14Nov 15, 21:28 UTC · 14Nov 15, 23:28 UTC · 14Nov 16, 01:28 UTC · 14Nov 16, 03:28 UTC · 14Nov 16, 05:28 UTC · 14

Questions

Is Factored One-sided Adam-Moment for Practical and Scalable SOAP trending on HF Daily Papers right now?

Yes. As of Nov 16, 05:28 UTC it is #17, holding steady at #17 over the last 6 checks.

What is the highest rank Factored One-sided Adam-Moment for Practical and Scalable SOAP reached?

#16 on Hugging Face · Daily papers, on Nov 13, 23:28 UTC. Trendoday checks this list every 120 minutes.

How long has Factored One-sided Adam-Moment for Practical and Scalable SOAP been trending?

It was first seen on Nov 13, 2026 and has been on the list on 4 days (36 checks) since.