Weight-normalized (WN) training is emerging as a compelling alternative to conventional weight decay (WD), with recent studies reporting substantial pretraining loss gains. We show that these gains ar
Effective learning rate trajectories primarily drive weight-normalized training gains, motivating an affine inverse-square-root schedule that improves large-scale pretraining loss and robustness.
Weight-normalized (WN) training is emerging as a compelling alternative to conventional weight decay (WD), with recent studies reporting substantial pretraining loss gains. We show that these gains are driven primarily by the induced effective learning rate (ELR) trajectory rather than by normalization alone. Matching ELR trajectories largely collapses the training loss dynamics of WN and WD. Schedule design should therefore target ELR rather than the nominal learning rate (LR) alone. Building on this principle, we propose AIR (Affine Inverse-square-Root decay), which combines a theoretically derived 1/t decay with an affine adjustment to satisfy prescribed peak and terminal LRs over a…
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training was first seen on Hugging Face · Daily papers on Nov 1, 2026. Since then it has appeared in 3 checks on 2 different days. Its best position was #16 of 30, reached on Nov 1, 09:28 UTC. It dropped off the list after Nov 2, 05:28 UTC.
Position on Hugging Face · Daily papers at every check (lower is better).
Best rank per day (UTC).
Platform metric as reported by Hugging Face.
Archived daily, weekly and monthly lists that include this entry.
Not at the moment. It was last on the list on Nov 2, 05:28 UTC, after 2 days of trending.
#16 on Hugging Face · Daily papers, on Nov 1, 09:28 UTC. Trendoday checks this list every 120 minutes.
It was first seen on Nov 1, 2026 and has been on the list on 2 days (3 checks) since.