Among 291 surveyed open-weight language model repositories (280 distinct embedding tables), 52.6% of untied gated and hybrid checkpoints (20/38) contain reachable input embeddings with norms below 10^
Near-zero input embeddings in untied gated language models cause extreme gradient amplification and training instability, which is addressed by GroupClip and GateFloor remedies.
Among 291 surveyed open-weight language model repositories (280 distinct embedding tables), 52.6% of untied gated and hybrid checkpoints (20/38) contain reachable input embeddings with norms below 10^{-6}, compared with 0% of weight-tied models. Such near-zero rows can arise during pretraining when unseen rare or template tokens receive zero input gradients while AdamW weight decay shrinks untied embeddings. In tested Transformer configurations, non-initial positions or a BOS token prevent this failure mode. In gated sequence models (Mamba, Gated DeltaNet, xLSTM), a near-zero token can amplify input-embedding gradients by more than 30 orders of magnitude (up to 35). At zero input,…
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models was first seen on Hugging Face · Daily papers on Nov 4, 2026. Since then it has appeared in 12 checks on 2 different days. Its best position was #22 of 30, reached on Nov 5, 03:28 UTC. It dropped off the list after Nov 5, 03:28 UTC.
Position on Hugging Face · Daily papers at every check (lower is better).
Best rank per day (UTC).
Platform metric as reported by Hugging Face.
Archived daily, weekly and monthly lists that include this entry.
Not at the moment. It was last on the list on Nov 5, 03:28 UTC, after 2 days of trending.
#22 on Hugging Face · Daily papers, on Nov 5, 03:28 UTC. Trendoday checks this list every 120 minutes.
It was first seen on Nov 4, 2026 and has been on the list on 2 days (12 checks) since.