Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are trained with backpropagation through time (BPTT). To
Looped transformers trained with full backpropagation through time outperform truncated alternatives under matched compute, with scaling laws identifying two loops as optimal when model size is adjustable.
Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are trained with backpropagation through time (BPTT). To save training compute, BPTT is often truncated to the last few loops, but truncation yields biased gradients that may weaken the benefit of looping. In this work, we study the value of truncation in looped transformer training in a systematic way: we jointly vary model size, training tokens, and forward and backward loop counts across nearly 600 training runs and fit a unified scaling law that captures the effects of looping and truncation through effective parameters. Despite the…
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation was first seen on Hugging Face · Daily papers on Oct 28, 2026. Since then it has appeared in 24 checks on 3 different days. Its best position was #5 of 30, reached on Oct 30, 05:28 UTC. It dropped off the list after Oct 30, 05:28 UTC.
Position on Hugging Face · Daily papers at every check (lower is better).
Best rank per day (UTC).
Platform metric as reported by Hugging Face.
Archived daily, weekly and monthly lists that include this entry.
Not at the moment. It was last on the list on Oct 30, 05:28 UTC, after 3 days of trending.
#5 on Hugging Face · Daily papers, on Oct 30, 05:28 UTC. Trendoday checks this list every 120 minutes.
It was first seen on Oct 28, 2026 and has been on the list on 3 days (24 checks) since.