About this paper
A looped Transformer that separately controls input strength, total update weight, and per-update allocation achieves lower validation loss and higher downstream accuracy than standard and looped residual baselines while using fewer parameters.
Looped Transformers reduce parameter count by repeatedly applying shared blocks, but parameter sharing alone does not determine how the results of repeated computation are reused. We take the depth-memory view of residual connections and analyze which representation is stored after each block application and how later reads weight it. A full block output contains both its input and its new update, so weighting that output also reweights the history already carried by the input. Reading individual updates instead, with the shared stack's input kept on a separate path, gives separate controls over input strength, total update weight, and allocation across updates. The Loopback Transformer…
Looped Transformersparameter sharingshared blocksresidual connectionsdepth-memory view of residual connectionsblock outputinput strengthtotal update weightallocation across updatesLoopback Transformer
- Upvotes
- 11
- Authors
- 6
- Published
- Nov 4, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
Loopback Transformer: Towards a Unified View of Residual Looped Transformers was first seen on Hugging Face · Daily papers on Nov 16, 2026. Since then it has appeared in 13 checks on 2 different days. Its best position was #9 of 30, reached on Nov 17, 01:28 UTC. It dropped off the list after Nov 17, 05:28 UTC.