About this paper
Hierarchical Residuals decouple residual width from per-layer compute width by connecting standard Transformer blocks through a wider outer residual path, improving efficiency and accuracy without specialized kernels.
Residual connections underpin Transformers, but they implicitly tie the residual width carried through depth to the compute width used inside each layer. We introduce Hierarchical Residuals (HiRes), a minimalist architectural change that decouples these dimensions, widening the residual state while keeping per-layer compute width fixed. HiRes groups layers into blocks: standard d-dimensional PreNorm layers operate inside each block, while a wider D-dimensional PostNorm outer residual connects blocks. Learned read/write projections bridge the two spaces at block boundaries, leaving individual Transformer layers unchanged. Across different model sizes and depths, HiRes consistently…
residual connectionsTransformersresidual widthcompute widthHierarchical Residuals (HiRes)PreNorm layersPostNorm outer residualread/write projectionsblock boundariesscaling laws
- Upvotes
- 7
- Authors
- 7
- Published
- Oct 30, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width was first seen on Hugging Face · Daily papers on Nov 10, 2026. Since then it has appeared in 11 checks on 2 different days. Its best position was #17 of 30, reached on Nov 10, 21:28 UTC. It dropped off the list after Nov 11, 01:28 UTC.