Trendoday

Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width

Residual connections underpin Transformers, but they implicitly tie the residual width carried through depth to the compute width used inside each layer. We introduce Hierarchical Residuals (HiRes), a

  • Upvotes7
–Rank now
#17Peak rank
2Days on list
120Score · 7d
Nov 10, 2026First seen
Rank · 7 days

About this paper

Hierarchical Residuals decouple residual width from per-layer compute width by connecting standard Transformer blocks through a wider outer residual path, improving efficiency and accuracy without specialized kernels.

Residual connections underpin Transformers, but they implicitly tie the residual width carried through depth to the compute width used inside each layer. We introduce Hierarchical Residuals (HiRes), a minimalist architectural change that decouples these dimensions, widening the residual state while keeping per-layer compute width fixed. HiRes groups layers into blocks: standard d-dimensional PreNorm layers operate inside each block, while a wider D-dimensional PostNorm outer residual connects blocks. Learned read/write projections bridge the two spaces at block boundaries, leaving individual Transformer layers unchanged. Across different model sizes and depths, HiRes consistently…

residual connectionsTransformersresidual widthcompute widthHierarchical Residuals (HiRes)PreNorm layersPostNorm outer residualread/write projectionsblock boundariesscaling laws
Upvotes
7
Authors
7
Published
Oct 30, 2026

Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.

Trending history

Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width was first seen on Hugging Face · Daily papers on Nov 10, 2026. Since then it has appeared in 11 checks on 2 different days. Its best position was #17 of 30, reached on Nov 10, 21:28 UTC. It dropped off the list after Nov 11, 01:28 UTC.

Rank history

Position on Hugging Face · Daily papers at every check (lower is better).

#1#11#20#30 05:2810:2815:2820:2801:28 Nov 10, 05:28 UTC · #25Nov 10, 07:28 UTC · #24Nov 10, 09:28 UTC · #26Nov 10, 11:28 UTC · #23Nov 10, 13:28 UTC · #21Nov 10, 15:28 UTC · #20Nov 10, 17:28 UTC · #19Nov 10, 19:28 UTC · #19Nov 10, 21:28 UTC · #17Nov 10, 23:28 UTC · #19Nov 11, 01:28 UTC · #19

Last 14 days

Best rank per day (UTC).

Nov 3today

Metric history

Platform metric as reported by Hugging Face.

0358 05:2810:2815:2820:2801:28 Nov 10, 05:28 UTC · 7Nov 10, 07:28 UTC · 7Nov 10, 09:28 UTC · 7Nov 10, 11:28 UTC · 7Nov 10, 13:28 UTC · 7Nov 10, 15:28 UTC · 7Nov 10, 17:28 UTC · 7Nov 10, 19:28 UTC · 8Nov 10, 21:28 UTC · 8Nov 10, 23:28 UTC · 8Nov 11, 01:28 UTC · 8

Questions

Is Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width trending on HF Daily Papers right now?

Not at the moment. It was last on the list on Nov 11, 01:28 UTC, after 2 days of trending.

What is the highest rank Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width reached?

#17 on Hugging Face · Daily papers, on Nov 10, 21:28 UTC. Trendoday checks this list every 120 minutes.

How long has Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width been trending?

It was first seen on Nov 10, 2026 and has been on the list on 2 days (11 checks) since.