Multi-Branch Transformer
Transformer language models are commonly scaled by making their blocks wider or stacking them deeper. In this work, we explore a complementary path: branching, the parallel composition of full Transfo
- Upvotes16
Transformer language models are commonly scaled by making their blocks wider or stacking them deeper. In this work, we explore a complementary path: branching, the parallel composition of full Transfo
Branching Transformer blocks into parallel parameterized branches, combined with shared KV caching, improves language model quality and inference efficiency compared with standard sequential Transformers.
Transformer language models are commonly scaled by making their blocks wider or stacking them deeper. In this work, we explore a complementary path: branching, the parallel composition of full Transformer blocks. We introduce the Multi-Branch Transformer (MBT), which processes the input through shared initial layers, executes independently parameterized branches of sequential blocks in parallel, and fuses their outputs before shared final layers. At comparable training compute, we study three MBT variants: Wide-Shallow preserves block width, Narrow-Deep preserves sequential depth, and Balanced interpolates between them. In controlled pretraining experiments at four mixture-of-experts…
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Multi-Branch Transformer was first seen on Hugging Face · Daily papers on Nov 18, 2026. Since then it has appeared in 13 checks on 2 different days. Its best position was #4 of 30, reached on Nov 18, 15:28 UTC. It dropped off the list after Nov 19, 03:28 UTC.
Position on Hugging Face · Daily papers at every check (lower is better).
Best rank per day (UTC).
Platform metric as reported by Hugging Face.
Archived daily, weekly and monthly lists that include this entry.
Not at the moment. It was last on the list on Nov 19, 03:28 UTC, after 2 days of trending.
#4 on Hugging Face · Daily papers, on Nov 18, 15:28 UTC. Trendoday checks this list every 120 minutes.
It was first seen on Nov 18, 2026 and has been on the list on 2 days (13 checks) since.