About this paper
A systematic study of Hyper-Connections architectures identifies the simpler efficient design SimpleHC, which removes residual mixing and uses shared memory access to match mHC performance while adding minimal training overhead and improving stability.
Hyper-Connections (HC) architectures are increasingly adopted in frontier language models, yet their components remain insufficiently understood and introduce non-negligible training overhead. In this work, we systematically study the HC architecture to understand its components and identify simpler, more efficient designs. We find that learned read/write gating without mixing and unconstrained mixing with fixed gates form two effective interaction groups: either alone achieves performance comparable to mHC. We also identify repeated access to the expanded residual state as a major source of training overhead, even without mixing. These findings lead to SimpleHC, which simplifies the…
Hyper-Connections architecturesfrontier language modelslearned read/write gatingunconstrained mixingfixed gatesmHCexpanded residual stateSimpleHCresidual mixingshared memory access
- Upvotes
- 29
- Authors
- 11
- Published
- Nov 4, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
On the Design of Hyper-Connections: Simplification, Stability, and Scalability was first seen on Hugging Face · Daily papers on Nov 13, 2026. Since then it has appeared in 37 checks on 4 different days. Its best position was #8 of 30, reached on Nov 13, 07:28 UTC. It is #10 right now, climbing from #11 to #10 over the last 6 checks.