About this paper
TPHC combines tensor-parallel shards with hyper-connection residual streams to enable more independent stream updates, improving language model loss and zero-shot reasoning with modest training overhead.
Hyper-Connections (HC) expand the residual stream of a Transformer into N parallel streams, and manifold-constrained HC (mHC) stabilizes them at scale. However, mHC benefits little from more than four streams, partly because each sublayer writes a single output to all streams, limiting how differently they can be updated. In this work, we propose TPHC, which integrates tensor-parallel (TP) shards with HC streams to promote greater independence across streams. Specifically, we route a separate mixture of the residual streams to each TP shard and keep the individual shard outputs separate during write-back, so each stream can receive a different mixture of those outputs. This design makes…
Hyper-Connectionsresidual streamTransformermanifold-constrained HCtensor-parallel shardsHC streamsmixture of residual streamswrite-backattention sublayersFFN sublayers
- Upvotes
- 8
- Authors
- 7
- Published
- Nov 17, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates was first seen on Hugging Face · Daily papers on Nov 18, 2026. Since then it has appeared in 2 checks on 1 different day. Its best position was #26 of 30, reached on Nov 18, 09:28 UTC. It dropped off the list after Nov 18, 11:28 UTC.