About this paper
DepthGraft improves deep language model training by hierarchically sharing key-value representations across layers, reducing memory use while boosting accuracy.
Scaling depth is a key route to stronger large language models, but Pre-LN architectures often suffer from depth-wise information dilution, limiting the effective use of deep layers. While cross-layer connectivity has proven effective at mitigating this problem, existing designs usually introduce additional computation or memory access. We revisit cross-layer KV sharing as an efficient form of such connectivity. Beyond cache compression, each KV reuse edge creates a direct gradient path from deep layers to source KV projections, turning KV sharing into an implicit structural regularizer for KV representation geometry. We provide a systematic analysis of this mechanism, showing that proper…
Pre-LNcross-layer KV sharinggradient pathKV representation geometryDepthGrafthierarchical KV-sharingGQAMLAMoE
- Upvotes
- 10
- Authors
- 7
- Published
- Nov 2, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction was first seen on Hugging Face · Daily papers on Nov 11, 2026. Since then it has appeared in 15 checks on 2 different days. Its best position was #12 of 30, reached on Nov 11, 13:28 UTC. It dropped off the list after Nov 12, 09:28 UTC.