About this paper
Chorus shares one set of n-gram embedding tables across all layers, letting each layer project and rescale the embedding using its hidden state, achieving better quality with much smaller tables and lower optimizer memory.
Language models can gain capacity at little computational cost from n-gram embedding tables: large tables of learned vectors for short sequences of consecutive tokens, which are looked up and added to the model's input or to its intermediate layers. Recent designs commonly use a separate n-gram embedding table for each layer, which makes the tables costly in memory and prevents layers from sharing what they learn. Sharing a single table, however, gives every layer the same embedding and leaves no room for layer-wise preferences, and it performs poorly in practice. We propose Chorus, in which one set of n-gram embedding tables serves the whole network and every layer reads it in its own…
n-gram embedding tablesembedding tablesn-gram methodsprojection layerhidden statelayer-wise preferencesmemory-efficient optimizerEmberAdamWtable-optimizer memory
- Upvotes
- 3
- Authors
- 2
- Published
- Nov 17, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
Chorus: Shared N-gram Embeddings with Layer-wise Adaptation was first seen on Hugging Face · Daily papers on Nov 19, 2026. Since then it has appeared in 7 checks on 1 different day. Its best position was #24 of 30, reached on Nov 19, 11:29 UTC. It dropped off the list after Nov 19, 23:29 UTC.