About this paper
A two-table weight-sharing method for Transformer language models improves output expressiveness and reduces validation loss and perplexity across several architectures and scales at the cost of an extra embedding table and vocabulary projection.
Weight tying reduces the parameter cost of language models, but the usual tied output head is limited to a single softmax. We introduce onóna, a two-table parameterization that increases output expressiveness while preserving input-output weight sharing. The Transformer takes the average of two embedding tables as input, and the output head combines the softmax distributions defined by those same tables. A gate conditioned only on the current input token determines the mixture weight. Its values can be precomputed for inference, and the Transformer body remains unchanged. We evaluate onóna across Dense, Gated DeltaNet hybrid, mixture-of-experts, and Attention Residual architectures at…
weight tyinglanguage modelstied output headsoftmaxtwo-table parameterizationinput-output weight sharingTransformerembedding tablesoutput headsoftmax distributions
- Upvotes
- 3
- Authors
- 7
- Published
- Nov 7, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
onóna: Twin Embeddings with Tied Unembeddings was first seen on Hugging Face · Daily papers on Nov 11, 2026. Since then it has appeared in 1 check on 1 different day. Its best position was #30 of 30, reached on Nov 11, 03:28 UTC. It dropped off the list after Nov 11, 03:28 UTC.