About this paper
Latent feedback, which feeds each token's top-layer state into the next token's input, improves compute and data efficiency over standard Transformers, especially when unique data is abundant relative to model size.
When high-quality data runs out, a common practice is to train on it for several epochs, but each repeat is worth less than the last. Latent feedback, which feeds each token's top-layer state into the next token's input, is another way to spend that compute, yet it has not been tested against a vanilla Transformer that spends the same compute on more epochs. We run this test across a ladder of model sizes, unique data and epochs, tuning both models at every size and amount of unique data. Latent feedback makes every FLOP and every token count for more except when unique data is scarce relative to model size, a regime that today's models are far from. We find that latent feedback serves as…
latent feedbackTransformertop-layer statescaling lawdownstream tasks
- Upvotes
- 5
- Authors
- 5
- Published
- Nov 16, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
Latent Feedback Is a Compute and Data Amplifier was first seen on Hugging Face · Daily papers on Nov 17, 2026. Since then it has appeared in 8 checks on 2 different days. Its best position was #15 of 30, reached on Nov 17, 23:28 UTC. It dropped off the list after Nov 18, 03:28 UTC.