When large language models write neural network weights without gradient-based training, they tend to design sparse table or associative-memory models that outperform same-sized Transformers with far fewer parameters and faster GPU inference, though pretrained LLMs remain more accurate.
Standard neural architectures such as the Transformer were chosen because they train well with gradient descent and run fast on GPUs. In weight-space programming, a large language model (LLM) writes target weights by hand without gradient-based training, so those reasons no longer apply. Should we keep the Transformer? Given minimal guidance, ten labeled examples, and a sandbox, mainly on multiple-choice questions, we task three frontier LLMs with either generating the weights of a fixed architecture or designing the architecture themselves. When free to design, all three compile plain-text facts into a sparse table queried by a fixed procedure, reaching up to 54.4% test accuracy. An…
Transformerneural architecturesgradient descentlarge language modelweight-space programmingtarget weightsgradient-based trainingsparse tableassociative-memory-based modelpretrained weights
- Upvotes
- 11
- Authors
- 2
- Published
- Nov 17, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.