Think Twice: The Metis-1.6 Technical Report

Metis-1.6 introduces small, efficient mixture-of-experts language models trained with limited data that achieve competitive math performance and Pareto-optimal efficiency despite training errors and architectural issues.

Metis is a bet that better data, better training methods and new architectures can buy what a bigger parameter count cannot, at a fraction of the cost. Metis-1.6 is its latest step: two language models built from scratch by a single researcher, Praxis (2.94B parameters, 0.46B active per pass) and Logos (10.23B parameters, 1.18B active per pass). Both run a hybrid Mamba-2 and attention mixture-of-experts stack twice for every token, so each token gets a second look. Each was trained on 1.02T tokens in total: 804.76B of pretraining, 18B of context extension to 131,072 tokens, and 200B of supervised fine-tuning. Metis-1.6 was never meant to challenge frontier models, and across 40…

Upvotes

4

Authors

1

Published

Oct 27, 2026

arXiv

arXiv abstract

Code

lernex/dromos

Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.