On-policy distillation tolerates substantial training-inference mismatch from quantized inference and asynchronous rollouts, with model- and layer-specific limits, and a lightweight admission check plus FP8 recipes can exploit this tolerance to improve training efficiency.
On-policy distillation (OPD) trains a student language model on its own responses, guided by a fixed teacher. Generating these responses is expensive, so systems use quantized inference and asynchronous rollouts to reduce the cost. Both, however, introduce training-inference mismatch between the rollout engine and the model being trained. Such mismatch can destabilize reinforcement learning (RL), but its effects on OPD remain unclear. This paper investigates how much mismatch OPD tolerates, where its effects arise, and how training systems can use this tolerance. Measuring all mismatch sources on one scale, we examine quantization across three student-teacher pairs, with additional tests…
on-policy distillationstudent language modelteacher modelquantized inferenceasynchronous rolloutstraining-inference mismatchrollout enginereinforcement learningquantizationstudent-teacher pairs
- Upvotes
- 5
- Authors
- 8
- Published
- Nov 8, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.