On-policy reinforcement learning for large language models assumes that training responses are sampled from the current policy. In practice, an inference engine samples the responses and a separate tr
On-policy reinforcement learning for large language models suffers from training-inference mismatch due to numerical discrepancies and policy lag, and a derived discrepancy-lag correction stabilizes training by centering errors, clipping importance ratios, and resynchronizing sampling.
On-policy reinforcement learning for large language models assumes that training responses are sampled from the current policy. In practice, an inference engine samples the responses and a separate trainer computes the gradients, and the training-inference mismatch between the two makes the training data off-policy. The mismatch has two sources, the numerics of the engine and the stale weights of asynchronous sampling, and a correction that stabilizes training under one source can fail under the other. We ask which property of a mismatch source decides the correction it needs. Evaluating the trainer at the sampler's weights splits each sampled-token error exactly into a numerical…
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning was first seen on Hugging Face · Daily papers on Nov 2, 2026. Since then it has appeared in 9 checks on 2 different days. Its best position was #20 of 30, reached on Nov 3, 03:28 UTC. It dropped off the list after Nov 3, 07:28 UTC.
Position on Hugging Face · Daily papers at every check (lower is better).
Best rank per day (UTC).
Platform metric as reported by Hugging Face.
Archived daily, weekly and monthly lists that include this entry.
Not at the moment. It was last on the list on Nov 3, 07:28 UTC, after 2 days of trending.
#20 on Hugging Face · Daily papers, on Nov 3, 03:28 UTC. Trendoday checks this list every 120 minutes.
It was first seen on Nov 2, 2026 and has been on the list on 2 days (9 checks) since.