Trendoday

Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts

Frontier post-training follows a common recipe: train one expert per domain with reinforcement learning (RL), then consolidate the experts into one model via multi-teacher on-policy distillation (MOPD

  • Upvotes4
–Rank now
#30Peak rank
1Days on list
4Score · 7d
Nov 20, 2026First seen

About this paper

Frontier post-training follows a common recipe: train one expert per domain with reinforcement learning (RL), then consolidate the experts into one model via multi-teacher on-policy distillation (MOPD). Such consolidation is expensive: the student samples in every environment again, an expert scores every sampled token, and the student inherits the drift of the final experts. Yet we argue that the experts' RL runs have already produced what consolidation needs: rollouts with their sampling probabilities and rewards, in the order the experts learned from them. We propose multi-expert replay policy distillation (MRPD), which replays these training logs through the student in the experts'…

Upvotes
4
Authors
7
Published
Nov 18, 2026

Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.

Trending history

Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts was first seen on Hugging Face · Daily papers on Nov 20, 2026. Since then it has appeared in 1 check on 1 different day. Its best position was #30 of 30, reached on Nov 20, 17:29 UTC. It dropped off the list after Nov 20, 17:29 UTC.

Rank history

Position on Hugging Face · Daily papers at every check (lower is better).

#1#11#20#30 16:5917:1417:2917:4417:59 Nov 20, 17:29 UTC · #30

Last 14 days

Best rank per day (UTC).

Nov 9today

Questions

Is Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts trending on HF Daily Papers right now?

Not at the moment. It was last on the list on Nov 20, 17:29 UTC, after 1 day of trending.

What is the highest rank Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts reached?

#30 on Hugging Face · Daily papers, on Nov 20, 17:29 UTC. Trendoday checks this list every 120 minutes.

How long has Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts been trending?

It was first seen on Nov 20, 2026 and has been on the list on 1 day (1 check) since.