At matched compute, dense process rewards such as Monte-Carlo rewards underperform standard GRPO for language-model reasoning, while selecting prompts with predicted success near 1/2 yields small consistent accuracy gains.
RLVR trains language models to reason by rewarding correct final answers. A widely used algorithm, group-relative policy optimization (GRPO), gives every token of a solution the same credit, so dense process rewards have been proposed to credit intermediate steps, for example with a Monte-Carlo reward that scores each step by sampling continuations from it. Comparing such a reward with GRPO per training update, or vetting it with a gradient test on a single batch, leaves out what it costs: in our setting, the continuations alone take 91% of all generated tokens. We introduce a matched-compute protocol that gives every training intervention the same budget, measured in generated tokens and…
RLVRgroup-relative policy optimization (GRPO)dense process rewardsMonte-Carlo rewardmatched-compute protocolper-token gradient testdense-reward variantslearning rateprompt selection by predicted success near 1/2Qwen2.5-1.5B
- Upvotes
- 3
- Authors
- 19
- Published
- Nov 10, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.