There are two RL approaches - process reward models (PRM) that provide feedback on each step of the reasoning chain, and outcome reward models (ORM) that only p

There are two RL approaches - process reward models (PRM) that provide feedback on each step of the reasoning chain, and outcome reward models (ORM) that only provide feedback on the complete chain. DeepSeek use an outcome model, and mention some of the difficulties of PRM, including both identifying an individual step as well as how to verify it. The trained reward model provides the gradient.