5 ms·

Supervised fine tuning on curated data is reinforcement learning

mandevil 1y ago
Interesting to see two independent researchers on this. Makes me curious as to what the back-story is? Side project?
- babelfish 1y ago
  Especially interesting given they both work for Google DeepMind.
- GabrielBianconi 1y ago
  Yeah, I hadn't noticed!
- jtspringenberg 1y ago
  Author here, just to clarify: we are both no longer working for DeepMind. This was purely an independent effort for the sake of research and understanding! Happy to answer any questions.
iandanforth 1y ago
How is this kind of analogy helpful? You can frame any optimization problem as RL if you try hard enough. RL is a method of optimization which calls the optimum "reward maximization". You can craft the reward function any which way you want. The key point about RL is that it is a sequential decision making process. If you don't have something (an agent) making multiple decisions over time while interacting with an environment, then why bother calling it RL?
- imtringued 1y ago
  I personally am quite disappointed by the abstract: "Building on existing literature, we clarify that SFT can be understood as maximizing a lower bound on the RL objective in a sparse reward setting." uh no? SFT is maximizing the RL objective in a dense reward setting. The entire point of RL, specifically actor-critic and Q-Learning, is that the RL method turns the sparse reward into a continuous dense reward against which a model can be trained on with classic gradient descent. I mean look at the definition of Q-Learning and the bellman equation it uses. It maximizes the current reward by choosing the current action based on whether it maximizes the predicted reward, not the actual reward, which doesn't have to be continuous or produce a gradient. You can build an RL based maze solver where only the goal gives a reward to the model and it would work, albeit it would train extremely slowly. Meanwhile supervised fine tuning always produces a continuous gradient on every single token.
  - chongliqin 1y ago
    TD-based approaches can have an advantage in sparse reward settings, but they come with a heap of other problems especially in the off-policy setting (see the deadly triad) and are typically not used for LLM training. We here make a connection to REINFORCE style policy gradients which would not show any of the behavior you mentioned above.
anndvision 1y ago
We recently ran similar experiments and saw that fine-tuning small models on automatically curated high-quality outputs from a large model can beat large-model performance while reducing inference costs by up to 30x and inference time by up to 4x. We benchmarked closed-source (OpenAI, Google) and open-source (Qwen) models on multi-turn maze navigation (BabyAI), agentic RAG (Multi-Hop), and agentic tool use (τ-bench). We're still running a few experiments and plan to update the post with additional results in a few days. Looking forward to trying out importance weighting soon! Curated Behavior Cloning: Small LLMs Can Beat Large Ones at 5-30x Lower Cost: https://www.tensorzero.com/blog/curated-behavior-cloning-small-llms-can-beat-large-ones-at-5-30x-lower-cost/ https://www.tensorzero.com/blog/curated-behavior-cloning-sma...
- chongliqin 1y ago
  Cool! If you are interested, we have open sourced our code: https://github.com/emmyqin/iw_sft https://github.com/emmyqin/iw_sft
  - anndvision 1y ago
    thanks
- TheTaytay 1y ago
  Thanks for this - I’ve spent the last hour reading your docs and blog. I like the primitives you’ve exposed in your APO, and particularly like the decision to separate out the structured inputs from the prompt when you record an LLM call, so I can finally perform optimizations and evals on past calls. Quick question : you mentioned unsloth in the blog post. Which of the fine tuning providers mentioned is using unsloth under the hood?
  - GabrielBianconi 1y ago
    [I'm his coworker.] We ran Unsloth ourselves on a GPU-by-the-hour server. We have a notebook in the repository showing how to query historical data and use it with Unsloth. It's a WIP PR that we plan to merge soon: https://github.com/tensorzero/tensorzero/pull/2273 https://github.com/tensorzero/tensorzero/pull/2273
henriquegodoy 1y ago
It's cool to see the perspective that many problems (somekinda communication problems, look at lawyers, compliance and etc...) can be solved by treating AI less as agents and more as modular components within a larger system. Once we build a working process—monitored through evals—we can then reduce costs by distilling these modules. That means starting with superintelligent models and later distilling them down to just a few billion parameters, instead of needing hundreds of billions.
stolencode 1y ago
> For example achieving 66.7% on the AIME 2024 dataset. We worked _really_ hard, burned _tons_ of cash, and we're proud of our D- output. No wonder there are more papers published than actual work being done.
- supermdguy 1y ago
  That corresponds to a 10/15, which is actually really good (median is around 6) https://artofproblemsolving.com/wiki/index.php/AMC_historical_results https://artofproblemsolving.com/wiki/index.php/AMC_historica...
  - stolencode 1y ago
    Isn't the test taken only by students under the age of 12? Meanwhile the model is trained on these specific types of problems, does not have an apparent time or resource limit, and does not have to take the test in a proctored environment. It's D- work. Compared to a 12 year old, okay, maybe it's B+. Is this really the point you wanted to make?
- jpcompartir 1y ago
  This is a nonsense critique. Modest results are worth publishing, as are bad results.
markisus 1y ago
Something seems off with equation (5). Just imagining Monte Carlo sampling it, the middle expectation will have a bunch of zeros due to the indicator function and the right expectation won’t. I can make the middle expectation be as close to zero as I like by making the success threshold sufficiently high.
- chongliqin 1y ago
  Ah yes you are right the rhs was meant to be proportional to the middle expectation (see the equation below), for equality the rhs needs to be multiplied by a normalization constant independent of theta. Note this doesn't affect the bounds as the constant is the same across equations. Will update the paper to incorporate.

Interesting to see two independent researchers on this. Makes me curious as to what the back-story is? Side project?

babelfish 1y ago
Especially interesting given they both work for Google DeepMind.
GabrielBianconi 1y ago
Yeah, I hadn't noticed!
jtspringenberg 1y ago
Author here, just to clarify: we are both no longer working for DeepMind. This was purely an independent effort for the sake of research and understanding! Happy to answer any questions.

How is this kind of analogy helpful? You can frame any optimization problem as RL if you try hard enough. RL is a method of optimization which calls the optimum "reward maximization". You can craft the reward function any which way you want. The key point about RL is that it is a sequential decision making process. If you don't have something (an agent) making multiple decisions over time while interacting with an environment, then why bother calling it RL?

imtringued 1y ago
I personally am quite disappointed by the abstract: "Building on existing literature, we clarify that SFT can be understood as maximizing a lower bound on the RL objective in a sparse reward setting." uh no? SFT is maximizing the RL objective in a dense reward setting. The entire point of RL, specifically actor-critic and Q-Learning, is that the RL method turns the sparse reward into a continuous dense reward against which a model can be trained on with classic gradient descent. I mean look at the definition of Q-Learning and the bellman equation it uses. It maximizes the current reward by choosing the current action based on whether it maximizes the predicted reward, not the actual reward, which doesn't have to be continuous or produce a gradient. You can build an RL based maze solver where only the goal gives a reward to the model and it would work, albeit it would train extremely slowly. Meanwhile supervised fine tuning always produces a continuous gradient on every single token.
- chongliqin 1y ago
  TD-based approaches can have an advantage in sparse reward settings, but they come with a heap of other problems especially in the off-policy setting (see the deadly triad) and are typically not used for LLM training. We here make a connection to REINFORCE style policy gradients which would not show any of the behavior you mentioned above.

anndvision 1y ago

We recently ran similar experiments and saw that fine-tuning small models on automatically curated high-quality outputs from a large model can beat large-model performance while reducing inference costs by up to 30x and inference time by up to 4x. We benchmarked closed-source (OpenAI, Google) and open-source (Qwen) models on multi-turn maze navigation (BabyAI), agentic RAG (Multi-Hop), and agentic tool use (τ-bench). We're still running a few experiments and plan to update the post with additional results in a few days. Looking forward to trying out importance weighting soon! Curated Behavior Cloning: Small LLMs Can Beat Large Ones at 5-30x Lower Cost: https://www.tensorzero.com/blog/curated-behavior-cloning-small-llms-can-beat-large-ones-at-5-30x-lower-cost/ https://www.tensorzero.com/blog/curated-behavior-cloning-sma...

chongliqin 1y ago
Cool! If you are interested, we have open sourced our code: https://github.com/emmyqin/iw_sft https://github.com/emmyqin/iw_sft
- anndvision 1y ago
  thanks
TheTaytay 1y ago
Thanks for this - I’ve spent the last hour reading your docs and blog. I like the primitives you’ve exposed in your APO, and particularly like the decision to separate out the structured inputs from the prompt when you record an LLM call, so I can finally perform optimizations and evals on past calls. Quick question : you mentioned unsloth in the blog post. Which of the fine tuning providers mentioned is using unsloth under the hood?
- GabrielBianconi 1y ago
  [I'm his coworker.] We ran Unsloth ourselves on a GPU-by-the-hour server. We have a notebook in the repository showing how to query historical data and use it with Unsloth. It's a WIP PR that we plan to merge soon: https://github.com/tensorzero/tensorzero/pull/2273 https://github.com/tensorzero/tensorzero/pull/2273

henriquegodoy 1y ago

It's cool to see the perspective that many problems (somekinda communication problems, look at lawyers, compliance and etc...) can be solved by treating AI less as agents and more as modular components within a larger system. Once we build a working process—monitored through evals—we can then reduce costs by distilling these modules. That means starting with superintelligent models and later distilling them down to just a few billion parameters, instead of needing hundreds of billions.

stolencode 1y ago

> For example achieving 66.7% on the AIME 2024 dataset. We worked _really_ hard, burned _tons_ of cash, and we're proud of our D- output. No wonder there are more papers published than actual work being done.

supermdguy 1y ago
That corresponds to a 10/15, which is actually really good (median is around 6) https://artofproblemsolving.com/wiki/index.php/AMC_historical_results https://artofproblemsolving.com/wiki/index.php/AMC_historica...
- stolencode 1y ago
  Isn't the test taken only by students under the age of 12? Meanwhile the model is trained on these specific types of problems, does not have an apparent time or resource limit, and does not have to take the test in a proctored environment. It's D- work. Compared to a 12 year old, okay, maybe it's B+. Is this really the point you wanted to make?
jpcompartir 1y ago
This is a nonsense critique. Modest results are worth publishing, as are bad results.

markisus 1y ago

Something seems off with equation (5). Just imagining Monte Carlo sampling it, the middle expectation will have a bunch of zeros due to the indicator function and the right expectation won’t. I can make the middle expectation be as close to zero as I like by making the success threshold sufficiently high.

chongliqin 1y ago
Ah yes you are right the rhs was meant to be proportional to the middle expectation (see the equation below), for equality the rhs needs to be multiplied by a normalization constant independent of theta. Note this doesn't affect the bounds as the constant is the same across equations. Will update the paper to incorporate.