7 ms·

RLHF a LLM in <50 lines of Python

223 points by patelajay285 3y ago

ilaksh 3y ago
How do you normally do DPO? Is that built in to PyTorch or something? Theoretically the hard part is collecting the examples with rejections etc.
- patelajay285 3y ago
  Collecting data is hard, but the library is also a synthetic data generation library, so for example you can create the data for DPO fully synthetically, check out the self-rewarding LLMs example: https://datadreamer.dev/docs/latest/pages/get_started/quick_ https://datadreamer.dev/docs/latest/pages/get_started/quick_...
aethelyon 3y ago
This is cool, but the data collection is the hard part, right?
- patelajay285 3y ago
  Yes it is :), but the library is also a synthetic data generation library, so for example you can create the data for DPO fully synthetically, check out the self-rewarding LLMs example: https://datadreamer.dev/docs/latest/pages/get_started/quick_tour/self_rewarding.html https://datadreamer.dev/docs/latest/pages/get_started/quick_...
  - sillysaurusx 3y ago
    I’m extremely skeptical of this approach. Until proven otherwise, with a model that users actually find useful, I don’t think this can work. It would be nice. But I’ve seen too many nice ideas completely fall apart in practice to accept this without some justification. Even if there are papers on the topic, and those papers show that the models rank highly according to some eval metrics, the only metric that truly matters is "the user likes the model and it solves their problems." By the way, on a separate topic, the 90/10 dataset split that you do in all of your examples turns out to be fraught with peril in practice. The issue is that the validation dataset quality turns out to be crucial, and randomly yeeting 10% of your data into the validation dataset without manual review is a recipe for problems.
    - patelajay285 3y ago
      It's a demo snippet of how to setup the workflow, it's not meant to be a working production example a self-rewarding model or a faithful reproduction of the original paper. Whether self-rewarding LLMs are a good idea or not, it's a valuable and very active area of research in the literature today. This is a library for ML researchers who should actively research and study these avenues along with the pitfalls you're mentioning. But in order for them to do that, building these workflows have to be accessible to them, which is what this library is meant to do. It's not meant for the "hobbyist" ML-community, they should not be using synthetic data today in this way, it would likely lead to subpar results for any practical model or task.
      - sillysaurusx 3y ago
        There’s a lot to unpack here. First, I’m an ML researcher. I don’t go around saying so because appeal to authority is bogus, but since every one of your comments seems to do this, it’s unavoidable. You say the code is for ML researchers, then flat out say that it’s not a working production example, nor is it a faithful reproduction of a paper. So what is it? Whether you want it to be or not, your audience is the hobbyist ML community, because without benchmarks to back up your code examples, no one from the research community will trust your examples without actual proof that they work. That’s the hard part of research, and it’s most of the effort. My advice is, write something that can train useful models. Implement a production grade workflow, and show some reasons why it works. If you’re trying to get the wider ML research community to buy in to this, there’s not much other way to do it. No one will want to take easy code that does the wrong thing, and most of your examples show the wrong thing to do, like the 90/10 split. You’re also a bit defensive about accepting feedback. Trust me that it’s better to accept that your code sucks and does the wrong thing, and then try to make it suck less and do the right thing. That’s how the majority of good software is written, unless you’re cperciva. But he’d also publish a paper explaining why his code is correct. Anyway, the whole point of posting this to HN is to get feedback on it. (If you were hoping that a bunch of people would suddenly use it, then you need to appeal to the hobbyist community. They’ve told you a bunch of things that you’ve straight up said is out of scope.) And it sounds like you were hoping for feedback from ML researchers. Maybe others will chime in, but for now, that’s the best I’ve got.
v4dok 3y ago
I feel like the current meta on finetuning LLMs is random accounts at X/Twitter. Google results are littered with SEO garbage or some kind of guides that fail to work the moment you need something slightly different.
rldjbpin 3y ago
it is very conflicting to see "do x in y LOC" in this field, especially when most of the workflow for different models are fragmented across non-overlapping frameworks/tooling. to actually do something from scratch or using the author's code requires adopting something esoteric just for this purpose. for these scenarios it is nice to appreciate hf and their abstraction. but the reinventing the wheel situation is very frustrating to work with. if you want to go beyond the demo, you have to deal with this painful reality. i hope there is more progress on this rather than making stacks of api.
cztomsik 3y ago
DPO is not RLHF.
patelajay285 3y ago
Hi everyone, there are no easy tools for synthetic data generation or training and aligning LLMs simply in Python. Most of the stuff out there are messy adhoc scripts. DataDreamer is an open source Python package with a nice API from the University of Pennsylvania that does all this that we’re actively developing. Will be here to answer questions. https://github.com/datadreamer-dev/DataDreamer https://github.com/datadreamer-dev/DataDreamer
- baggiponte 3y ago
  The API looks nice, congratulations. Will experiment with it. One small silly question: why did you choose to specify the dependencies inside the src dir with the requirements format - rather than inside the pyproject?
  - patelajay285 3y ago
    Thanks! It makes it easier to run with the existing run scripts I have on our large university GPU cluster. :) no other reason
g4zj 3y ago
Very cool, but I can't help but feel like titles that reference low-LOC are a bit clickbait-y when nearly all the heavy lifting is done by imported libraries.
- patelajay285 3y ago
  Appreciate the feedback on the title, this is developed for ML researchers, so I assure there is a lot it’s doing under the hood to make this process easier (for example introducing automatic caching and resumability). However, we also tried to simplify the API and have sensible defaults to make it usable for anyone / make ML research code cleaner :)
jerpint 3y ago
I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable Title should be instead “Library for low-code RLHF in python”
- tgsovlerkhgsel 3y ago
  Honestly the amount of complicated boilerplate that you're supposed to write from scratch every time you do something with ML (in some of the major frameworks) deterred me from touching anything ML-related for a long time. As far as I understand, what the training loop is supposed to be doing is pretty static and you don't need to understand most of it in order to "do ML", but at the same time it's full of complicated things to get right (which would be much easier to understand when controlled through well defined parameters instead of mixing boilerplate and config).
- patelajay285 3y ago
  This is developed for researchers, so I assure it’s very hackable and configurable. ;-) but appreciate the feedback on the title!
- brigadier132 3y ago
  I always appreciate these projects because I just dive into the code itself and copy out what I need once the wrapper becomes too much of a burden.
  - patelajay285 3y ago
    That’s totally valid and something we would even encourage! This project is for researchers so if there is a point where the abstraction is no longer useful, by all means configure, or subclass, or copy code.
  - rovr138 3y ago
    Of course. And they're not saying they don't have a place. They're saying why does it matter if it's 50 vs 60 or even 100. It's a wrapper, which should be less lines. That's the whole point. Abstracting things even further and making assumptions. Of course you can use them. Of course you can remove them after and use the underlying code. But the LOC shouldn't be the important part of it
imjonse 3y ago
The first paragraphs says RLHF can be used to align models, and the seconds say here's how to do it by using DPO. These two methods are not the same, and the latter is not an instance of the former.
- Der_Einzige 3y ago
  The latter is strictly superior to the former though. RlHF has been abandoned in the open source world.
  - numeri 3y ago
    I don't know about strictly superior. It's certainly strictly easier for people with a budget, who just need "good enough" results the first try. I don't have any evidence whatsoever, but I'd expect that enough tuning and retries can get squeeze a bit more performance out of RLHF than you can get out of DPO.
  - imjonse 3y ago
    I am just saying the intro paragraphs are confusing.
    - patelajay285 3y ago
      Thanks, appreciate the feedback, will update when I get a chance!
  - patelajay285 3y ago
    Yep, DPO is not technically “RL” and implicitly uses the LLM itself as a reward model, but training with DPO is far more stable for that reason.
    - patelajay285 3y ago
      I tend to agree @espadrine, it's semantics for the most part
    - 3y ago
lopkeny12ko 3y ago
It's not 50 lines of code if all the real work is done by importing a library... That's like saying, I can solve any problem in 2 lines of code. I'll publish a library for it first, then: import foo; foo.do_the_thing() Magic!
- therealdrag0 3y ago
  [relevant xkcd]
- antonvs 3y ago
  Software developers hate this one simple trick!
- peab 3y ago
  did people say the same thing when assembly code got abstracted away?
  - orblivion 3y ago
    There's levels of abstraction. "lines of Python" to me roughly means the standard library.
    - sciolist 3y ago
      Agreed. If it was "RLHF a LLM in <50 lines" I feel it would be clear, but "lines of Python" implies something different.
  - skelpmargyar 3y ago
    Importing a library is not abstraction any more than closing your eyes is abstracting the world to black.
    - __loam 3y ago
      The point of libraries is abstraction. You get to use the library without writing the library or knowing the underlying implementation.
mk_stjames 3y ago
I feel the preparation and loading of the dataset has been abstracted too far away. I have no idea what type of data format I need or how it is loaded for this (it is using a pre-prepared huggingface dataset?). If I have local data how should it be loaded? What does that even look like? Is it expecting some sort of JSON? When you get so far as to abstracting every step to loading a one-liner from huggingface, including the downloading of a prepared dataset with no example of doing the same on custom local dataset, you've abstracted too far to be useful for anyone other than the first user.
- ganeshkrishnan 3y ago
  The dataset is here I presume: https://huggingface.co/datasets/Intel/orca_dpo_pairs https://huggingface.co/datasets/Intel/orca_dpo_pairs you can look at the samples. Mostly its questions and accepted/rejected answers.
- patelajay285 3y ago
  Thanks for the question. This is built for ML researchers, so in examples we use the defacto source for datasets researchers often use, HF Hub. However, there is a lot of documentation on the site to help guide users. This documentation page shows you can load in data via local datasets as well. For example, JSON, CSV, text files, a local HF Dataset folder, or even from a Python `dict` or `list`: https://datadreamer.dev/docs/latest/datadreamer.steps.html#types-of-steps https://datadreamer.dev/docs/latest/datadreamer.steps.html#t... We'll definitely keep improving documentation, guides, and examples. We have a lot of it already, and more to come! This has only recently become a public project :) If anyone has any questions on using it, feel free to email me directly (email on the site and HN bio) for help in the meantime.
  - mk_stjames 3y ago
    I did glance at the docs first before commenting but I was looking in 'datasets' to try and understand importing a potential CSV/JOSN etc and all I saw was verbage on accessing the output. I would not have guessed that the base input data processing would have been filed under 'steps'. But now I kinda see how you are working, but I admit I'm not the target audience. If you want this to really take off for people outside of a very, very specific class of researchers... setup an example on your landing page that calls to a local JSON of user prompts/answers/rejects finetuning a llama model with your datadreamer.steps.JSONDataSource into the loader. Or, a txt file with the system/user/assistant prompts tagged and examples given. Yes, your 'lines of code' for your frontpage example may grow a bit! Maybe there are a lot of 'ML researchers' that are used to the type of super-abstract OOP API, load-it-from-huggingface-scheme-people you are targeting but also know that there are a ton that aren't.
bbstats 3y ago
I can abstract this to 2 lines
MrYellowP 3y ago
I don't prefer aligned models and I'm a human. It's not okay to claim that that's what humans prefer. There might be a subset of humans who can't handle words, but they're not even remotely in the majority. Algined models are dumber, treat everyone like they're stupid immature idiots who can't handle words and they're a wannabe moral authority.
spdustin 3y ago
It occurs to me that there must be a model that's been "aligned" opposite to the usual RLHF. Or has nobody done that?
proto-n 3y ago
Yeah well in bash I can do it in one line: `python train.py`. I hate examples like this, the 50loc statement is totally useless (and so is the code example, as I can't learning anything from it).
rrr_oh_man 3y ago
RLHF = Reinforcement Learning from Human Feedback
potatoman22 3y ago
This seems useful, thanks!
theptip 3y ago
Interested if local RLHF is actually viable; can you get meaningful steering from 1k feedback points on a narrow task? I feel that annotation count is achievable with a single dedicated annotator making a few comments per minute (though tedious), 10k would be a week of work so achievable for a very dedicated hobbyist, and 100k seems out of reach for a hobby project. Say for simple conversation usecases (eg customer support for a specific product, interactive fiction, things like that without deep technical knowledge). I was also wondering if it’s possible to do such RLHF for SD running locally.

223 points by patelajay285 3y ago

How do you normally do DPO? Is that built in to PyTorch or something? Theoretically the hard part is collecting the examples with rejections etc.

patelajay285 3y ago
Collecting data is hard, but the library is also a synthetic data generation library, so for example you can create the data for DPO fully synthetically, check out the self-rewarding LLMs example: https://datadreamer.dev/docs/latest/pages/get_started/quick_ https://datadreamer.dev/docs/latest/pages/get_started/quick_...

aethelyon 3y ago

This is cool, but the data collection is the hard part, right?

patelajay285 3y ago
Yes it is :), but the library is also a synthetic data generation library, so for example you can create the data for DPO fully synthetically, check out the self-rewarding LLMs example: https://datadreamer.dev/docs/latest/pages/get_started/quick_tour/self_rewarding.html https://datadreamer.dev/docs/latest/pages/get_started/quick_...
- sillysaurusx 3y ago
  I’m extremely skeptical of this approach. Until proven otherwise, with a model that users actually find useful, I don’t think this can work. It would be nice. But I’ve seen too many nice ideas completely fall apart in practice to accept this without some justification. Even if there are papers on the topic, and those papers show that the models rank highly according to some eval metrics, the only metric that truly matters is "the user likes the model and it solves their problems." By the way, on a separate topic, the 90/10 dataset split that you do in all of your examples turns out to be fraught with peril in practice. The issue is that the validation dataset quality turns out to be crucial, and randomly yeeting 10% of your data into the validation dataset without manual review is a recipe for problems.
  - patelajay285 3y ago
    It's a demo snippet of how to setup the workflow, it's not meant to be a working production example a self-rewarding model or a faithful reproduction of the original paper. Whether self-rewarding LLMs are a good idea or not, it's a valuable and very active area of research in the literature today. This is a library for ML researchers who should actively research and study these avenues along with the pitfalls you're mentioning. But in order for them to do that, building these workflows have to be accessible to them, which is what this library is meant to do. It's not meant for the "hobbyist" ML-community, they should not be using synthetic data today in this way, it would likely lead to subpar results for any practical model or task.
    - sillysaurusx 3y ago
      There’s a lot to unpack here. First, I’m an ML researcher. I don’t go around saying so because appeal to authority is bogus, but since every one of your comments seems to do this, it’s unavoidable. You say the code is for ML researchers, then flat out say that it’s not a working production example, nor is it a faithful reproduction of a paper. So what is it? Whether you want it to be or not, your audience is the hobbyist ML community, because without benchmarks to back up your code examples, no one from the research community will trust your examples without actual proof that they work. That’s the hard part of research, and it’s most of the effort. My advice is, write something that can train useful models. Implement a production grade workflow, and show some reasons why it works. If you’re trying to get the wider ML research community to buy in to this, there’s not much other way to do it. No one will want to take easy code that does the wrong thing, and most of your examples show the wrong thing to do, like the 90/10 split. You’re also a bit defensive about accepting feedback. Trust me that it’s better to accept that your code sucks and does the wrong thing, and then try to make it suck less and do the right thing. That’s how the majority of good software is written, unless you’re cperciva. But he’d also publish a paper explaining why his code is correct. Anyway, the whole point of posting this to HN is to get feedback on it. (If you were hoping that a bunch of people would suddenly use it, then you need to appeal to the hobbyist community. They’ve told you a bunch of things that you’ve straight up said is out of scope.) And it sounds like you were hoping for feedback from ML researchers. Maybe others will chime in, but for now, that’s the best I’ve got.

v4dok 3y ago

I feel like the current meta on finetuning LLMs is random accounts at X/Twitter. Google results are littered with SEO garbage or some kind of guides that fail to work the moment you need something slightly different.

rldjbpin 3y ago

it is very conflicting to see "do x in y LOC" in this field, especially when most of the workflow for different models are fragmented across non-overlapping frameworks/tooling. to actually do something from scratch or using the author's code requires adopting something esoteric just for this purpose. for these scenarios it is nice to appreciate hf and their abstraction. but the reinventing the wheel situation is very frustrating to work with. if you want to go beyond the demo, you have to deal with this painful reality. i hope there is more progress on this rather than making stacks of api.

cztomsik 3y ago

DPO is not RLHF.

patelajay285 3y ago

Hi everyone, there are no easy tools for synthetic data generation or training and aligning LLMs simply in Python. Most of the stuff out there are messy adhoc scripts. DataDreamer is an open source Python package with a nice API from the University of Pennsylvania that does all this that we’re actively developing. Will be here to answer questions. https://github.com/datadreamer-dev/DataDreamer https://github.com/datadreamer-dev/DataDreamer

baggiponte 3y ago
The API looks nice, congratulations. Will experiment with it. One small silly question: why did you choose to specify the dependencies inside the src dir with the requirements format - rather than inside the pyproject?
- patelajay285 3y ago
  Thanks! It makes it easier to run with the existing run scripts I have on our large university GPU cluster. :) no other reason

g4zj 3y ago

Very cool, but I can't help but feel like titles that reference low-LOC are a bit clickbait-y when nearly all the heavy lifting is done by imported libraries.

patelajay285 3y ago
Appreciate the feedback on the title, this is developed for ML researchers, so I assure there is a lot it’s doing under the hood to make this process easier (for example introducing automatic caching and resumability). However, we also tried to simplify the API and have sensible defaults to make it usable for anyone / make ML research code cleaner :)

jerpint 3y ago

I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable Title should be instead “Library for low-code RLHF in python”

tgsovlerkhgsel 3y ago
Honestly the amount of complicated boilerplate that you're supposed to write from scratch every time you do something with ML (in some of the major frameworks) deterred me from touching anything ML-related for a long time. As far as I understand, what the training loop is supposed to be doing is pretty static and you don't need to understand most of it in order to "do ML", but at the same time it's full of complicated things to get right (which would be much easier to understand when controlled through well defined parameters instead of mixing boilerplate and config).
patelajay285 3y ago
This is developed for researchers, so I assure it’s very hackable and configurable. ;-) but appreciate the feedback on the title!
brigadier132 3y ago
I always appreciate these projects because I just dive into the code itself and copy out what I need once the wrapper becomes too much of a burden.
- patelajay285 3y ago
  That’s totally valid and something we would even encourage! This project is for researchers so if there is a point where the abstraction is no longer useful, by all means configure, or subclass, or copy code.
- rovr138 3y ago
  Of course. And they're not saying they don't have a place. They're saying why does it matter if it's 50 vs 60 or even 100. It's a wrapper, which should be less lines. That's the whole point. Abstracting things even further and making assumptions. Of course you can use them. Of course you can remove them after and use the underlying code. But the LOC shouldn't be the important part of it

imjonse 3y ago

The first paragraphs says RLHF can be used to align models, and the seconds say here's how to do it by using DPO. These two methods are not the same, and the latter is not an instance of the former.

Der_Einzige 3y ago
The latter is strictly superior to the former though. RlHF has been abandoned in the open source world.
- numeri 3y ago
  I don't know about strictly superior. It's certainly strictly easier for people with a budget, who just need "good enough" results the first try. I don't have any evidence whatsoever, but I'd expect that enough tuning and retries can get squeeze a bit more performance out of RLHF than you can get out of DPO.
- imjonse 3y ago
  I am just saying the intro paragraphs are confusing.
  - patelajay285 3y ago
    Thanks, appreciate the feedback, will update when I get a chance!
- patelajay285 3y ago
  Yep, DPO is not technically “RL” and implicitly uses the LLM itself as a reward model, but training with DPO is far more stable for that reason.
  - patelajay285 3y ago
    I tend to agree @espadrine, it's semantics for the most part
  - 3y ago

lopkeny12ko 3y ago

It's not 50 lines of code if all the real work is done by importing a library... That's like saying, I can solve any problem in 2 lines of code. I'll publish a library for it first, then: import foo; foo.do_the_thing() Magic!

therealdrag0 3y ago
[relevant xkcd]
antonvs 3y ago
Software developers hate this one simple trick!
peab 3y ago
did people say the same thing when assembly code got abstracted away?
- orblivion 3y ago
  There's levels of abstraction. "lines of Python" to me roughly means the standard library.
  - sciolist 3y ago
    Agreed. If it was "RLHF a LLM in <50 lines" I feel it would be clear, but "lines of Python" implies something different.
- skelpmargyar 3y ago
  Importing a library is not abstraction any more than closing your eyes is abstracting the world to black.
  - __loam 3y ago
    The point of libraries is abstraction. You get to use the library without writing the library or knowing the underlying implementation.

mk_stjames 3y ago

I feel the preparation and loading of the dataset has been abstracted too far away. I have no idea what type of data format I need or how it is loaded for this (it is using a pre-prepared huggingface dataset?). If I have local data how should it be loaded? What does that even look like? Is it expecting some sort of JSON? When you get so far as to abstracting every step to loading a one-liner from huggingface, including the downloading of a prepared dataset with no example of doing the same on custom local dataset, you've abstracted too far to be useful for anyone other than the first user.

ganeshkrishnan 3y ago
The dataset is here I presume: https://huggingface.co/datasets/Intel/orca_dpo_pairs https://huggingface.co/datasets/Intel/orca_dpo_pairs you can look at the samples. Mostly its questions and accepted/rejected answers.
patelajay285 3y ago
Thanks for the question. This is built for ML researchers, so in examples we use the defacto source for datasets researchers often use, HF Hub. However, there is a lot of documentation on the site to help guide users. This documentation page shows you can load in data via local datasets as well. For example, JSON, CSV, text files, a local HF Dataset folder, or even from a Python `dict` or `list`: https://datadreamer.dev/docs/latest/datadreamer.steps.html#types-of-steps https://datadreamer.dev/docs/latest/datadreamer.steps.html#t... We'll definitely keep improving documentation, guides, and examples. We have a lot of it already, and more to come! This has only recently become a public project :) If anyone has any questions on using it, feel free to email me directly (email on the site and HN bio) for help in the meantime.
- mk_stjames 3y ago
  I did glance at the docs first before commenting but I was looking in 'datasets' to try and understand importing a potential CSV/JOSN etc and all I saw was verbage on accessing the output. I would not have guessed that the base input data processing would have been filed under 'steps'. But now I kinda see how you are working, but I admit I'm not the target audience. If you want this to really take off for people outside of a very, very specific class of researchers... setup an example on your landing page that calls to a local JSON of user prompts/answers/rejects finetuning a llama model with your datadreamer.steps.JSONDataSource into the loader. Or, a txt file with the system/user/assistant prompts tagged and examples given. Yes, your 'lines of code' for your frontpage example may grow a bit! Maybe there are a lot of 'ML researchers' that are used to the type of super-abstract OOP API, load-it-from-huggingface-scheme-people you are targeting but also know that there are a ton that aren't.

bbstats 3y ago

I can abstract this to 2 lines

MrYellowP 3y ago

I don't prefer aligned models and I'm a human. It's not okay to claim that that's what humans prefer. There might be a subset of humans who can't handle words, but they're not even remotely in the majority. Algined models are dumber, treat everyone like they're stupid immature idiots who can't handle words and they're a wannabe moral authority.

spdustin 3y ago

It occurs to me that there must be a model that's been "aligned" opposite to the usual RLHF. Or has nobody done that?

proto-n 3y ago

Yeah well in bash I can do it in one line: `python train.py`. I hate examples like this, the 50loc statement is totally useless (and so is the code example, as I can't learning anything from it).

rrr_oh_man 3y ago

RLHF = Reinforcement Learning from Human Feedback

potatoman22 3y ago

This seems useful, thanks!

theptip 3y ago

Interested if local RLHF is actually viable; can you get meaningful steering from 1k feedback points on a narrow task? I feel that annotation count is achievable with a single dedicated annotator making a few comments per minute (though tedious), 10k would be a week of work so achievable for a very dedicated hobbyist, and 100k seems out of reach for a hobby project. Say for simple conversation usecases (eg customer support for a specific product, interactive fiction, things like that without deep technical knowledge). I was also wondering if it’s possible to do such RLHF for SD running locally.