5 ms·
Show HN: LlamaGym – fine-tune LLM agents with online reinforcement learning
- adawg4 3y agoThanks for making this! Helps simplify it nicely
- 3abiton 3y agoInteresting project, basically a wrapper too around openai gym-like functionality that can handle open llms.
- KhoomeiK 3y agoYup, it does simplify LLM agent inference on Gym environments but the main technical contribution is reducing your would-be code overhead for online RL
- raidicy 3y agoThanks for creating this!
- KhoomeiK 3y agoTwitter thread: https://x.com/khoomeik/status/1766805213644800011?s=46 https://x.com/khoomeik/status/1766805213644800011?s=46
- internet101010 3y agoThank you for making this. Simplifying any aspect of RL is always welcome.
- dennisy 3y agoCan this be used outside of OpenAI environments? If yes I think an example would be great!
- KhoomeiK 3y agoGymnasium is now maintained by the Farama Fpundation, an open-source consortium, not OpenAI. But most RL environment work for the past 5+ years has been Gym-compliant. The TextWord example in the repo, for example, instantiates a Gym-style environment but it doesn’t import from Gymnasium (uses textworld.gym instead).
- neodypsis 3y agoVery interesting!
- zeroq 3y agoWhen 150 lines of boilerplate can land you the first page on HN, maybe it is, in fact, the end of programming?
- yen223 3y agoLet's not be one of those people who measure developer productivity by number of lines
- KhoomeiK 3y agoKarpathy’s micrograd [1] is literally 154 lines. Guess programming ended 4 years ago. [1] https://github.com/karpathy/micrograd https://github.com/karpathy/micrograd
- DSingularity 3y agoYou think autograd is boilerplate?
- klysm 3y agoI’m not really sure what your point is. Is it not remarkable that valuable things can be done in 150 lines?
- xpe 3y agoI agree with you. / Above, I wouldn't assume a single nor clearly intended "point". Reading it I got an impression more of concern, even fear. I'm guessing one underlying driver may be a concern that AI is creeping into more and more programming. Which is true.
- yowlingcat 3y agoCarmack's infamous fast inverse square root was only 13 lines. Measuring code by line metrics rather than its contents reflects shallow, questionable comprehension.
- sevagh 3y ago
- kayson 3y agoI want to make a Discord bot that impersonates all my friends and continues to refine the model as the conversations continue. Basically this [1] post, but with a more modern model and, ideally, reinforcement learning. Seems like this would fit the bill.... Is there anything else that would make this easier? [1] https://www.izzy.co/blogs/robo-boys.html https://www.izzy.co/blogs/robo-boys.html
- mzl 3y agoYou could perhaps adapt the Doppel Bot slack bot from Modal Labs: https://github.com/modal-labs/doppel-bot https://github.com/modal-labs/doppel-bot
- potatoman22 3y agoCould someone help me understand the kinds of things you can build with this? Is this like RLHF?
- katzenversteher 3y agoFrom the title I misunderstood what it does. However, now I'm wondering if what I thought is was (don't ask my why I thought it) is possible: I have a PC that is able to run e.g. Mistral Instruct 7B Q4 inference with around 30 token/s. How (computation and memory) expensive would it be to also run backpropagation in addition to inference? I'm aware that the models are typically fed with much more and better data than what is typically provided during normal conversations but on the other hand if I could finetune my local model a teeny tiny bit during during / after each conversation I have with it anyways, it would after a while be perfectly customize for me. I'm also aware that this could be problematic for models that are used by multiple users but my intended use case would be personal use by a single user.
- dartos 3y agoVery expensive. AFAIK the model can’t be quantized during backprop, so right there you’d need a ton of RAM. Backprop is faster bc it can be parallelized, but IIRC you need to hold an entire copy of the model for each backprop process.
- scribu 3y agoActually, there have been attempts to do quantized backprop, but not sure how successfully.
- tveita 3y agoFor an idea of what's possible you might be interested in this story that was just on HN where they fine tune a quantized 70b model in 48GB VRAM: https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html
- SuhanaJabin 3y agoSimplified the concept. Nicely done!
- ponderchan 3y agollamagym.com for sale