7 ms·

The upcoming GPT-3 moment for RL

232 points by jxmorris12 1y ago

fnord77 1y ago
where can I learn about the nitty gritty of RL and RL training? For instance, I want to understand how say software could be used as input (tokenization/vectorization of the code?)
criemen 1y ago
They have a point about RLs increasing importance. From my outsider perspective, all major advances in model capabilities in the last period of time come from RL, so it's natural to expect that we can "milk" RL more for performance gains. Scaling RL is a natural way to attempt that. What I don't necessarily see is the generalization factor - say, we improve software engineering and math performance through RL learning (probably easier for software engineering than math due to available training corpus). If that generalization factor doesn't hold, due the economics still work out? An expert-level software model would be useful to our profession, sure, but would it be enough to recoup the training costs if it's not applicable to other industries?
- janalsncm 1y ago
  One detail the OP glosses over is the increasing costs of RL as the sequence length increases. If we’re just reasoning through an simple arithmetic problem, it’s a pretty manageable number of reasoning tokens and answer tokens. For a complete piece of software the answer might be 10 million tokens, and that doesn’t even count the reasoning. Now imagine that there was a mistake at some point. The model will need to go back to fix it, and understand the cascade of things the bugfix changed. It might be possible to keep that all in the context window but that seems like it won’t scale.
  - criemen 1y ago
    I'd expect that's manageable by some sort of agent-of-agent pattern. You have a high-level planning instance that calls upon fresh LLM instances (new context window!) for executing more targeted tasks or bug-fixes. Currently, an LLM with everything under the sun in the context window behaves rather poorly and gets confused by that, even if we're not exceeding the context window length. Although it'd be certainly also interesting to train for increasing the maximum _actually_ usable context window length, I don't know how feasible that would be.
amelius 1y ago
Step 1. Train a VLM to supervise the RL training. Step 2. Train the RL network. In the mean time drink coffee or work on plan of world domination.
- criemen 1y ago
  My understanding is that this is essentially how RLHF works, and it doesn't scale. As you run RL for longer, the model will learn how to cheat the imperfections of the grader, instead of getting better at the task at hand. Therefore, to scale RL you really need good graders, and determinism is king.
  - clbrmbr 1y ago
    Do you think constitutional approaches would help here? (Verifiable reward for the main score, but then asking the model to self-critique for security and quality.)
  - amelius 1y ago
    You're talking about training an LLM. I'm talking about training robotic/motor skills and haptic feedback.
mohsen1 1y ago
I’ve been exploring this too, since I rely on LLMs a lot to build software. I’ve noticed that our dev loop-writing, testing-is often mostly human-guided, but language models frequently outperform us in reasoning. If we plug in more automation; MCP tools controlling browsers, documentation readers, requirement analysers, we can make the cycle much more automated, with less human involvement. This article suggests scaling up RL by exposing models to thousands of environments I think we can already achieve something similar by chaining multiple agents: 1. A “requirement” agent that uses browser tools to craft detailed specs from docs. 2. A coding agent that sets up environments (Docker, build tools) via browser or CLI. 3. A testing agent that validates code against specs, again through tooling. 4. A feedback loop where the tester guides the coder based on results. Put together, this system becomes a fully autonomous development pipeline-especially for small projects. In practice, I’ve left my machine running overnight, and these agents propose new features, implement them, run tests, and push to repo once they pass. It works surprisingly well. The main barrier is cost—spinning up many powerful models is expensive. But on a modest scale, this method is remarkably effective.
- YetAnotherNick 1y ago
  RL is a training method and it improves the model itself. So basically one step(e.g. successful test run, finding search result) could create positive and negative examples for the other step(e.g. coding agent, search agent). And using this the base itself will improve to satisfy other demands and if it reaches close to 100% accuracy(which I believe it could as models mostly fail due to dumb mistakes in tests), you don't need the testing agent altogether.
  - szundi 1y ago
    [dead]
- kuruczgy 1y ago
  > but language models frequently outperform us in reasoning what 99% of the time their reasoning is laughable. Or even if their reasoning is on the right track, they often just ignore it in the final answer, and do the stupid thing anyway.
Michelangelo11 1y ago
> Each replication task consists of a detailed specification and a reference implementation. The central idea is that AI models are trained to produce an implementation that precisely matches the reference behavior. This clear-cut approach significantly simplifies evaluation, as the grading criteria are objective and direct: either the generated implementation behaves identically to the reference, or it doesn’t. OK, but then you have to produce the detailed specification, working backward from the reference implementation. This is extremely non-trivial and it significantly weakens the TFA's parallels to pre-training, in which you don't need really need inputs other than raw text corpora. I'm not saying this eliminates the idea outright, but I do think it hobbles it badly.
- dist-epoch 1y ago
  The detailed specification is the output for a particular input. And you can use a fuzzer to augument that.
- YetAnotherNick 1y ago
  When prompted correctly, models could generate good specification in form of pretty exhaustive tests. While all tests have weaknesses and are not formal specification, they could get us 99% there.
- vessenes 1y ago
  I’d like to courteously disagree. I think existing models and existing tools are good enough to bootstrap this at least. I’d propose the following architecture: Step 1: Microsoft phi style - read code and write specifications using a frontier model. You could use an ensemble here to nitpick the spec; it’s only going to get written once. We also have of course many many rfcs and codebases that conform to them or where they do not we have an existing repository of bug reports, patches, forum complaints, etc. Step 2-4: implement multilayer evaluation: does it compile? Does an existing model think the code complies with the spec on inspection? When it’s run on qemu are the key evals the same as the original software? I propose most of steps 2-4 are automatable and rely on existing tooling and provide a framework that is, if not cheap, achievable. I’m also sure someone could improve this plan with a few more minutes of thought. To me the interesting question is - will this add capabilities at current model sizes? My prior is yes in that the current behemoth size models feel like they are only incrementally better than 1/10 size distills. I interpret that to mean we haven’t gotten the most out of these larger scales. I will note Dario disagrees on this - he’s publicly said we need at least 10x more scale than we have now.
Szpadel 1y ago
with RL it's hard to define score function in many categories. rhis is especially visible in current coding capabilities. LLM will very often create sloppy solutions because they work well in RL. hardcoding API keys? ignoring errors? disabling lints? those pass in automated evaluation therefore are reinforced in training. are they good solutions? of course not. It's very hard to define (in way to create lints) what makes core readable and maintainable. Using other LLM for this task could cause original model to game the system by abusing some weaknesses in the other model. for other tasks, how do you even evaluate thinks like eg user experience/app design? how to properly evaluate pelican ridding bicycle?
- szundi 1y ago
  [dead]
- CuriouslyC 1y ago
  You can project them onto a linear space by gathering enough pairwise evaluations. PelicanElo.
- esperent 1y ago
  > hardcoding API keys? ignoring errors? disabling lints? These kind of "rookie mistakes" are not things that any modern LLM is likely to do. Indeed, I had to argue quite strongly with Gemini recently when I was learning a new tool (so basically just playing around with a fully local setup) and I hardcoded an API key then tried to commit it. The LLM did NOT like that! I had to carefully explain that this was a toy repo. The argument against this (by Gemini) was that toy repos often grow into production tools so it's best to follow basic security rules from the start. Which, to be fair, is a good argument. I still committed the key though (and deleted the repo a day or so later).
  - levzzz 1y ago
    [dead]
nikanj 1y ago
What is RL in this context?
- hazn 1y ago
  Reinforcement Learning
gcanyon 1y ago
The "GPT-3 moment" framing is a bit hype-y I think? GPT-3 eliminated the need for task-specific fine-tuning, but from the article RL wouldn't replace LLM-style pretraining. So this is more of an incremental advance than the paradigm shift GPT-3 represented. That said, if it unlocks RL generalization that would be huge. The core claim that massive-scale RL will unlock generalization doesn't seem that surprising since we've seen the scaling hypothesis play out across ML. But "replication training" on software is interesting: learning by copying existing programs potentially unlocks a ton of complex training data with objective evaluation criteria. To me, the big unanswered question is whether skills learned from replicating software would generalize to other reasoning tasks. That's a significant "if" - great if it works, pointless if it doesn't.
- kevindamm 1y ago
  It's a very big "if" because other fields are comparatively underspecified. There's no equivalent to a compiler or interpreter in most cases (with spreadsheets being the lingua franca that comes even close for most industries). It would "work" but I think it will need even more scrutiny by experts to confirm what's correct and what needs to be re-generated. Please please no vibe accounting.
  - cjblomqvist 1y ago
    > Please please no vibe accounting. Funny you mention; There are multiple companies in Sweden working on AI/ML based accounting. It's not so different from AI/ML based automated driving.
    - kevindamm 1y ago
      I've seen some of those but all of the ones I've looked at also had a panel of experts who could give it a once-over (or re-work) before sending it back to the client. I'd compare it more to cruise control or driver-assist but not quite automated driving.
      - cjblomqvist 1y ago
cs702 1y ago
TL;DR: The OP believes that if we train large AI models via RL to duplicate the behavior of existing software (for example, train them to duplicate the behavior of an existing spreadsheet, an existing command-line tool, or an existing application), large AI models will get good at: * reading and understanding long, complicated, detailed instructions, * executing those instructions meticulously and precisely, without errors, * noticing its mistakes, if there are any along the way, and recovering from them, * not settling prematurely for solutions that look "good enough" but aren't, and * undertaking large, complicated projects which previously could be completed only by teams of human experts. There's a good chance the OP is right, in my view. We sure live in interesting times!
throwaway992673 1y ago
What a relief, I was horrified this was going to be an atrocious Rocket League update.
pfdietz 1y ago
I like this idea. It's adjacent to differential testing of manually created software (as in compilers) and to mutation testing for evaluation and generation of test suites.
OtherShrezzing 1y ago
> Simple command-line tools that implement obscure hashing and encryption algorithms are straightforward initial targets, but this approach can easily extend to more complex software, such as websites, professional software, and games. >Each replication task consists of a detailed specification and a reference implementation. The central idea is that AI models are trained to produce an implementation that precisely matches the reference behavior. I really don't see the connection from the statements in the article's content, and the assertion near the start that: >Doing this effectively will produce RL models with strong few-shot, task-agnostic abilities capable of quickly adapting to entirely new tasks. There's no clear reason outlined in the piece to describe why narrow & well-scoped 1-person-day tasks might scale up to 10,000-person-year projects. If they did, we should expect far more 10,000-person-year projects in the real economy, because the learning curve for firms scaling would be something approximating a straight line. There are very few 10,000-person-year projects, and very many 1-person-day projects. It seems more like this will spend an unimaginable amount of compute, in order to produce models which are incredibly good at a very precise form of IP theft, and not especially good at any generalisable skills. It's so ludicrously rare that an engineer (or author, illustrator, etc) is tasked with "create a pixel-perfect reimplementation of this existing tool".
- rightbyte 1y ago
  > models which are incredibly good at a very precise form of IP theft A smell big success? Copyright laundering is the killer app of AI this far.
m3kw9 1y ago
Upcoming sure, everything can be upcoming like the famous ASI
- raccomandoo 1y ago
  When I see all the bad verbal reasoning being spewed around LLMs it becomes easier to understand why so many people think these LLMs are intelligent. Non rigorous reasoning is at the root of the problem here. AI hype is often indistinguishable fro AI slop, because those who believe it are also not very good at demanding and asserting rigor.
curious_cat_163 1y ago
> Rather than fine-tuning models on a small number of environments, we expect the field will shift toward massive-scale training across thousands of diverse environments. This is a great hypothesis for you to prove one way or the other. > Doing this effectively will produce RL models with strong few-shot, task-agnostic abilities capable of quickly adapting to entirely new tasks. I am not sure if I buy that, frankly. Even if you were to develop radically efficient means to create "effective and comprehensive" test suites that power replication training, it is not at all a given that it will translate to entirely new tasks. Yes, there is the bitter lesson and all that but we don't know if this is _the_ right hill to climb. Again, at best, this is a hypothesis. > But achieving this will require training environments at a scale and diversity that dwarf anything currently available. Yes. You should try it. Let us know if it works. All the best!
visarga 1y ago
This works great for software, math and games where you can have cheap validation. But what about messy real world tasks? I think hindsight learning from chat logs could fit the bill. What do I mean? Imagine a long conversation. It is hard to judge if an AI response was useful or not immediately, but if you know the following 20 messages, it might be easy to infer. Not only you can see how it went, but sometimes you get real world validation. For example a user comes to a LLM with a task, takes an idea, tries it in reality. Later they return, maybe in a new chat session, and continue iterating. You get real world testing of LLM responses through people. This can be used to generate "preference scores", and train a preference model, with which you can do RLHF. So the user privacy is protected. I call this the human-AI experience flywheel. Of course the larger the user base, the more experience the model collects. At the moment OpenAI has 500M users, they probably generate 0.5T interactive tokens/day. Those tokens go both into human brains and LLM logs. It’s not about environment engineering anymore, it's about consequence harvesting. Meaningful validation emerges from systems actually being used by humans for real purposes.
- adampk 1y ago
  I worked through this for a tax company. They had a huge pile of artifacts from tax questions worked up for clients. What we did is we "reverse engineered" the process of the questions that would lead to that tax memo and the research steps to find the sources and conclusions. It worked well and we were able to replicate the process which the SME's created these memos. For a given tax question, could you come up with the same memo quoting the same sources and same conclusion?
throw5829646 1y ago
I had the opportunity to meet Tamay not too long ago, very sharp guy. A lot of people I know are working on approaches to meta RL or exploration-based RL, where the goal is to build a foundation model of sorts with a really good world model across diverse tasks, and can predict good policies (or good policy updates) from limited rollouts and/or a sparse reward signal. We're not there quite yet, but as Altman recently said, "we don't have AGI until we have something that learns continuously", and there's a huge race in this space to make that happen.
nullc 1y ago
> Simple command-line tools that implement obscure hashing and encryption algorithms So your plan is to train a MLP to black box replicate complex and highly non-linear encryption algorithms through gradient descent?
- janalsncm 1y ago
  Not black box, no. The spec presumably tells the model everything it needs to know or look up. But in contrast with Fibonacci, the exact code is unlikely to be in the training set verbatim.
Zenst 1y ago
A few things that this made me think about: 1) If only there was a cryptocurrency tied to training AI models and make crypto grinding useful than maths that solve no real-world problem external to the token creation itself. 2) Larger and larger AI models, you start to get more hallucinations, maybe we should focus on dedicated highly tuned models for dedicated aspects and have a higher up conductor model that knows what to farm out to which models and from there combine and send out further requests to other models etc to come to a result. Certainly, the need for highly tuned niche models, after all, language recognition as an example, a model that could identify the language, local dialect and accent, that would then use a language model tuned better for that speaker it is recognising. That approach feels like the way over one large model that does it all itself.
- N3cr0ph4g1st 1y ago
  For 1) isn't prime intellect doing that or something like it?
- rcbdev 1y ago
  I remember mining something called Gridcoin over a decade ago. It's a cryptocurrency tied to the BOINC project and rewards providing computing power to science. I have sadly lost access to my wallet since.
- packetlost 1y ago
  There are a couple: https://ambient.xyz/ https://ambient.xyz/ https://www.primeintellect.ai/ https://www.primeintellect.ai/
ThrowawayTestr 1y ago
I like to imagine a world where we embraced nuclear and renewables where energy is free and we could throw gigawatthours at whatever crazy computation we can think of.
jxmorris12 1y ago
I recently wrote a post about scaling RL that has some similar ideas: > How to Scale RL to 10^26 FLOPs (blog.jxmo.io/p/how-to-scale-rl-to-1026-flops) The basic premise behind both essays is that for AI to make another big jump in capabilities, we need to find new data to train on. My proposal was reusing text from the Internet and doing RL on next-token prediction. The linked post here instead suggests doing 'replication training', which they define as "tasking AIs with duplicating existing software products, or specific features within them".
ltbarcly3 1y ago
This makes no sense. RL training data is predicated on past behavior of the agent. Whoever wrote this doesn't seem to fundamentally grasp what they are saying. LLMs can be trained on an unsupervised way on static documents. That is really the key feature that lets them be as smart and effective as they are. If you had every other technology that LLMs are built on, and you didn't have hundreds of terabytes of text laying around, there would be no practical way to make them even a tiny tiny fraction as effective as they are currently.
- sva_ 1y ago
  > Whoever wrote this doesn't seem to fundamentally grasp what they are saying. RL != only online learning. There's a ton of research on offline and imitation-based RL where the training data isn't tied to an agents past policy - which is exactly what this article is pointing to.
  - physix 1y ago
    I'm not sufficiently familiar with the details on ML to assess the proposition made in the article. From my understanding, RL is a tuning approach on LLMs, so the outcome is still the same kind of beast, albeit with a different parameter set. So empirically, I actually thought that the lead companies would already be strongly focused on improving coding capabilities, since this is where LLMs are very effective, and where they have huge cashflows from token consumptions. So, either the motivation isn't there, or they're already doing something like that, or they know it's not as effective as the approaches they already have. I wonder which one it is.
    - sva_ 1y ago
      > From my understanding, RL is a tuning approach on LLMs, What you're referring to is actually just one application of RL (RLHF). RL itself is much more than that
      - physix 1y ago
        Actually I didn't. Correct me if I am wrong, but my understanding is that RL is still an LLM tuning approach, i.e. an optimization of its parameter set, no matter if it's done at scale or via HF.
novia 1y ago
I wish these guys would hire me
simonebrunozzi 1y ago
Bad title. I thought RL = Real Life, instead of Reinforcement Learning. It should be clearly indicated in the title.
- peteforde 1y ago
  I don't know if not ever defining the acronym is a sort of passive aggressive "if you have to ask, you're not the audience we're looking to hire" filter, but I am with you. Having AI investment dollars doesn't excuse you from standard expectations that you clearly state what you're describing.
siwatanejo 1y ago
What is RL? Real Life? Or Reinforced Learning?
- dboreham 1y ago
  Real Life would be IRL.
octoberfranklin 1y ago
Isn't "replication training" just adversarial training, as in GANs? One network tries to clean-room implement the hash function. The other network tries to find an input for which the reference implementation behaves differently.
ankit219 1y ago
Think we will see something like this in the near future. But there are two very bold claims: one is hope that RL will lead to generalization in other domains, which based on current evidence seems far fetched. Other is the softwares and specs are good "RL-gym" data. The whole idea behind RL is that model explores the best paths, but if the softwares written are suboptimal when it comes to agent interaction paradigm (they were written for humans, not agents), there is a high chance even with the compute the model would be suboptimal. There is a parallel trend where current AI systems are abstracting entire workflows. Not accounting for that would lead to outcomes which are not cognizant of the current requirements.
randomNumber7 1y ago
The issue is that it is a stretch to call it reinforcement learning when all we do currently (in the context of LLMs) is to multiply the reward with the learning rate. It sounds cool as marketing. It helps improve LLMs a bit. And it will never yield s.th. like an AGI or anything that is "reasoning". Unless you also redefine the word reasoning of course.
lysecret 1y ago
Excellent commentary from Kaparthy on RL: https://x.com/karpathy/status/1944435412489171119 https://x.com/karpathy/status/1944435412489171119
deleted 1y ago
[deleted]

232 points by jxmorris12 1y ago

where can I learn about the nitty gritty of RL and RL training? For instance, I want to understand how say software could be used as input (tokenization/vectorization of the code?)

criemen 1y ago

They have a point about RLs increasing importance. From my outsider perspective, all major advances in model capabilities in the last period of time come from RL, so it's natural to expect that we can "milk" RL more for performance gains. Scaling RL is a natural way to attempt that. What I don't necessarily see is the generalization factor - say, we improve software engineering and math performance through RL learning (probably easier for software engineering than math due to available training corpus). If that generalization factor doesn't hold, due the economics still work out? An expert-level software model would be useful to our profession, sure, but would it be enough to recoup the training costs if it's not applicable to other industries?

janalsncm 1y ago
One detail the OP glosses over is the increasing costs of RL as the sequence length increases. If we’re just reasoning through an simple arithmetic problem, it’s a pretty manageable number of reasoning tokens and answer tokens. For a complete piece of software the answer might be 10 million tokens, and that doesn’t even count the reasoning. Now imagine that there was a mistake at some point. The model will need to go back to fix it, and understand the cascade of things the bugfix changed. It might be possible to keep that all in the context window but that seems like it won’t scale.
- criemen 1y ago
  I'd expect that's manageable by some sort of agent-of-agent pattern. You have a high-level planning instance that calls upon fresh LLM instances (new context window!) for executing more targeted tasks or bug-fixes. Currently, an LLM with everything under the sun in the context window behaves rather poorly and gets confused by that, even if we're not exceeding the context window length. Although it'd be certainly also interesting to train for increasing the maximum _actually_ usable context window length, I don't know how feasible that would be.

amelius 1y ago

Step 1. Train a VLM to supervise the RL training. Step 2. Train the RL network. In the mean time drink coffee or work on plan of world domination.

criemen 1y ago
My understanding is that this is essentially how RLHF works, and it doesn't scale. As you run RL for longer, the model will learn how to cheat the imperfections of the grader, instead of getting better at the task at hand. Therefore, to scale RL you really need good graders, and determinism is king.
- clbrmbr 1y ago
  Do you think constitutional approaches would help here? (Verifiable reward for the main score, but then asking the model to self-critique for security and quality.)
- amelius 1y ago
  You're talking about training an LLM. I'm talking about training robotic/motor skills and haptic feedback.

mohsen1 1y ago

I’ve been exploring this too, since I rely on LLMs a lot to build software. I’ve noticed that our dev loop-writing, testing-is often mostly human-guided, but language models frequently outperform us in reasoning. If we plug in more automation; MCP tools controlling browsers, documentation readers, requirement analysers, we can make the cycle much more automated, with less human involvement. This article suggests scaling up RL by exposing models to thousands of environments I think we can already achieve something similar by chaining multiple agents: 1. A “requirement” agent that uses browser tools to craft detailed specs from docs. 2. A coding agent that sets up environments (Docker, build tools) via browser or CLI. 3. A testing agent that validates code against specs, again through tooling. 4. A feedback loop where the tester guides the coder based on results. Put together, this system becomes a fully autonomous development pipeline-especially for small projects. In practice, I’ve left my machine running overnight, and these agents propose new features, implement them, run tests, and push to repo once they pass. It works surprisingly well. The main barrier is cost—spinning up many powerful models is expensive. But on a modest scale, this method is remarkably effective.

YetAnotherNick 1y ago
RL is a training method and it improves the model itself. So basically one step(e.g. successful test run, finding search result) could create positive and negative examples for the other step(e.g. coding agent, search agent). And using this the base itself will improve to satisfy other demands and if it reaches close to 100% accuracy(which I believe it could as models mostly fail due to dumb mistakes in tests), you don't need the testing agent altogether.
- szundi 1y ago
  [dead]
kuruczgy 1y ago
> but language models frequently outperform us in reasoning what 99% of the time their reasoning is laughable. Or even if their reasoning is on the right track, they often just ignore it in the final answer, and do the stupid thing anyway.

Michelangelo11 1y ago

> Each replication task consists of a detailed specification and a reference implementation. The central idea is that AI models are trained to produce an implementation that precisely matches the reference behavior. This clear-cut approach significantly simplifies evaluation, as the grading criteria are objective and direct: either the generated implementation behaves identically to the reference, or it doesn’t. OK, but then you have to produce the detailed specification, working backward from the reference implementation. This is extremely non-trivial and it significantly weakens the TFA's parallels to pre-training, in which you don't need really need inputs other than raw text corpora. I'm not saying this eliminates the idea outright, but I do think it hobbles it badly.

dist-epoch 1y ago
The detailed specification is the output for a particular input. And you can use a fuzzer to augument that.
YetAnotherNick 1y ago
When prompted correctly, models could generate good specification in form of pretty exhaustive tests. While all tests have weaknesses and are not formal specification, they could get us 99% there.
vessenes 1y ago
I’d like to courteously disagree. I think existing models and existing tools are good enough to bootstrap this at least. I’d propose the following architecture: Step 1: Microsoft phi style - read code and write specifications using a frontier model. You could use an ensemble here to nitpick the spec; it’s only going to get written once. We also have of course many many rfcs and codebases that conform to them or where they do not we have an existing repository of bug reports, patches, forum complaints, etc. Step 2-4: implement multilayer evaluation: does it compile? Does an existing model think the code complies with the spec on inspection? When it’s run on qemu are the key evals the same as the original software? I propose most of steps 2-4 are automatable and rely on existing tooling and provide a framework that is, if not cheap, achievable. I’m also sure someone could improve this plan with a few more minutes of thought. To me the interesting question is - will this add capabilities at current model sizes? My prior is yes in that the current behemoth size models feel like they are only incrementally better than 1/10 size distills. I interpret that to mean we haven’t gotten the most out of these larger scales. I will note Dario disagrees on this - he’s publicly said we need at least 10x more scale than we have now.

Szpadel 1y ago

with RL it's hard to define score function in many categories. rhis is especially visible in current coding capabilities. LLM will very often create sloppy solutions because they work well in RL. hardcoding API keys? ignoring errors? disabling lints? those pass in automated evaluation therefore are reinforced in training. are they good solutions? of course not. It's very hard to define (in way to create lints) what makes core readable and maintainable. Using other LLM for this task could cause original model to game the system by abusing some weaknesses in the other model. for other tasks, how do you even evaluate thinks like eg user experience/app design? how to properly evaluate pelican ridding bicycle?

szundi 1y ago
[dead]
CuriouslyC 1y ago
You can project them onto a linear space by gathering enough pairwise evaluations. PelicanElo.
esperent 1y ago
> hardcoding API keys? ignoring errors? disabling lints? These kind of "rookie mistakes" are not things that any modern LLM is likely to do. Indeed, I had to argue quite strongly with Gemini recently when I was learning a new tool (so basically just playing around with a fully local setup) and I hardcoded an API key then tried to commit it. The LLM did NOT like that! I had to carefully explain that this was a toy repo. The argument against this (by Gemini) was that toy repos often grow into production tools so it's best to follow basic security rules from the start. Which, to be fair, is a good argument. I still committed the key though (and deleted the repo a day or so later).
- levzzz 1y ago
  [dead]

nikanj 1y ago

What is RL in this context?

hazn 1y ago
Reinforcement Learning

gcanyon 1y ago

The "GPT-3 moment" framing is a bit hype-y I think? GPT-3 eliminated the need for task-specific fine-tuning, but from the article RL wouldn't replace LLM-style pretraining. So this is more of an incremental advance than the paradigm shift GPT-3 represented. That said, if it unlocks RL generalization that would be huge. The core claim that massive-scale RL will unlock generalization doesn't seem that surprising since we've seen the scaling hypothesis play out across ML. But "replication training" on software is interesting: learning by copying existing programs potentially unlocks a ton of complex training data with objective evaluation criteria. To me, the big unanswered question is whether skills learned from replicating software would generalize to other reasoning tasks. That's a significant "if" - great if it works, pointless if it doesn't.

kevindamm 1y ago
It's a very big "if" because other fields are comparatively underspecified. There's no equivalent to a compiler or interpreter in most cases (with spreadsheets being the lingua franca that comes even close for most industries). It would "work" but I think it will need even more scrutiny by experts to confirm what's correct and what needs to be re-generated. Please please no vibe accounting.
- cjblomqvist 1y ago
  > Please please no vibe accounting. Funny you mention; There are multiple companies in Sweden working on AI/ML based accounting. It's not so different from AI/ML based automated driving.
  - kevindamm 1y ago
    I've seen some of those but all of the ones I've looked at also had a panel of experts who could give it a once-over (or re-work) before sending it back to the client. I'd compare it more to cruise control or driver-assist but not quite automated driving.
    - cjblomqvist 1y ago

cs702 1y ago

TL;DR: The OP believes that if we train large AI models via RL to duplicate the behavior of existing software (for example, train them to duplicate the behavior of an existing spreadsheet, an existing command-line tool, or an existing application), large AI models will get good at: * reading and understanding long, complicated, detailed instructions, * executing those instructions meticulously and precisely, without errors, * noticing its mistakes, if there are any along the way, and recovering from them, * not settling prematurely for solutions that look "good enough" but aren't, and * undertaking large, complicated projects which previously could be completed only by teams of human experts. There's a good chance the OP is right, in my view. We sure live in interesting times!

throwaway992673 1y ago

What a relief, I was horrified this was going to be an atrocious Rocket League update.

pfdietz 1y ago

I like this idea. It's adjacent to differential testing of manually created software (as in compilers) and to mutation testing for evaluation and generation of test suites.

OtherShrezzing 1y ago

> Simple command-line tools that implement obscure hashing and encryption algorithms are straightforward initial targets, but this approach can easily extend to more complex software, such as websites, professional software, and games. >Each replication task consists of a detailed specification and a reference implementation. The central idea is that AI models are trained to produce an implementation that precisely matches the reference behavior. I really don't see the connection from the statements in the article's content, and the assertion near the start that: >Doing this effectively will produce RL models with strong few-shot, task-agnostic abilities capable of quickly adapting to entirely new tasks. There's no clear reason outlined in the piece to describe why narrow & well-scoped 1-person-day tasks might scale up to 10,000-person-year projects. If they did, we should expect far more 10,000-person-year projects in the real economy, because the learning curve for firms scaling would be something approximating a straight line. There are very few 10,000-person-year projects, and very many 1-person-day projects. It seems more like this will spend an unimaginable amount of compute, in order to produce models which are incredibly good at a very precise form of IP theft, and not especially good at any generalisable skills. It's so ludicrously rare that an engineer (or author, illustrator, etc) is tasked with "create a pixel-perfect reimplementation of this existing tool".

rightbyte 1y ago
> models which are incredibly good at a very precise form of IP theft A smell big success? Copyright laundering is the killer app of AI this far.

m3kw9 1y ago

Upcoming sure, everything can be upcoming like the famous ASI

raccomandoo 1y ago
When I see all the bad verbal reasoning being spewed around LLMs it becomes easier to understand why so many people think these LLMs are intelligent. Non rigorous reasoning is at the root of the problem here. AI hype is often indistinguishable fro AI slop, because those who believe it are also not very good at demanding and asserting rigor.

curious_cat_163 1y ago

> Rather than fine-tuning models on a small number of environments, we expect the field will shift toward massive-scale training across thousands of diverse environments. This is a great hypothesis for you to prove one way or the other. > Doing this effectively will produce RL models with strong few-shot, task-agnostic abilities capable of quickly adapting to entirely new tasks. I am not sure if I buy that, frankly. Even if you were to develop radically efficient means to create "effective and comprehensive" test suites that power replication training, it is not at all a given that it will translate to entirely new tasks. Yes, there is the bitter lesson and all that but we don't know if this is _the_ right hill to climb. Again, at best, this is a hypothesis. > But achieving this will require training environments at a scale and diversity that dwarf anything currently available. Yes. You should try it. Let us know if it works. All the best!

visarga 1y ago

This works great for software, math and games where you can have cheap validation. But what about messy real world tasks? I think hindsight learning from chat logs could fit the bill. What do I mean? Imagine a long conversation. It is hard to judge if an AI response was useful or not immediately, but if you know the following 20 messages, it might be easy to infer. Not only you can see how it went, but sometimes you get real world validation. For example a user comes to a LLM with a task, takes an idea, tries it in reality. Later they return, maybe in a new chat session, and continue iterating. You get real world testing of LLM responses through people. This can be used to generate "preference scores", and train a preference model, with which you can do RLHF. So the user privacy is protected. I call this the human-AI experience flywheel. Of course the larger the user base, the more experience the model collects. At the moment OpenAI has 500M users, they probably generate 0.5T interactive tokens/day. Those tokens go both into human brains and LLM logs. It’s not about environment engineering anymore, it's about consequence harvesting. Meaningful validation emerges from systems actually being used by humans for real purposes.

adampk 1y ago
I worked through this for a tax company. They had a huge pile of artifacts from tax questions worked up for clients. What we did is we "reverse engineered" the process of the questions that would lead to that tax memo and the research steps to find the sources and conclusions. It worked well and we were able to replicate the process which the SME's created these memos. For a given tax question, could you come up with the same memo quoting the same sources and same conclusion?

throw5829646 1y ago

I had the opportunity to meet Tamay not too long ago, very sharp guy. A lot of people I know are working on approaches to meta RL or exploration-based RL, where the goal is to build a foundation model of sorts with a really good world model across diverse tasks, and can predict good policies (or good policy updates) from limited rollouts and/or a sparse reward signal. We're not there quite yet, but as Altman recently said, "we don't have AGI until we have something that learns continuously", and there's a huge race in this space to make that happen.

nullc 1y ago

> Simple command-line tools that implement obscure hashing and encryption algorithms So your plan is to train a MLP to black box replicate complex and highly non-linear encryption algorithms through gradient descent?

janalsncm 1y ago
Not black box, no. The spec presumably tells the model everything it needs to know or look up. But in contrast with Fibonacci, the exact code is unlikely to be in the training set verbatim.

Zenst 1y ago

A few things that this made me think about: 1) If only there was a cryptocurrency tied to training AI models and make crypto grinding useful than maths that solve no real-world problem external to the token creation itself. 2) Larger and larger AI models, you start to get more hallucinations, maybe we should focus on dedicated highly tuned models for dedicated aspects and have a higher up conductor model that knows what to farm out to which models and from there combine and send out further requests to other models etc to come to a result. Certainly, the need for highly tuned niche models, after all, language recognition as an example, a model that could identify the language, local dialect and accent, that would then use a language model tuned better for that speaker it is recognising. That approach feels like the way over one large model that does it all itself.

N3cr0ph4g1st 1y ago
For 1) isn't prime intellect doing that or something like it?
rcbdev 1y ago
I remember mining something called Gridcoin over a decade ago. It's a cryptocurrency tied to the BOINC project and rewards providing computing power to science. I have sadly lost access to my wallet since.
packetlost 1y ago
There are a couple: https://ambient.xyz/ https://ambient.xyz/ https://www.primeintellect.ai/ https://www.primeintellect.ai/

ThrowawayTestr 1y ago

I like to imagine a world where we embraced nuclear and renewables where energy is free and we could throw gigawatthours at whatever crazy computation we can think of.

jxmorris12 1y ago

I recently wrote a post about scaling RL that has some similar ideas: > How to Scale RL to 10^26 FLOPs (blog.jxmo.io/p/how-to-scale-rl-to-1026-flops) The basic premise behind both essays is that for AI to make another big jump in capabilities, we need to find new data to train on. My proposal was reusing text from the Internet and doing RL on next-token prediction. The linked post here instead suggests doing 'replication training', which they define as "tasking AIs with duplicating existing software products, or specific features within them".

ltbarcly3 1y ago

This makes no sense. RL training data is predicated on past behavior of the agent. Whoever wrote this doesn't seem to fundamentally grasp what they are saying. LLMs can be trained on an unsupervised way on static documents. That is really the key feature that lets them be as smart and effective as they are. If you had every other technology that LLMs are built on, and you didn't have hundreds of terabytes of text laying around, there would be no practical way to make them even a tiny tiny fraction as effective as they are currently.

sva_ 1y ago
> Whoever wrote this doesn't seem to fundamentally grasp what they are saying. RL != only online learning. There's a ton of research on offline and imitation-based RL where the training data isn't tied to an agents past policy - which is exactly what this article is pointing to.
- physix 1y ago
  I'm not sufficiently familiar with the details on ML to assess the proposition made in the article. From my understanding, RL is a tuning approach on LLMs, so the outcome is still the same kind of beast, albeit with a different parameter set. So empirically, I actually thought that the lead companies would already be strongly focused on improving coding capabilities, since this is where LLMs are very effective, and where they have huge cashflows from token consumptions. So, either the motivation isn't there, or they're already doing something like that, or they know it's not as effective as the approaches they already have. I wonder which one it is.
  - sva_ 1y ago
    > From my understanding, RL is a tuning approach on LLMs, What you're referring to is actually just one application of RL (RLHF). RL itself is much more than that
    - physix 1y ago
      Actually I didn't. Correct me if I am wrong, but my understanding is that RL is still an LLM tuning approach, i.e. an optimization of its parameter set, no matter if it's done at scale or via HF.

novia 1y ago

I wish these guys would hire me

simonebrunozzi 1y ago

Bad title. I thought RL = Real Life, instead of Reinforcement Learning. It should be clearly indicated in the title.

peteforde 1y ago
I don't know if not ever defining the acronym is a sort of passive aggressive "if you have to ask, you're not the audience we're looking to hire" filter, but I am with you. Having AI investment dollars doesn't excuse you from standard expectations that you clearly state what you're describing.

siwatanejo 1y ago

What is RL? Real Life? Or Reinforced Learning?

dboreham 1y ago
Real Life would be IRL.

octoberfranklin 1y ago

Isn't "replication training" just adversarial training, as in GANs? One network tries to clean-room implement the hash function. The other network tries to find an input for which the reference implementation behaves differently.

ankit219 1y ago

Think we will see something like this in the near future. But there are two very bold claims: one is hope that RL will lead to generalization in other domains, which based on current evidence seems far fetched. Other is the softwares and specs are good "RL-gym" data. The whole idea behind RL is that model explores the best paths, but if the softwares written are suboptimal when it comes to agent interaction paradigm (they were written for humans, not agents), there is a high chance even with the compute the model would be suboptimal. There is a parallel trend where current AI systems are abstracting entire workflows. Not accounting for that would lead to outcomes which are not cognizant of the current requirements.

randomNumber7 1y ago

The issue is that it is a stretch to call it reinforcement learning when all we do currently (in the context of LLMs) is to multiply the reward with the learning rate. It sounds cool as marketing. It helps improve LLMs a bit. And it will never yield s.th. like an AGI or anything that is "reasoning". Unless you also redefine the word reasoning of course.

lysecret 1y ago

Excellent commentary from Kaparthy on RL: https://x.com/karpathy/status/1944435412489171119 https://x.com/karpathy/status/1944435412489171119

deleted 1y ago

[deleted]