5 ms·

AI Coding with CodeRL: Toward Mastering Program Synthesis with Deep RL

66 points by moyix 4y ago

moyix 4y ago
Reposting my commentary: The TL;DR is that this appears to beat both OpenAI Codex (12 billion params) and DeepMind's AlphaCode (1 billion params), despite having only 770M parameters. There is also a paper: https://arxiv.org/abs/2207.01780 https://arxiv.org/abs/2207.01780 And the code is available: https://github.com/salesforce/CodeRL https://github.com/salesforce/CodeRL So this seems to buck the trend of needing enormous models, by being smarter about taking into account the output from running the AI-generated program on test cases.
- algo_trader 4y ago
  RL is an "obvious" extension to code generating LLMs Try to compile, try to run, estimate reward, iterate. Possibly there is a cost issue, because sampling LLMs is more expensive than running starcraft?!
- algo_trader 4y ago
  EDIT: ha! turns out I am following you on twitter https://twitter.com/moyix/status/1551932519796408321 https://twitter.com/moyix/status/1551932519796408321 Please release this as a docker or something.. Ask the AI to do it for you
  - moyix 4y ago
    I'm working on integrating it with a VSCode plugin right now :) After that I'll write up a post and try to release it as something easy to use!
- a-dub 4y ago
  > So this seems to buck the trend of needing enormous models, by being smarter about taking into account the output from running the AI-generated program on test cases. i'd be curious if it finds degenerate solutions and what they look like when the test cases aren't conceptually complete...
- PeterisP 4y ago
  > the trend of needing enormous models I would rather say that there isn't a trend of needing enormous models but rather a trend of observing that enormous models (and straightforward increases in size) work. We've already seen many other tricks applied for transformer models (essentially, most of "BERTology") that demonstrated that being smarter about taking into account some stuff in training can get a big benefit, but that is all orthogonal to the size issue. That trend would be bucked only if it turned out that for this task 770M is sufficient, i.e. that scaling this 'being smarter' approach from 770M parameters to the 12B parameters of OpenAI Codex does not get a further improvement - however, I am quite convinced that it would be noticeably better than this; and my expectations would be that the eventual outcome of this technique would be that OpenAI Codex v2 (or however that would be called) would beat this by having a model that includes this technique and some other improvements to training process and increases the size beyond 12B parameters.
lotw_dot_site 4y ago
"AI-powered code editors"? Hmmm. The first I heard of this idea was about a week ago watching Peter Zeihan dismantle the techo-utopians at https://ark-invest.com/ https://ark-invest.com/. It seems like one of those future things (like fusion and flying cars) that are always going to remain in the future. I'll stick with boring old vim!
- qualudeheart 4y ago
  Where does he do that? Do you have a link? I’m interested.
  - losteric 4y ago
    Possibly referencing https://www.youtube.com/watch?v=t_gw4eTr-hM https://www.youtube.com/watch?v=t_gw4eTr-hM (discussion: https://www.reddit.com/r/ArkInvestorsClub/comments/uv278a/what_do_you_think_about_peter_zeihans_rebuttal_to/ https://www.reddit.com/r/ArkInvestorsClub/comments/uv278a/wh...)
    - qualudeheart 4y ago
      Thank you. I watched the video. Zeihan’s arguments I mostly can’t comment on, but it’s intriguing how he thinks the supply gap of programmers will be fixed by gen-z entering the software industry. I’m expecting AI to fill the gap. Yes, LLMs currently kind of suck at program synthesis. But the tech will get better. AGI may still be far off, Metaculus’ 2030 centered predictions may be wrong. Scalable, human level, program synthesis may be far off. But can it improve enough to reduce the demand for programming talent by, all else being equal, 20%? 25%? That seems doable within the decade.
      - ausbah 4y ago
        I think it's more likely platforms services and low/no code offerings will fill the gap
- ShamelessC 4y ago
YeGoblynQueenne 4y ago
When I see a title with "toward[s]" in it, the first thing I wonder about is: why "toward[s]"? Why aren't we there yet? Scrolling down to "Performance Results" it's obvious that the results of this approach are creeping upwards at a snail's pace. 40% correct programs when the model is given 1000 "guesses" (one thousand) is the best result out of all reported results. It's the result for the simplest, "intro"-level, problems. Fromt here, results get progressively worse for harder categories, like harder problem sets or "1@k" measurements, where the first program generated by a model is evaluated. For "intro" problems that's about 7%. NB: that's accuracy. Not error. 7% accuracy. 7% of the time the model got the coding problem right first try. Despite the slow creep upwards, this is really bad for this kind of approach. The main problem of Large Language Models as code generators is that they can generate code, but they don't know if it satisfies a specification (in this case, given by a combination of natural language text and input/output examples). The proposed approach (using reinforcement learning to capture, and use, the signal from unit tests) is supposed to address that precise limitation, but the results show that it just doesn't do very well at all. Not even when the system is allowed 1000 (one thousand) "guesses". Why not? Because if you start with a code generator that generates almost (not quite) random code, you end up with almost (not always) incorrect programs. After that it doesn't matter how well you "filter" for correct results, you're unlikely to have any in your generated set. The solution? Generate fewer programs that are more likely to be correct. How to do that with a LLM? Who knows. And a small correction to the article: >> Recent advances in deep learning, such as pretrained language models (LMs), have led to remarkable progress in program synthesis. These "advances" in deep learning have led to remarkable progress in neural program synthesis. The approach proposed here, combining a generator with a tester, is certainly not a "remarkable advance" for the broader program synthesis field. Rather it is a primitive approach that has been tried and refined continuously since the 1970's or so. The program synthesis field has gone a lot further than what's described in the article. I'll point to Gulwani et al. again for a recent overview of the field: Program Synthesis, Sumit Gulwani, Oleksandr Polozov, Rishabh Singh https://www.semanticscholar.org/paper/Program-Synthesis-Gulwani-Polozov/f0a79cb16c3e53e9c536b65ec7b2157db1d352f1 https://www.semanticscholar.org/paper/Program-Synthesis-Gulw... Other details noted in the article about "program synthesis" (e.g. the poor performance on complex problems) also only applies to neural program synthesis. For example, the following paper includes an experiment learning a program that identifies a palindrome (this is the hard part in the length-of-a-not-palindrome example in the article) together with other similarly complex programs (i.e. ones that are best expressed recursively): Learning Higher-Order Programs without Meta-Interpretive Learning, Stanisław J. Purgał, David M. Cerna, Cezary Kaliszyk https://arxiv.org/abs/2112.14603v1 https://arxiv.org/abs/2112.14603v1

66 points by moyix 4y ago

Reposting my commentary: The TL;DR is that this appears to beat both OpenAI Codex (12 billion params) and DeepMind's AlphaCode (1 billion params), despite having only 770M parameters. There is also a paper: https://arxiv.org/abs/2207.01780 https://arxiv.org/abs/2207.01780 And the code is available: https://github.com/salesforce/CodeRL https://github.com/salesforce/CodeRL So this seems to buck the trend of needing enormous models, by being smarter about taking into account the output from running the AI-generated program on test cases.

algo_trader 4y ago
RL is an "obvious" extension to code generating LLMs Try to compile, try to run, estimate reward, iterate. Possibly there is a cost issue, because sampling LLMs is more expensive than running starcraft?!
algo_trader 4y ago
EDIT: ha! turns out I am following you on twitter https://twitter.com/moyix/status/1551932519796408321 https://twitter.com/moyix/status/1551932519796408321 Please release this as a docker or something.. Ask the AI to do it for you
- moyix 4y ago
  I'm working on integrating it with a VSCode plugin right now :) After that I'll write up a post and try to release it as something easy to use!
a-dub 4y ago
> So this seems to buck the trend of needing enormous models, by being smarter about taking into account the output from running the AI-generated program on test cases. i'd be curious if it finds degenerate solutions and what they look like when the test cases aren't conceptually complete...
PeterisP 4y ago
> the trend of needing enormous models I would rather say that there isn't a trend of needing enormous models but rather a trend of observing that enormous models (and straightforward increases in size) work. We've already seen many other tricks applied for transformer models (essentially, most of "BERTology") that demonstrated that being smarter about taking into account some stuff in training can get a big benefit, but that is all orthogonal to the size issue. That trend would be bucked only if it turned out that for this task 770M is sufficient, i.e. that scaling this 'being smarter' approach from 770M parameters to the 12B parameters of OpenAI Codex does not get a further improvement - however, I am quite convinced that it would be noticeably better than this; and my expectations would be that the eventual outcome of this technique would be that OpenAI Codex v2 (or however that would be called) would beat this by having a model that includes this technique and some other improvements to training process and increases the size beyond 12B parameters.

lotw_dot_site 4y ago

"AI-powered code editors"? Hmmm. The first I heard of this idea was about a week ago watching Peter Zeihan dismantle the techo-utopians at https://ark-invest.com/ https://ark-invest.com/. It seems like one of those future things (like fusion and flying cars) that are always going to remain in the future. I'll stick with boring old vim!

qualudeheart 4y ago
Where does he do that? Do you have a link? I’m interested.
- losteric 4y ago
  Possibly referencing https://www.youtube.com/watch?v=t_gw4eTr-hM https://www.youtube.com/watch?v=t_gw4eTr-hM (discussion: https://www.reddit.com/r/ArkInvestorsClub/comments/uv278a/what_do_you_think_about_peter_zeihans_rebuttal_to/ https://www.reddit.com/r/ArkInvestorsClub/comments/uv278a/wh...)
  - qualudeheart 4y ago
    Thank you. I watched the video. Zeihan’s arguments I mostly can’t comment on, but it’s intriguing how he thinks the supply gap of programmers will be fixed by gen-z entering the software industry. I’m expecting AI to fill the gap. Yes, LLMs currently kind of suck at program synthesis. But the tech will get better. AGI may still be far off, Metaculus’ 2030 centered predictions may be wrong. Scalable, human level, program synthesis may be far off. But can it improve enough to reduce the demand for programming talent by, all else being equal, 20%? 25%? That seems doable within the decade.
    - ausbah 4y ago
      I think it's more likely platforms services and low/no code offerings will fill the gap
ShamelessC 4y ago

YeGoblynQueenne 4y ago

When I see a title with "toward[s]" in it, the first thing I wonder about is: why "toward[s]"? Why aren't we there yet? Scrolling down to "Performance Results" it's obvious that the results of this approach are creeping upwards at a snail's pace. 40% correct programs when the model is given 1000 "guesses" (one thousand) is the best result out of all reported results. It's the result for the simplest, "intro"-level, problems. Fromt here, results get progressively worse for harder categories, like harder problem sets or "1@k" measurements, where the first program generated by a model is evaluated. For "intro" problems that's about 7%. NB: that's accuracy. Not error. 7% accuracy. 7% of the time the model got the coding problem right first try. Despite the slow creep upwards, this is really bad for this kind of approach. The main problem of Large Language Models as code generators is that they can generate code, but they don't know if it satisfies a specification (in this case, given by a combination of natural language text and input/output examples). The proposed approach (using reinforcement learning to capture, and use, the signal from unit tests) is supposed to address that precise limitation, but the results show that it just doesn't do very well at all. Not even when the system is allowed 1000 (one thousand) "guesses". Why not? Because if you start with a code generator that generates almost (not quite) random code, you end up with almost (not always) incorrect programs. After that it doesn't matter how well you "filter" for correct results, you're unlikely to have any in your generated set. The solution? Generate fewer programs that are more likely to be correct. How to do that with a LLM? Who knows. And a small correction to the article: >> Recent advances in deep learning, such as pretrained language models (LMs), have led to remarkable progress in program synthesis. These "advances" in deep learning have led to remarkable progress in neural program synthesis. The approach proposed here, combining a generator with a tester, is certainly not a "remarkable advance" for the broader program synthesis field. Rather it is a primitive approach that has been tried and refined continuously since the 1970's or so. The program synthesis field has gone a lot further than what's described in the article. I'll point to Gulwani et al. again for a recent overview of the field: Program Synthesis, Sumit Gulwani, Oleksandr Polozov, Rishabh Singh https://www.semanticscholar.org/paper/Program-Synthesis-Gulwani-Polozov/f0a79cb16c3e53e9c536b65ec7b2157db1d352f1 https://www.semanticscholar.org/paper/Program-Synthesis-Gulw... Other details noted in the article about "program synthesis" (e.g. the poor performance on complex problems) also only applies to neural program synthesis. For example, the following paper includes an experiment learning a program that identifies a palindrome (this is the hard part in the length-of-a-not-palindrome example in the article) together with other similarly complex programs (i.e. ones that are best expressed recursively): Learning Higher-Order Programs without Meta-Interpretive Learning, Stanisław J. Purgał, David M. Cerna, Cezary Kaliszyk https://arxiv.org/abs/2112.14603v1 https://arxiv.org/abs/2112.14603v1