4 ms·

Teaching an AI to write Python code with Python code

91 points by dsr12 10y ago

RodgerTheGreat 10y ago
Python's context-sensitive grammar makes it a particularly unsuitable target for text-based generation of code in this manner. If you continue with this project I expect you will find you get far better results from generating ASTs or choosing a language with a simple grammar in which most possible compositions of tokens are valid.
- jasikpark 10y ago
  So... some form of lisp would probably be best, right? Or maybe just binary or assembly?
- woah 10y ago
  Read the article, these type of algorithms have an easy time with syntax. The real issue is that the code produced is meaningless.
  - kordless 10y ago
    > The real issue is that the code produced is meaningless. Sounds like something I'd code up.
  - deleted 10y ago
    [deleted]
  - dasboth 10y ago
    Exactly, the difficulty is not really in producing valid code but meaningful code. Same with text generation - while a recurrent neural network might produce grammatically correct sentences, I don't think we're close to it producing an story with an actual plot, discernible characters etc.
    - deleted 10y ago
      [deleted]
    - iandanforth 10y ago
      What about a statically analyzable language? That could be factored into the loss function.
      - king_phil 10y ago
        I was thinking the same. Not only that, if you would add multiple networks that, for example, weigh the possibility that a full line is "good" code, and another that evaluates a whole block (like google deep mind did with AlphaGo) so it looks like dense and complex code this could be a nice program trolling with pull requests on github :-D
sevenless 10y ago
Look, I was told that I wouldn't be automated out of my job for at least the next five years. Can't they start with Java?
- skykooler 10y ago
  Java is so verbose, any character-based network will probably get sidetracked before finishing a function declaration...
  - harryf 10y ago
    ...thereby becoming indistinguishable from the average Java developer
    - n0us 10y ago
      You can never fire the AI because without it the whole place would burn down and nobody else understands how the code works.
ElHacker 10y ago
This is a cool project. I did something similar with LSTM in Tensorflow to generate Reddit comments. The biggest problem I found in my experiment is that the comments generated had not real meaning, nor even tried to deliver a message. Real Reddit users identified this right away, resulting in my Bot being banned. I think the author faced the same issues here. Does anyone has any suggestions on papers that try to solve that problem?
- transcranial 10y ago
  The problem is that the latent space of a model trained on all comments is still massive. For any sort of coherence, you need to condition the probabilities on smaller spaces, or enforce structural bounds. Here's an example of a paper where they simultaneously optimize high-level semantics with token-level sequence probabilities: http://arxiv.org/pdf/1606.00776v2.pdf http://arxiv.org/pdf/1606.00776v2.pdf.
- fao_ 10y ago
  I don't have any papers, but I guess you could try and scrape other short comments on the same article and basing it of those?
aangjie 10y ago
It'd be interesting to make sure the minimum data (code) sent to the neural network is all meaningful or purely functional. Perhaps as somebody pointed out taking ast and then figuring out stateless chunks might be a good idea?? I'll take a shot..
upatricck 10y ago
What should we do, If computers are going to be coding?
- sevenless 10y ago
  Unionize?
aab0 10y ago
So it's just a char-RNN? I was hoping this was using one of the seq2seq frameworks for actually writing code, like https://arxiv.org/abs/1605.06640 https://arxiv.org/abs/1605.06640
- transcranial 10y ago
  Yeah, there's no effort to direct or encode intent. It's just producing what it probabilistically wants to produce. If it happens to run...cool? Here's a interesting example from Google DeepMind learning to generate code for Magic The Gathering and Hearthstone: http://arxiv.org/pdf/1603.06744v2.pdf http://arxiv.org/pdf/1603.06744v2.pdf
bbctol 10y ago
This is cute, but the title got me excited for something well beyond this. Heck, even if an AI produced spontaneously code that defined a function and later used it, I'd be ecstatic. This is just a fun piece of art.
pmiller2 10y ago
Here's a similar example that writes LaTeX code: http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/
ankurdhama 10y ago
Much like I can learn to speak Chinese by listening lots of sound tracks of people speaking Chinese and over time building a probabilistic model of what sound will come next.

91 points by dsr12 10y ago

Python's context-sensitive grammar makes it a particularly unsuitable target for text-based generation of code in this manner. If you continue with this project I expect you will find you get far better results from generating ASTs or choosing a language with a simple grammar in which most possible compositions of tokens are valid.

jasikpark 10y ago
So... some form of lisp would probably be best, right? Or maybe just binary or assembly?
woah 10y ago
Read the article, these type of algorithms have an easy time with syntax. The real issue is that the code produced is meaningless.
- kordless 10y ago
  > The real issue is that the code produced is meaningless. Sounds like something I'd code up.
- deleted 10y ago
  [deleted]
- dasboth 10y ago
  Exactly, the difficulty is not really in producing valid code but meaningful code. Same with text generation - while a recurrent neural network might produce grammatically correct sentences, I don't think we're close to it producing an story with an actual plot, discernible characters etc.
  - deleted 10y ago
    [deleted]
  - iandanforth 10y ago
    What about a statically analyzable language? That could be factored into the loss function.
    - king_phil 10y ago
      I was thinking the same. Not only that, if you would add multiple networks that, for example, weigh the possibility that a full line is "good" code, and another that evaluates a whole block (like google deep mind did with AlphaGo) so it looks like dense and complex code this could be a nice program trolling with pull requests on github :-D

sevenless 10y ago

Look, I was told that I wouldn't be automated out of my job for at least the next five years. Can't they start with Java?

skykooler 10y ago
Java is so verbose, any character-based network will probably get sidetracked before finishing a function declaration...
- harryf 10y ago
  ...thereby becoming indistinguishable from the average Java developer
  - n0us 10y ago
    You can never fire the AI because without it the whole place would burn down and nobody else understands how the code works.

ElHacker 10y ago

This is a cool project. I did something similar with LSTM in Tensorflow to generate Reddit comments. The biggest problem I found in my experiment is that the comments generated had not real meaning, nor even tried to deliver a message. Real Reddit users identified this right away, resulting in my Bot being banned. I think the author faced the same issues here. Does anyone has any suggestions on papers that try to solve that problem?

transcranial 10y ago
The problem is that the latent space of a model trained on all comments is still massive. For any sort of coherence, you need to condition the probabilities on smaller spaces, or enforce structural bounds. Here's an example of a paper where they simultaneously optimize high-level semantics with token-level sequence probabilities: http://arxiv.org/pdf/1606.00776v2.pdf http://arxiv.org/pdf/1606.00776v2.pdf.
fao_ 10y ago
I don't have any papers, but I guess you could try and scrape other short comments on the same article and basing it of those?

aangjie 10y ago

It'd be interesting to make sure the minimum data (code) sent to the neural network is all meaningful or purely functional. Perhaps as somebody pointed out taking ast and then figuring out stateless chunks might be a good idea?? I'll take a shot..

upatricck 10y ago

What should we do, If computers are going to be coding?

sevenless 10y ago
Unionize?

aab0 10y ago

So it's just a char-RNN? I was hoping this was using one of the seq2seq frameworks for actually writing code, like https://arxiv.org/abs/1605.06640 https://arxiv.org/abs/1605.06640

transcranial 10y ago
Yeah, there's no effort to direct or encode intent. It's just producing what it probabilistically wants to produce. If it happens to run...cool? Here's a interesting example from Google DeepMind learning to generate code for Magic The Gathering and Hearthstone: http://arxiv.org/pdf/1603.06744v2.pdf http://arxiv.org/pdf/1603.06744v2.pdf

bbctol 10y ago

This is cute, but the title got me excited for something well beyond this. Heck, even if an AI produced spontaneously code that defined a function and later used it, I'd be ecstatic. This is just a fun piece of art.

pmiller2 10y ago

Here's a similar example that writes LaTeX code: http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/

ankurdhama 10y ago

Much like I can learn to speak Chinese by listening lots of sound tracks of people speaking Chinese and over time building a probabilistic model of what sound will come next.