3 ms·

Thank you for posting this. Code generation is an interest of mine and a core foundation of the technology that I build. It will take me some time to read thro

Thank you for posting this. Code generation is an interest of mine and a core foundation of the technology that I build. It will take me some time to read through this blog, but I have a question: Is there any research being done where people are using large language models to generate or transform syntax trees for programs as opposed to operating with programs as simply streams of tokens?

le-mark 4y ago
> large language models Can you clarify what this means? In this context seems like it could mean either machine learning language model or programming language grammar/semantics.
- uoaei 4y ago
  LLM is a colloquial term for the latest greatest NLP text generation / AI chatbot models.
- zomglings 4y ago
  Not sure if there is a technical definition of the term. I am using it to refer to the transformer-based neural networks that are used to build models like Open AI's Codex (which GitHub Copilot uses IIRC). I believe they are called large because they contain orders of magnitude more parameters than the models that came before them. They do not fit in memory on a single training device (GPU or TPU usually) and require serious infrastructure work to train effectively. The models are also trained on large datasets. These datasets are usually text streams. In the case of GitHub Copilot, it was the textual source code of a large percentage of the git repositories on GitHub. (Maybe supplemented with other code?) I am curious if anyone is parsing source code into syntax trees and then training large language models to operate on the resulting graph structures. Edit: On a related note - GitHub Copilot is extremely good at generating syntactically correct code. Has anyone studied which parameters in the language model are related to learning programming language syntax? Are the parameters in the model related to javascript syntax "close to" the parameters in the model which learn Go syntax? Edit 2: For GitHub Copilot, did they use the content of git diffs to learn the kinds of edits people make to existing code? Edit 3: I realize that a random HN post about some blog is probably not the best place to be asking these questions. If anyone could point me to a better place, I would appreciate it.
- evanthebouncy 4y ago
  I meant gpt3 and its cousins.
mindcrime 4y ago
Is there any research being done where people are using large language models to generate or transform syntax trees for programs I saw a mention of this on Twitter the other day. I have not read the paper, and it's anonymous, so I don't know how serious and/or credible this is. But it was apparently an actual ICLR submission, so it might be worth a look. Not sure it's exactly what you're interested in, but it seems like it might be in the same neighborhood. https://openreview.net/forum?id=SKat5ZX5RET https://openreview.net/forum?id=SKat5ZX5RET
- zomglings 4y ago
  Thanks for the link. This is interesting work, although not related to operations on program syntax trees. From what I understand, they are using reinforcement learning techniques inspired by AutoML to learn architectures for language models that are effective at code generation. They have reinforcement learning models optimizing the neural networks that generate code. Pretty cool and reminds of work that followed on the heels of AutoML to build neural networks to automatically solve arbitrary ML problems (with the problem as the input parameter).
evanthebouncy 4y ago
Here's an obtuse answer. The operations that transform syntax trees are just programs as well which... Can be represented as a steams of tokens. But this is likely helpful if it makes it easier to learn by the NN. I'm certain there are works that do this, but not off the top of my head rn

le-mark 4y ago

> large language models Can you clarify what this means? In this context seems like it could mean either machine learning language model or programming language grammar/semantics.

uoaei 4y ago
LLM is a colloquial term for the latest greatest NLP text generation / AI chatbot models.
zomglings 4y ago
Not sure if there is a technical definition of the term. I am using it to refer to the transformer-based neural networks that are used to build models like Open AI's Codex (which GitHub Copilot uses IIRC). I believe they are called large because they contain orders of magnitude more parameters than the models that came before them. They do not fit in memory on a single training device (GPU or TPU usually) and require serious infrastructure work to train effectively. The models are also trained on large datasets. These datasets are usually text streams. In the case of GitHub Copilot, it was the textual source code of a large percentage of the git repositories on GitHub. (Maybe supplemented with other code?) I am curious if anyone is parsing source code into syntax trees and then training large language models to operate on the resulting graph structures. Edit: On a related note - GitHub Copilot is extremely good at generating syntactically correct code. Has anyone studied which parameters in the language model are related to learning programming language syntax? Are the parameters in the model related to javascript syntax "close to" the parameters in the model which learn Go syntax? Edit 2: For GitHub Copilot, did they use the content of git diffs to learn the kinds of edits people make to existing code? Edit 3: I realize that a random HN post about some blog is probably not the best place to be asking these questions. If anyone could point me to a better place, I would appreciate it.
evanthebouncy 4y ago
I meant gpt3 and its cousins.

mindcrime 4y ago

Is there any research being done where people are using large language models to generate or transform syntax trees for programs I saw a mention of this on Twitter the other day. I have not read the paper, and it's anonymous, so I don't know how serious and/or credible this is. But it was apparently an actual ICLR submission, so it might be worth a look. Not sure it's exactly what you're interested in, but it seems like it might be in the same neighborhood. https://openreview.net/forum?id=SKat5ZX5RET https://openreview.net/forum?id=SKat5ZX5RET

zomglings 4y ago
Thanks for the link. This is interesting work, although not related to operations on program syntax trees. From what I understand, they are using reinforcement learning techniques inspired by AutoML to learn architectures for language models that are effective at code generation. They have reinforcement learning models optimizing the neural networks that generate code. Pretty cool and reminds of work that followed on the heels of AutoML to build neural networks to automatically solve arbitrary ML problems (with the problem as the input parameter).

evanthebouncy 4y ago

Here's an obtuse answer. The operations that transform syntax trees are just programs as well which... Can be represented as a steams of tokens. But this is likely helpful if it makes it easier to learn by the NN. I'm certain there are works that do this, but not off the top of my head rn