4 ms·

Shawn from Weights & Biases here. We've been working with Github and Microsoft Research on this for just about a year now and we're super excited to launch it t

by slewis 7y ago

Shawn from Weights & Biases here. We've been working with Github and Microsoft Research on this for just about a year now and we're super excited to launch it today. We've seen huge advances in human language modeling and translation due to the success of deep learning. Often new directions start with a really motivated team producing a new kind of dataset. Who better to do that for code as language than Github! This started as a grassroots effort inside of Github, and went through many iterations. When it was presented to Github's CEO six months ago, he correctly pointed out that we needed to go back and include Github's most popular language (javascript). As the project went on many smart people chipped in, and we produced something that we think is truly useful. Check out the paper here: https://arxiv.org/abs/1909.09436 https://arxiv.org/abs/1909.09436 We overcame plenty of challenges to pull this off. For example: how do you clean this data? how do you label it? We've got folks from Github, Microsoft Research and Weights & Biases here to answer any and all questions you might have. Can't wait to see where this goes!

codetrotter 7y ago
> We've seen huge advances in human language modeling and translation due to the success of deep learning. I wonder if we’ll eventually see a system where instead of writing code you describe in natural language what you want the program to do and then ML is applied in order to generate the code for that. I mean, a lot of people have been interested in the past in making human programming languages, and had varying degrees of success. Personally I love writing code but, it could be, couldn’t it? Write some unit tests, a human description of what it does and based on the source code and description of existing software the system would basically “debug the program into existence” for you. That’d be kind of freaky, kind of cool and a little bit scary.
- mloncode 7y ago
  Yes that would be really cool. The field of program synthesis has made strides in this area but it doesn't appear you can create anything more than trivial programs from human languages at the moment. I think that you are more likely to see technology that augments the human significantly -- for example better code completion, error detection etc. that allows you to work much faster. It will be exciting to see how machine learning shapes developer tools and workflows in the future.
- ndarwincorn 7y ago
  > Write some unit tests, a human description of what it does and based on the source code and description of existing software the system would basically “debug the program into existence” for you. Sounds more or less like the mechanism by which developer jobs succumb to automation. Hopefully those of us that are working class have seized the capital by then.
  - dspillett 7y ago
    > Sounds more or less like the mechanism by which developer jobs succumb to automation. To spin that more positively, it may automate the basic boring stuff (both for us technical types and potentially for the simian on the street) and leave us more time to spend time in more fun & challenging playgrounds.
- nbardy 7y ago
  Great idea, I would love to see this. Tools for programming are going make some interesting leaps with all the new work going into language modeling. So much of the code we write is tweaks and combinations of a not particularly large set of patterns: loops, functions, merge, reduce, sort, filter, interleave, etc... Generating large blocks that are near your target result would be really useful for saving time, especially on the more repetitive tasks like writing tests or simple CRUD API endpoints. Microsoft was showing off similar work from their own code datasets this year at ICLR, I couldn't find a link online, but the demos had block suggestions from method signatures for C#. It should be possible to get similar results with natural language queries.
  - mallamanis 7y ago
    [I'm one of the Microsoft Research people who worked on this] (thanks Nick. Here are the links) Generative Code Modeling with Graphs: https://arxiv.org/abs/1805.08490 https://arxiv.org/abs/1805.08490 Learning to Represent Programs with Graphs: https://arxiv.org/abs/1711.00740 https://arxiv.org/abs/1711.00740
- mikekchar 7y ago
  Find a human who wants a computer program written (A). Find a human who can program computers (B). Have A describe what they want and have B code it without asking questions to further clarify the issues. What do you expect the result to be like? For me, my experience tells me that it will be a total failure. The problem with programming is not the encoding of the requirements in programming language for the most part. The problem is that the specifier (A in this example) usually does not have a full grasp of what they actually want. In fact, they usually don't have any idea at all. "Give me a e-shopping system to sell comic books" is the level of detail they can understand. The closer A can come to expressing the requirements they need, the closer they are to actually being B in reality. B's real skill is not in knowing the syntax and grammar of the computer language, it's in knowing that in order to make a system that will satisfy A we need to do X, Y, and Z to the tiniest detail. When we get into trouble with our software is when we write code that is dramatically more complex than the problem we are trying to represent. This doesn't happen so much because we don't know how to program. This happens because we are slowly extending the code base over time with imperfect knowledge about what we are ultimately building at any given time. We also have to trade-off the benefit for getting something done with discovering generalities that allow us to simplify the expression of code that we already have. I don't think we will ever replace "programmers" with AI -- at least not until the AI can be trained to ask the important questions about what the system really needs to be (and for that we need turing-test passing level AI). I think it's much more likely that we will build more and better tools that help programmers visualise and plan the programming situation. I think we will have automatic code generation because we already have it: Look at "derive" in Haskell and Rust, for example. But I think that's the level of automatic code generation we're going to want for at leas the next 20 years or so. Interestingly for testing, I think we'll actually go the opposite direction: We will spend more time thinking about requirements and the computer will help us by writing tests that challenge our assumptions: I've broken this function, are you sure you got it right? Again, we already have these kinds of systems and I think that this is the most appropriate direction to invest in research.
  - neuronexmachina 7y ago
    The only way I think something like what you described in your first paragraph would work is if you had an AI system B that could present questions and prototypes back to requirement-setter A for feedback. Of course, that'd be a very difficult problem, even if you limit it to a constrained domain.
- YeGoblynQueenne 7y ago
  >> Write some unit tests, a human description of what it does and based on the source code and description of existing software the system would basically “debug the program into existence” for you. This is already possible, but not with deep learning which is probably the reason you haven't heard of it. Learning programs from specifications (not necessarily in natural language) is the subject of the field of Program Synthesis [1]. Learning programs from examples of their input and outputs (which is basicaly writing unit tests) is the subject of Inductive Programming [2]. Inductive Programming encompasses the fields of Inductive Logic Programming and Inductive Functional Programming, that target logic and functional programming languages, respectively. You won't find anything learning from examples in javascript or python directly, though- imperative languages are too sloppy for that sort of thing. ___________ [1] https://en.wikipedia.org/wiki/Program_synthesis https://en.wikipedia.org/wiki/Program_synthesis [2] https://en.wikipedia.org/wiki/Inductive_programming https://en.wikipedia.org/wiki/Inductive_programming
- arandr0x 7y ago
  It has a lot less real-world value if the AI is not capable of debugging the software in future conditions, or explain why it made it the way it made it. I would love for the "writing programs into existence" parts to be automated, but it does make further investigation costlier (since the human investigating also does have to get acquainted with the patterns in the code). I do think there would be value in machine-written code if it made code in general more alike, so that you don't have to relearn the weird tricks any new writer could have chosen to use.
zamfi 7y ago
Just curious, is there a technical reason you initially omitted JavaScript?
- mabrocks 7y ago
  [I'm one of the Microsoft Research people who worked on this] There wasn't a technical reason (unless you count lazyness as a technical reason) -- we simply had infrastructure for Python specifically lying around from past research projects, which we initially reused. After we got Nat's feedback, we redid our data processing pipeline completely to be based TreeSitter (which wasn't around when we started thinking about parsing Python), which makes it much easier to scale to the number of programming languages on GitHub.
  - Dowwie 7y ago
    In other words, the system is ready to be trained for languages not on the initial list, without requiring additional development?
    - mabrocks 7y ago
      You would need to extend the data-processing pipeline for the new language, which in the best case only requires to adapt the standard wrapper around the Tree Sitter parser. The wrapper needs to take care of language-specific details (e.g., where to find the documentation for a function, what methods should be filtered out, etc.) but can be fairly small. See https://github.com/github/CodeSearchNet/blob/master/function_parser/function_parser/parsers/ https://github.com/github/CodeSearchNet/blob/master/function... for examples.
ckdarby 7y ago
Spoken like a true Kool aid drinker

by slewis 7y ago

codetrotter 7y ago

> We've seen huge advances in human language modeling and translation due to the success of deep learning. I wonder if we’ll eventually see a system where instead of writing code you describe in natural language what you want the program to do and then ML is applied in order to generate the code for that. I mean, a lot of people have been interested in the past in making human programming languages, and had varying degrees of success. Personally I love writing code but, it could be, couldn’t it? Write some unit tests, a human description of what it does and based on the source code and description of existing software the system would basically “debug the program into existence” for you. That’d be kind of freaky, kind of cool and a little bit scary.

mloncode 7y ago
Yes that would be really cool. The field of program synthesis has made strides in this area but it doesn't appear you can create anything more than trivial programs from human languages at the moment. I think that you are more likely to see technology that augments the human significantly -- for example better code completion, error detection etc. that allows you to work much faster. It will be exciting to see how machine learning shapes developer tools and workflows in the future.
ndarwincorn 7y ago
> Write some unit tests, a human description of what it does and based on the source code and description of existing software the system would basically “debug the program into existence” for you. Sounds more or less like the mechanism by which developer jobs succumb to automation. Hopefully those of us that are working class have seized the capital by then.
- dspillett 7y ago
  > Sounds more or less like the mechanism by which developer jobs succumb to automation. To spin that more positively, it may automate the basic boring stuff (both for us technical types and potentially for the simian on the street) and leave us more time to spend time in more fun & challenging playgrounds.
nbardy 7y ago
Great idea, I would love to see this. Tools for programming are going make some interesting leaps with all the new work going into language modeling. So much of the code we write is tweaks and combinations of a not particularly large set of patterns: loops, functions, merge, reduce, sort, filter, interleave, etc... Generating large blocks that are near your target result would be really useful for saving time, especially on the more repetitive tasks like writing tests or simple CRUD API endpoints. Microsoft was showing off similar work from their own code datasets this year at ICLR, I couldn't find a link online, but the demos had block suggestions from method signatures for C#. It should be possible to get similar results with natural language queries.
- mallamanis 7y ago
  [I'm one of the Microsoft Research people who worked on this] (thanks Nick. Here are the links) Generative Code Modeling with Graphs: https://arxiv.org/abs/1805.08490 https://arxiv.org/abs/1805.08490 Learning to Represent Programs with Graphs: https://arxiv.org/abs/1711.00740 https://arxiv.org/abs/1711.00740
mikekchar 7y ago
Find a human who wants a computer program written (A). Find a human who can program computers (B). Have A describe what they want and have B code it without asking questions to further clarify the issues. What do you expect the result to be like? For me, my experience tells me that it will be a total failure. The problem with programming is not the encoding of the requirements in programming language for the most part. The problem is that the specifier (A in this example) usually does not have a full grasp of what they actually want. In fact, they usually don't have any idea at all. "Give me a e-shopping system to sell comic books" is the level of detail they can understand. The closer A can come to expressing the requirements they need, the closer they are to actually being B in reality. B's real skill is not in knowing the syntax and grammar of the computer language, it's in knowing that in order to make a system that will satisfy A we need to do X, Y, and Z to the tiniest detail. When we get into trouble with our software is when we write code that is dramatically more complex than the problem we are trying to represent. This doesn't happen so much because we don't know how to program. This happens because we are slowly extending the code base over time with imperfect knowledge about what we are ultimately building at any given time. We also have to trade-off the benefit for getting something done with discovering generalities that allow us to simplify the expression of code that we already have. I don't think we will ever replace "programmers" with AI -- at least not until the AI can be trained to ask the important questions about what the system really needs to be (and for that we need turing-test passing level AI). I think it's much more likely that we will build more and better tools that help programmers visualise and plan the programming situation. I think we will have automatic code generation because we already have it: Look at "derive" in Haskell and Rust, for example. But I think that's the level of automatic code generation we're going to want for at leas the next 20 years or so. Interestingly for testing, I think we'll actually go the opposite direction: We will spend more time thinking about requirements and the computer will help us by writing tests that challenge our assumptions: I've broken this function, are you sure you got it right? Again, we already have these kinds of systems and I think that this is the most appropriate direction to invest in research.
- neuronexmachina 7y ago
  The only way I think something like what you described in your first paragraph would work is if you had an AI system B that could present questions and prototypes back to requirement-setter A for feedback. Of course, that'd be a very difficult problem, even if you limit it to a constrained domain.
YeGoblynQueenne 7y ago
>> Write some unit tests, a human description of what it does and based on the source code and description of existing software the system would basically “debug the program into existence” for you. This is already possible, but not with deep learning which is probably the reason you haven't heard of it. Learning programs from specifications (not necessarily in natural language) is the subject of the field of Program Synthesis [1]. Learning programs from examples of their input and outputs (which is basicaly writing unit tests) is the subject of Inductive Programming [2]. Inductive Programming encompasses the fields of Inductive Logic Programming and Inductive Functional Programming, that target logic and functional programming languages, respectively. You won't find anything learning from examples in javascript or python directly, though- imperative languages are too sloppy for that sort of thing. ___________ [1] https://en.wikipedia.org/wiki/Program_synthesis https://en.wikipedia.org/wiki/Program_synthesis [2] https://en.wikipedia.org/wiki/Inductive_programming https://en.wikipedia.org/wiki/Inductive_programming
arandr0x 7y ago
It has a lot less real-world value if the AI is not capable of debugging the software in future conditions, or explain why it made it the way it made it. I would love for the "writing programs into existence" parts to be automated, but it does make further investigation costlier (since the human investigating also does have to get acquainted with the patterns in the code). I do think there would be value in machine-written code if it made code in general more alike, so that you don't have to relearn the weird tricks any new writer could have chosen to use.

zamfi 7y ago

Just curious, is there a technical reason you initially omitted JavaScript?

mabrocks 7y ago
[I'm one of the Microsoft Research people who worked on this] There wasn't a technical reason (unless you count lazyness as a technical reason) -- we simply had infrastructure for Python specifically lying around from past research projects, which we initially reused. After we got Nat's feedback, we redid our data processing pipeline completely to be based TreeSitter (which wasn't around when we started thinking about parsing Python), which makes it much easier to scale to the number of programming languages on GitHub.
- Dowwie 7y ago
  In other words, the system is ready to be trained for languages not on the initial list, without requiring additional development?
  - mabrocks 7y ago
    You would need to extend the data-processing pipeline for the new language, which in the best case only requires to adapt the standard wrapper around the Tree Sitter parser. The wrapper needs to take care of language-specific details (e.g., where to find the documentation for a function, what methods should be filtered out, etc.) but can be fairly small. See https://github.com/github/CodeSearchNet/blob/master/function_parser/function_parser/parsers/ https://github.com/github/CodeSearchNet/blob/master/function... for examples.

ckdarby 7y ago

Spoken like a true Kool aid drinker