6 ms·

This seems to have a narrower scope than GitHub Copilot. It generates more lines of code to a more holistic problem vs. GitHub Copilot that works as a "more adv

by msoad 5y ago

This seems to have a narrower scope than GitHub Copilot. It generates more lines of code to a more holistic problem vs. GitHub Copilot that works as a "more advanced autocomplete" in code editors. Sure Copilot can synthesize full functions and classes but for me, it's the most useful when it suggests another test case's title or writes repetitive code like this.foo = foo; this.bar = bar etc... Having used Copilot I can assure you that this technology won't replace you as a programmer but it will make your job easier by doing things that programmers don't like to do as much like writing tests and comments.

ipnon 5y ago
The big question seems to be whether par with professional programmers is a matter of increasing training set and flop size, or whether different model or multi-model architectures are required. It does look like we've entered an era where programmers who don't use AI assistants will be disadvantaged, and that this era has an expiration date.
sharemywin 5y ago
To me it's not about it's current capabilities. It's the trajectory. This tech wasn't even a thing 2 years ago. There's billions being poured into it and every time someone uses these tools there's more free training data.
jxcole 5y ago
I feel like you are very defensive here and I want to be sure we take time to recognize this as a real accomplishment. Seriously though, I do doubt I can be fully replaced by a robot any time soon, it may be the case that soon enough I can make high-level written descriptions of programs and hand them off to an AI to do most of the work. This wouldn't completely replace me, but it could make developers 50x productive. The question is how elastic is the market...can the market grow in step with our increase in productivitiy? Also, please remember that as with anything, within 5 years we should see vast improvements to this AI. I think it will be an important thing to watch.
- visarga 5y ago
  The GPT family has already shown more than 50x productivity increase by being able to solve not one, but hundreds and perhaps thousands of tasks on the same model. We used to need much more data, and the model would be more fragile, and finding the right architecture would be a problem. Now we plug a transformer with a handful of samples and it works. I just hope LMs will prove to be just as useful in software development as they are in their own field.
- thomasahle 5y ago
  If you make developers 50x more efficient, won't you need 50x fewer developers?
  - bmh100 5y ago
    Not necessarily. Demand may be much higher than available supply right now. Tech companies will continue to compete, requiring spending on developers to remain competitive. Software is unlike manufacturing, in that the output is a service, not a widget. Worker productivity in general has not decreased the demand for full work weeks, despite projections in the early 20th century to the contrary. Of course, it is possible that fewer developers would be needed, but I don't think it's likely, yet.
  - kevlened 5y ago
    Greater efficiency leads to greater consumption unless demand is saturated. Given software’s ability to uncover more problems that are solvable by software, we’re more likely to build 50x more software.
  - alasdair_ 5y ago
    >If you make developers 50x more efficient, won't you need 50x fewer developers? Developers today are 50X more efficient than when they had to input machine code on punched tape, yet the number of developers needed today is far larger than it was in those times.
    - throw10920 5y ago
chongli 5y ago
repetitive code like this.foo = foo; this.bar = bar etc... This sort of boilerplate code is best solved by the programming language. Either via better built-in syntax or macros. Using an advanced machine learning model to generate this code is both error-prone and a big source of noise and code bloat. This is not an issue that will go away with better tooling; it will only get worse.
- xmprt 5y ago
  I don't think I agree. Most people spend more time reading than writing code so programming languages should be optimized to be easier to read whereas tooling should be made to simplify writing code. New syntax or macros sounds like it would make the language harder to read. I agree that an advanced machine learning model for generating boilerplate code isn't the right approach but I also don't think we should extend languages for this. Tooling like code generators and linters are a good middle ground.
  - orangecat 5y ago
    New syntax or macros sounds like it would make the language harder to read. Often the opposite is true. For example Java records are far easier to read and understand than the pages of boilerplate that they replace.
    - xmprt 5y ago
      That sounds like an issue with how Java was designed. There are plenty of languages that solve Java's boilerplate problems without adding new syntax for records.
      - chongli 5y ago
        If you’ll review my original comment, I never said new syntax. I said better syntax. If your language design leads to a lot of boilerplate in idiomatic use then it needs to be better. Adding new syntax is just putting a bandaid on the problem.
  - 5y ago
stupidcar 5y ago
Having used Copilot for a while, I am quite certain it will replace me as a programmer. It appears to me that when it comes to language models, intelligence = experience * context. Where experience is the amount what's encoded in the model, and context is the prompt. And the biggest limitation on Copilot currently is context. It behaves as an "advanced autocomplete" because it all is has to go on is what regular autocomplete sees, e.g. the last few characters and lines of code. So, you can write a function name called createUserInDB() and it will attempt to complete it for you. But how does it know what DB technology you're using? Or what your user record looks like? It doesn't, and so you typically end up with a "generic" looking function using the most common DB tech and naming conventions for your language of choice. But now imagine a future version of Copilot that is automatically provided with a lot more context. It also gets fed a list of your dependencies, from which it can derive which DB library you're using. It gets any locatable SQL schema file, so it can determine the columns in the user table. It gets the text of the Jira ticket, so it can determine the requirements. As a programmer a great deal of time is spent checking these different sources and synthesising them in your head into an approach, which you then code. But they are all just text, of one form or another, and language models can work with them just as easily, and much faster, than you can. And one the ML train coding gets running, it'll only get faster. Sooner or later Github will have a "Copilot bot" that can automatically make a stab at fixing issues, which you then approve, reject, or fix. And as thousands of these issues pile up, the training set will get bigger, and the model will get better. Sooner or later it'll be possible to create a repo, start filing issues, and rely on the bot to implement everything.
- TSiege 5y ago
  I have a feeling this is the correct read in terms of progression. But I'm skeptical if it'll ever be able to synthesize a program entirely. I imagine that in the future we'll have some sort of computer language more like written language that will be used by some sort of AI to generate software to meet certain demands, but might need some manual connections when requirements are hazy or needs a more human touch in the UI/UX
  - Veedrac 5y ago

by msoad 5y ago

ipnon 5y ago

The big question seems to be whether par with professional programmers is a matter of increasing training set and flop size, or whether different model or multi-model architectures are required. It does look like we've entered an era where programmers who don't use AI assistants will be disadvantaged, and that this era has an expiration date.

sharemywin 5y ago

To me it's not about it's current capabilities. It's the trajectory. This tech wasn't even a thing 2 years ago. There's billions being poured into it and every time someone uses these tools there's more free training data.

jxcole 5y ago

I feel like you are very defensive here and I want to be sure we take time to recognize this as a real accomplishment. Seriously though, I do doubt I can be fully replaced by a robot any time soon, it may be the case that soon enough I can make high-level written descriptions of programs and hand them off to an AI to do most of the work. This wouldn't completely replace me, but it could make developers 50x productive. The question is how elastic is the market...can the market grow in step with our increase in productivitiy? Also, please remember that as with anything, within 5 years we should see vast improvements to this AI. I think it will be an important thing to watch.

visarga 5y ago
The GPT family has already shown more than 50x productivity increase by being able to solve not one, but hundreds and perhaps thousands of tasks on the same model. We used to need much more data, and the model would be more fragile, and finding the right architecture would be a problem. Now we plug a transformer with a handful of samples and it works. I just hope LMs will prove to be just as useful in software development as they are in their own field.
thomasahle 5y ago
If you make developers 50x more efficient, won't you need 50x fewer developers?
- bmh100 5y ago
  Not necessarily. Demand may be much higher than available supply right now. Tech companies will continue to compete, requiring spending on developers to remain competitive. Software is unlike manufacturing, in that the output is a service, not a widget. Worker productivity in general has not decreased the demand for full work weeks, despite projections in the early 20th century to the contrary. Of course, it is possible that fewer developers would be needed, but I don't think it's likely, yet.
- kevlened 5y ago
  Greater efficiency leads to greater consumption unless demand is saturated. Given software’s ability to uncover more problems that are solvable by software, we’re more likely to build 50x more software.
- alasdair_ 5y ago
  >If you make developers 50x more efficient, won't you need 50x fewer developers? Developers today are 50X more efficient than when they had to input machine code on punched tape, yet the number of developers needed today is far larger than it was in those times.
  - throw10920 5y ago

chongli 5y ago

repetitive code like this.foo = foo; this.bar = bar etc... This sort of boilerplate code is best solved by the programming language. Either via better built-in syntax or macros. Using an advanced machine learning model to generate this code is both error-prone and a big source of noise and code bloat. This is not an issue that will go away with better tooling; it will only get worse.

xmprt 5y ago
I don't think I agree. Most people spend more time reading than writing code so programming languages should be optimized to be easier to read whereas tooling should be made to simplify writing code. New syntax or macros sounds like it would make the language harder to read. I agree that an advanced machine learning model for generating boilerplate code isn't the right approach but I also don't think we should extend languages for this. Tooling like code generators and linters are a good middle ground.
- orangecat 5y ago
  New syntax or macros sounds like it would make the language harder to read. Often the opposite is true. For example Java records are far easier to read and understand than the pages of boilerplate that they replace.
  - xmprt 5y ago
    That sounds like an issue with how Java was designed. There are plenty of languages that solve Java's boilerplate problems without adding new syntax for records.
    - chongli 5y ago
      If you’ll review my original comment, I never said new syntax. I said better syntax. If your language design leads to a lot of boilerplate in idiomatic use then it needs to be better. Adding new syntax is just putting a bandaid on the problem.
- 5y ago

stupidcar 5y ago

Having used Copilot for a while, I am quite certain it will replace me as a programmer. It appears to me that when it comes to language models, intelligence = experience * context. Where experience is the amount what's encoded in the model, and context is the prompt. And the biggest limitation on Copilot currently is context. It behaves as an "advanced autocomplete" because it all is has to go on is what regular autocomplete sees, e.g. the last few characters and lines of code. So, you can write a function name called createUserInDB() and it will attempt to complete it for you. But how does it know what DB technology you're using? Or what your user record looks like? It doesn't, and so you typically end up with a "generic" looking function using the most common DB tech and naming conventions for your language of choice. But now imagine a future version of Copilot that is automatically provided with a lot more context. It also gets fed a list of your dependencies, from which it can derive which DB library you're using. It gets any locatable SQL schema file, so it can determine the columns in the user table. It gets the text of the Jira ticket, so it can determine the requirements. As a programmer a great deal of time is spent checking these different sources and synthesising them in your head into an approach, which you then code. But they are all just text, of one form or another, and language models can work with them just as easily, and much faster, than you can. And one the ML train coding gets running, it'll only get faster. Sooner or later Github will have a "Copilot bot" that can automatically make a stab at fixing issues, which you then approve, reject, or fix. And as thousands of these issues pile up, the training set will get bigger, and the model will get better. Sooner or later it'll be possible to create a repo, start filing issues, and rely on the bot to implement everything.

TSiege 5y ago
I have a feeling this is the correct read in terms of progression. But I'm skeptical if it'll ever be able to synthesize a program entirely. I imagine that in the future we'll have some sort of computer language more like written language that will be used by some sort of AI to generate software to meet certain demands, but might need some manual connections when requirements are hazy or needs a more human touch in the UI/UX
- Veedrac 5y ago