5 ms·

This is technology that simply didn't exist in any form 2 years ago. For no amount of money could you buy a program that did what this one does. Having been wat

by ctoth 5y ago

This is technology that simply didn't exist in any form 2 years ago. For no amount of money could you buy a program that did what this one does. Having been watching the growth of Transformer-based models for a couple years now really has hammered home that just as soon as we figure out how an AI can do X, X is no longer AI, or at least no longer impressive. How this happens is with comments like yours, and I'd really like to push back against it for once. Also 5 years? So assuming that we have all of the future ahead of us, to think that we only have 5 years left of being the top in programming competitions seems like it's somehow important and shouldn't be dismissed with "I don't think it's quite as impressive as you make it out to be."

BobbyJo 5y ago
I don't think that's what happening. Let's talk about this case: programming. It's not that people are saying "an AI programming" isn't impressive or isn't AI, it's that when people say "an AI programming" they aren't talking about ridiculously controlled environments like in this case. It's like self-driving cars. A car driving itself for the first time in a controlled environment, I'm sure, was an impressive feat, and it wouldn't be inaccurate to call it a self-driving car. However, that's not what we're all waiting for when we talk about the arrival of self-driving cars.
- ctoth 5y ago
  And if AI programming were limited to completely artificial contexts you would have a point, though I'd still be concerned. We live in a world, however, where programmers routinely call on the powers of an AI to complete their real code and get real value out of it. This is based on the same technology that brought us this particular win, so clearly this technology is useful outside "ridiculously controlled environments."
  - Retric 5y ago
    Programmers do setup completely artificial contexts so AI can work. None of the self driving systems where setup by giving the AI access to sensors, a car, and the drivers handbook and saying well you figure it out from there. The general trend is solve this greatly simplified problem, this more complex one, up to dealing with the real world.
    - ctoth 5y ago
      By AI programming I mean the AI doing programming, not programming the AI. Though soon enough the first will be doing the second and that's where the loop really closes...
  - BobbyJo 5y ago
    That's not significantly different than how programming has worked for the last 40 years though. We slowly push certain types of decisions and tasks down into the tools we use, and what's left over is what we call 'programming'. It's cool, no doubt, but as long as companies need to hire 'prorammers', then it's not the huge thing we're all looking out over the horizon waiting for.
YeGoblynQueenne 5y ago
>> This is technology that simply didn't exist in any form 2 years ago. A few examples of neural program synthesis from at least 2 years ago: https://sunblaze-ucb.github.io/program-synthesis/index.html https://sunblaze-ucb.github.io/program-synthesis/index.html Another example from June 2020: DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning https://arxiv.org/abs/2006.08381 https://arxiv.org/abs/2006.08381 RobustFill, from 2017: RobustFill: Neural Program Learning under Noisy I/O https://www.microsoft.com/en-us/research/wp-content/uploads/2017/03/robustfill.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... I could go on. And those are only examples from neural program synthesis. Program synthesis, in general, is a field that goes way back. I'd suggest as usual not making big proclamations about its state of the art without being acquainted with the literature. Because if you don't know what others have done every announcement by DeepMind, OpenAI et al seems like a huge advance... when it really isn't.
- ctoth 5y ago
  Of course program synthesis has been a thing for years, I remember some excellent papers out of MSR 10 years ago. But which of those could read a prompt and build the program from the prompt? Setting up a whole bunch of constraints and having your optimizer spit out a program that fulfills them is program synthesis and is super interesting, but not at all what I think of when I'm told we can make the computer program for us. For instance, RobustFill takes its optimization criteria from a bundle of pre-completed inputs and outputs of how people want the program to behave instead of having the problem described in natural language and creating the solution program.
  - YeGoblynQueenne 5y ago
    Program synthesis from natural language specifications has existed for many years, also. It's not my specialty (neither am I particularly interested in it), but here's a paper I found from 2017, with a quick search: https://www.semanticscholar.org/paper/Program-Synthesis-from-Natural-Language-Using-Lin/8ec6abfdc5009b4e490e975991c871dfeec05434 https://www.semanticscholar.org/paper/Program-Synthesis-from... AlphaCode is not particularly good at it, either. In the arxiv preprint, besides the subjetive and pretty meaningless "evaluation" against human coders it's also tested on a formal program synthesis benchmark, the APPS dataset. The best performing AlphaCode variant reported in the arxiv preprint solves 25% of the "introductory" APPS tasks (the least challenging ones). All AlphaCode variants tested solve less than 10% of the "interview" and "competition" (intermediary and advanced) tasks. These more objective results are not reported in the article above, I think for obvious reasons (because they are extremely poor). So it's not doing anything radically new and it's not doing it particularlly well either. Please be better informed before propagating hype. Edit: really, from a technical point of view, AlphaCode is a brute-force, generate-and-test approach to program synthesis that was state-of-the-art 40 years ago. It's just a big generator that spams programs hoping it will hit a good one. I have no idea who came up with this. Oriol Vinyals is the last author and I've seen enough of that guy's work to know he knows better than bet on such a primitive, even backwards approach. I'm really shocked that this is DeepMind work.

by ctoth 5y ago

BobbyJo 5y ago

I don't think that's what happening. Let's talk about this case: programming. It's not that people are saying "an AI programming" isn't impressive or isn't AI, it's that when people say "an AI programming" they aren't talking about ridiculously controlled environments like in this case. It's like self-driving cars. A car driving itself for the first time in a controlled environment, I'm sure, was an impressive feat, and it wouldn't be inaccurate to call it a self-driving car. However, that's not what we're all waiting for when we talk about the arrival of self-driving cars.

ctoth 5y ago
And if AI programming were limited to completely artificial contexts you would have a point, though I'd still be concerned. We live in a world, however, where programmers routinely call on the powers of an AI to complete their real code and get real value out of it. This is based on the same technology that brought us this particular win, so clearly this technology is useful outside "ridiculously controlled environments."
- Retric 5y ago
  Programmers do setup completely artificial contexts so AI can work. None of the self driving systems where setup by giving the AI access to sensors, a car, and the drivers handbook and saying well you figure it out from there. The general trend is solve this greatly simplified problem, this more complex one, up to dealing with the real world.
  - ctoth 5y ago
    By AI programming I mean the AI doing programming, not programming the AI. Though soon enough the first will be doing the second and that's where the loop really closes...
- BobbyJo 5y ago
  That's not significantly different than how programming has worked for the last 40 years though. We slowly push certain types of decisions and tasks down into the tools we use, and what's left over is what we call 'programming'. It's cool, no doubt, but as long as companies need to hire 'prorammers', then it's not the huge thing we're all looking out over the horizon waiting for.

YeGoblynQueenne 5y ago

>> This is technology that simply didn't exist in any form 2 years ago. A few examples of neural program synthesis from at least 2 years ago: https://sunblaze-ucb.github.io/program-synthesis/index.html https://sunblaze-ucb.github.io/program-synthesis/index.html Another example from June 2020: DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning https://arxiv.org/abs/2006.08381 https://arxiv.org/abs/2006.08381 RobustFill, from 2017: RobustFill: Neural Program Learning under Noisy I/O https://www.microsoft.com/en-us/research/wp-content/uploads/2017/03/robustfill.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... I could go on. And those are only examples from neural program synthesis. Program synthesis, in general, is a field that goes way back. I'd suggest as usual not making big proclamations about its state of the art without being acquainted with the literature. Because if you don't know what others have done every announcement by DeepMind, OpenAI et al seems like a huge advance... when it really isn't.

ctoth 5y ago
Of course program synthesis has been a thing for years, I remember some excellent papers out of MSR 10 years ago. But which of those could read a prompt and build the program from the prompt? Setting up a whole bunch of constraints and having your optimizer spit out a program that fulfills them is program synthesis and is super interesting, but not at all what I think of when I'm told we can make the computer program for us. For instance, RobustFill takes its optimization criteria from a bundle of pre-completed inputs and outputs of how people want the program to behave instead of having the problem described in natural language and creating the solution program.
- YeGoblynQueenne 5y ago
  Program synthesis from natural language specifications has existed for many years, also. It's not my specialty (neither am I particularly interested in it), but here's a paper I found from 2017, with a quick search: https://www.semanticscholar.org/paper/Program-Synthesis-from-Natural-Language-Using-Lin/8ec6abfdc5009b4e490e975991c871dfeec05434 https://www.semanticscholar.org/paper/Program-Synthesis-from... AlphaCode is not particularly good at it, either. In the arxiv preprint, besides the subjetive and pretty meaningless "evaluation" against human coders it's also tested on a formal program synthesis benchmark, the APPS dataset. The best performing AlphaCode variant reported in the arxiv preprint solves 25% of the "introductory" APPS tasks (the least challenging ones). All AlphaCode variants tested solve less than 10% of the "interview" and "competition" (intermediary and advanced) tasks. These more objective results are not reported in the article above, I think for obvious reasons (because they are extremely poor). So it's not doing anything radically new and it's not doing it particularlly well either. Please be better informed before propagating hype. Edit: really, from a technical point of view, AlphaCode is a brute-force, generate-and-test approach to program synthesis that was state-of-the-art 40 years ago. It's just a big generator that spams programs hoping it will hit a good one. I have no idea who came up with this. Oriol Vinyals is the last author and I've seen enough of that guy's work to know he knows better than bet on such a primitive, even backwards approach. I'm really shocked that this is DeepMind work.