8 ms·

This is extremely impressive, but I do think it’s worth noting that these two things were provided: - a very well defined problem. (One of the things I like a

This is extremely impressive, but I do think it’s worth noting that these two things were provided: - a very well defined problem. (One of the things I like about competitive programming and the like is just getting to implement a clearly articulated problem, not something I experience on most days.) - existing test data. This is definitely a great accomplishment, but I think those two features of competitive programming are notably different than my experience of daily programming. I don’t mean to suggest these will always be limitations of this kind of technology, though.

jakub_g 5y ago
100% agree. Someone (who?) had to take time and write the detailed requirements. In real jobs you rarely get good tickets with well defined expectations; it's one of most important developer's jobs to transform fuzzy requirement into a good ticket. (Side note: I find that many people skip this step, and go straight from fuzzy-requirement-only-discussed-on-zoom-with-Bob to code; open a pull request without much context or comments; and then a code reviewer is supposed to review it properly without really knowing what problem is actually being solved, and whether the code is solving a proper problem at all).
- ohwellhere 5y ago
  Is the next step in the evolution of programming having the programmer become the specifier? Fuzzy business requirements -> programmer specifies and writes tests -> AI codes
  - buscoquadnary 5y ago
    That's all we've ever been since we invented software. First we specified the exact flow of the bits with punch cards. Then we got assembly and we specified the machine instructions. Then we got higher level languages and we specified how the memory was to be managed and what data to store where. Now we have object oriented languages that allow us to work with domain models, and functional languages that allow us to work data structures and algorithms. The next level may be writing business rules, and specifying how services talk to each other, who knows, but it will be no different than it is now just a higher level.
  - chinabot 5y ago
    If its anything like my job while(1) { Fuzzy business requirements -> programmer specifies and writes tests -> AI codes }
- machiaweliczny 5y ago
  But it's easy to create AI conversation that will refine problem.
- ctoth 5y ago
  So what happens when OpenAI releases TicketFixer 0.8 which synthesizes everything from transcripts of your meetings to the comments to the JIRA ticket to the existing codebase and spits out better tickets to feed into the programming side?
  - jakub_g 5y ago
    Take my money.
  - solarmist 5y ago
    Yup, I hope that'll happen. Then engineers would just end up being done at a higher level of abstraction closer to what designers do with wireframes and mockups. Kind of the opposite of the way graphic design has evolved. Instead of getting more involved in the process and, in many cases, becoming front-end developers, it'll become more abstract where humans make the decisions and reason about what to include/exclude, how it'll flow, etc. Even TicketFixer wouldn't be able to do more than offer a handful of possible solutions to design-type issues.
    - bmhin 5y ago
      Yeah, we need our TicketFixer to also include the No_Bob 0.2 plugin that figures out that a decent percentage of the time whatever "Bob" is asking for in that meeting is not what "Bob" thinks he is asking for or should be asking for and can squash those tickets. Without that we're gonna somehow end up with spreadsheets in everything.
      - solarmist 5y ago
        Haha, yeah, there's that, but there are also things like "adding a dark mode." There are a dozen ways to accomplish that kind of thing, and every company's solution will diverge when you get down to the details.
- jensensbutton 5y ago
  Maybe the problem transformation will be both the beginning _and_ end of the developer's role.
baobabKoodaa 5y ago
> One of the things I like about competitive programming and the like is just getting to implement a clearly articulated problem English versions of Codeforces problems may be well-defined but they are often very badly articulated and easy to misunderstand as a human reader. I still can't understand how they got AI to be able to generate plausible solutions from these problem statements.
- zbobet2012 5y ago
  They used the tests. The specification being very approximate is fine, because they had a prebuilt way to "check" if their result was good.
  - baobabKoodaa 5y ago
    Wait what, they cheated to get this result? Only pretests are available to competitors before submitting. If they had access to the full test suite, then they had a HUGE advantage over actual competitors, and this result is way less impressive than claimed. Can you provide a source for this claim? I don't want to read the full paper.
    - sireat 5y ago
      If AlphaCode had access to full test suite then the result is not surprising at all. You can fit anything given enough parameters. https://fermatslibrary.com/s/drawing-an-elephant-with-four-complex-parameters#email-newsletter https://fermatslibrary.com/s/drawing-an-elephant-with-four-c...
e4e78a06 5y ago
I don't think it's quite as impressive as you make it out to be. Median performance in a Codeforces programming competition is solving the easiest 1-2 problems out of 5-6 problems. Like all things programming the top 1% is much, much better than the median. There's also the open problem of verifying correctness in solutions and providing some sort of flag when the model is not confident in its correctness. I give it another 5 years in the optimistic case before AlphaCode can reliably compete at the top 1% level.
- Jensson 5y ago
  Top 1% competitive programming level means that it can start solving research problems, problem difficulty and creativity needed for problems goes up exponentially for harder problems and programming contests have lead to research papers before. It would be cool if we got there in 5 years but I doubt it. But if we got there it would revolutionize so many things in society.
- ctoth 5y ago
  This is technology that simply didn't exist in any form 2 years ago. For no amount of money could you buy a program that did what this one does. Having been watching the growth of Transformer-based models for a couple years now really has hammered home that just as soon as we figure out how an AI can do X, X is no longer AI, or at least no longer impressive. How this happens is with comments like yours, and I'd really like to push back against it for once. Also 5 years? So assuming that we have all of the future ahead of us, to think that we only have 5 years left of being the top in programming competitions seems like it's somehow important and shouldn't be dismissed with "I don't think it's quite as impressive as you make it out to be."
  - BobbyJo 5y ago
    I don't think that's what happening. Let's talk about this case: programming. It's not that people are saying "an AI programming" isn't impressive or isn't AI, it's that when people say "an AI programming" they aren't talking about ridiculously controlled environments like in this case. It's like self-driving cars. A car driving itself for the first time in a controlled environment, I'm sure, was an impressive feat, and it wouldn't be inaccurate to call it a self-driving car. However, that's not what we're all waiting for when we talk about the arrival of self-driving cars.
elb2020 5y ago
I think they will always be limitations. Software is, ultimately, always about humans. Software is always there to serve a human need. And the "intelligence" that designs software will always, at some level, need to be intelligence that understands the human mind, with all it's knowledge, needs, and intricacies. There are no shortcuts to this. So, I think AI as a replacement for software development professionals, that's currently more like a pipe dream. I think AI will give us powerful new tools, but I do not think it will replace, or even reduce, the need for software development professionals. In total it might even increase the need for software development professionals, because it adds another level to the development stack. Another level of abstraction, and another level of complexity that needs to be understood.
deleted 5y ago
[deleted]

jakub_g 5y ago

100% agree. Someone (who?) had to take time and write the detailed requirements. In real jobs you rarely get good tickets with well defined expectations; it's one of most important developer's jobs to transform fuzzy requirement into a good ticket. (Side note: I find that many people skip this step, and go straight from fuzzy-requirement-only-discussed-on-zoom-with-Bob to code; open a pull request without much context or comments; and then a code reviewer is supposed to review it properly without really knowing what problem is actually being solved, and whether the code is solving a proper problem at all).

ohwellhere 5y ago
Is the next step in the evolution of programming having the programmer become the specifier? Fuzzy business requirements -> programmer specifies and writes tests -> AI codes
- buscoquadnary 5y ago
  That's all we've ever been since we invented software. First we specified the exact flow of the bits with punch cards. Then we got assembly and we specified the machine instructions. Then we got higher level languages and we specified how the memory was to be managed and what data to store where. Now we have object oriented languages that allow us to work with domain models, and functional languages that allow us to work data structures and algorithms. The next level may be writing business rules, and specifying how services talk to each other, who knows, but it will be no different than it is now just a higher level.
- chinabot 5y ago
  If its anything like my job while(1) { Fuzzy business requirements -> programmer specifies and writes tests -> AI codes }
machiaweliczny 5y ago
But it's easy to create AI conversation that will refine problem.
ctoth 5y ago
So what happens when OpenAI releases TicketFixer 0.8 which synthesizes everything from transcripts of your meetings to the comments to the JIRA ticket to the existing codebase and spits out better tickets to feed into the programming side?
- jakub_g 5y ago
  Take my money.
- solarmist 5y ago
  Yup, I hope that'll happen. Then engineers would just end up being done at a higher level of abstraction closer to what designers do with wireframes and mockups. Kind of the opposite of the way graphic design has evolved. Instead of getting more involved in the process and, in many cases, becoming front-end developers, it'll become more abstract where humans make the decisions and reason about what to include/exclude, how it'll flow, etc. Even TicketFixer wouldn't be able to do more than offer a handful of possible solutions to design-type issues.
  - bmhin 5y ago
    Yeah, we need our TicketFixer to also include the No_Bob 0.2 plugin that figures out that a decent percentage of the time whatever "Bob" is asking for in that meeting is not what "Bob" thinks he is asking for or should be asking for and can squash those tickets. Without that we're gonna somehow end up with spreadsheets in everything.
    - solarmist 5y ago
      Haha, yeah, there's that, but there are also things like "adding a dark mode." There are a dozen ways to accomplish that kind of thing, and every company's solution will diverge when you get down to the details.
jensensbutton 5y ago
Maybe the problem transformation will be both the beginning _and_ end of the developer's role.

baobabKoodaa 5y ago

> One of the things I like about competitive programming and the like is just getting to implement a clearly articulated problem English versions of Codeforces problems may be well-defined but they are often very badly articulated and easy to misunderstand as a human reader. I still can't understand how they got AI to be able to generate plausible solutions from these problem statements.

zbobet2012 5y ago
They used the tests. The specification being very approximate is fine, because they had a prebuilt way to "check" if their result was good.
- baobabKoodaa 5y ago
  Wait what, they cheated to get this result? Only pretests are available to competitors before submitting. If they had access to the full test suite, then they had a HUGE advantage over actual competitors, and this result is way less impressive than claimed. Can you provide a source for this claim? I don't want to read the full paper.
  - sireat 5y ago
    If AlphaCode had access to full test suite then the result is not surprising at all. You can fit anything given enough parameters. https://fermatslibrary.com/s/drawing-an-elephant-with-four-complex-parameters#email-newsletter https://fermatslibrary.com/s/drawing-an-elephant-with-four-c...

e4e78a06 5y ago

I don't think it's quite as impressive as you make it out to be. Median performance in a Codeforces programming competition is solving the easiest 1-2 problems out of 5-6 problems. Like all things programming the top 1% is much, much better than the median. There's also the open problem of verifying correctness in solutions and providing some sort of flag when the model is not confident in its correctness. I give it another 5 years in the optimistic case before AlphaCode can reliably compete at the top 1% level.

Jensson 5y ago
Top 1% competitive programming level means that it can start solving research problems, problem difficulty and creativity needed for problems goes up exponentially for harder problems and programming contests have lead to research papers before. It would be cool if we got there in 5 years but I doubt it. But if we got there it would revolutionize so many things in society.
ctoth 5y ago
This is technology that simply didn't exist in any form 2 years ago. For no amount of money could you buy a program that did what this one does. Having been watching the growth of Transformer-based models for a couple years now really has hammered home that just as soon as we figure out how an AI can do X, X is no longer AI, or at least no longer impressive. How this happens is with comments like yours, and I'd really like to push back against it for once. Also 5 years? So assuming that we have all of the future ahead of us, to think that we only have 5 years left of being the top in programming competitions seems like it's somehow important and shouldn't be dismissed with "I don't think it's quite as impressive as you make it out to be."
- BobbyJo 5y ago
  I don't think that's what happening. Let's talk about this case: programming. It's not that people are saying "an AI programming" isn't impressive or isn't AI, it's that when people say "an AI programming" they aren't talking about ridiculously controlled environments like in this case. It's like self-driving cars. A car driving itself for the first time in a controlled environment, I'm sure, was an impressive feat, and it wouldn't be inaccurate to call it a self-driving car. However, that's not what we're all waiting for when we talk about the arrival of self-driving cars.

elb2020 5y ago

I think they will always be limitations. Software is, ultimately, always about humans. Software is always there to serve a human need. And the "intelligence" that designs software will always, at some level, need to be intelligence that understands the human mind, with all it's knowledge, needs, and intricacies. There are no shortcuts to this. So, I think AI as a replacement for software development professionals, that's currently more like a pipe dream. I think AI will give us powerful new tools, but I do not think it will replace, or even reduce, the need for software development professionals. In total it might even increase the need for software development professionals, because it adds another level to the development stack. Another level of abstraction, and another level of complexity that needs to be understood.

deleted 5y ago

[deleted]