5 ms·

As someone who is skeptical, but open minded, about the impact these technologies will have on practical programming I think I'm one of the "people" in "people

As someone who is skeptical, but open minded, about the impact these technologies will have on practical programming I think I'm one of the "people" in "people are complaining..." The article makes some assumptions about what such people think that certainly aren't true for me: 1. That we are unimpressed. I'm gobsmacked. 2. That we don't think these are significant advances. They're obviously huge advances. 3. That we don't think these models will have practical applications. It's hard to imagine they won't. 4. That we think these systems are rubbish because they get things wrong. I'm a programmer. I make mistakes all the time. Having countered those views, the article then seems to imply that it follows that "we’ve now entered a world where 'programming' will look different." As someone who makes a living writing software I obviously have an interest in knowing whether that's true. I don't see much evidence of it yet. These systems are certainly not (yet) capable of replacing a human programmer altogether, and whether they could ever do so is unknown. I'm interested in the implications of the technologies that have been developed so far - i.e. with the claim that "we've now entered a world..." So the question is about how useful these systems can be for human programmers, as tools. The reason I'm skeptical of it is that the only model I've seen so far for such tooling is "have the machine generate code and have the human review it, select from candidate solutions and fix bugs". The problem is that doing so is, I expect, harder for the human than writing the code in the first place. I've mentioned this concern several times and not seen anybody even attempt to explain to me why I'm wrong about it. For example, at [1] I pointed out why some generated solutions for a particular problem would have only made my job harder and got accused of "screaming at a child for imperfect grammar." Reviewing and fixing code is harder than writing it. Please explain why I'm wrong about that (it's certainly true for me, but maybe most people don't feel that way?), why it won't be a problem in practice or what planned applications there are for these technologies that would avoid the problem. Please don't accuse me of cruelty to dogs or children. [1] https://news.ycombinator.com/item?id=30180067 https://news.ycombinator.com/item?id=30180067

ogogmad 5y ago
I think using these tools might become a science or an art form in its own right. You'll have to give these tools the input they need to produce the most useful answers to you. In the short term at least, this is not going to take away your need to think. But it might change how you think, and it might make you more productive when your problem aligns well with these tools.
- omnicognate 5y ago
  Potentially, but that doesn't address the problem of replacing writing code with the (harder) process of reading, verifying and fixing it. However well I engineer my inputs I'll still have to review, verify and fix the outputs.
  - derekja 5y ago
    This problem has been (albeit imperfectly) addressed in speech recognition. When error corrections are made through the the UI the engine can learn from those corrections. Presumably over time the corrections needed in alphacode will become more semantic than syntactical. But you're right, correcting subtly flawed code or text is way harder than writing from scratch.
    - falcor84 5y ago
      One of the most difficult problems in software development in general is coming up with the right test cases that cover the real-world domain's needs. If you have an AI like this at your disposal, that you can throw test cases at and it will give you the stupidest piece of code that makes them pass, then at the least you have something that helps you iterate on your test cases a lot more effectively, which would be a great boon.
- mark_l_watson 5y ago
  I think that you are correct, we will see AI used as a copilot for artists and content creators. I have had access to the OpenAI APIs for GPT-3 since last summer, and in addition to using this for NLP tasks, I have also experimented with using GPT-3 to help me work on a sci-fi short book that I have been occasionally writing for years. It has been useful for expanding sections that I have been stuck on. I feel that I should copyright my story as being authored by The Internet since every content creator whose text, programming listings, etc. that goes into training these language models has in some sense been a contributor to my short story.
  - meken 5y ago
zamalek 5y ago
Agreed, and also: > Post: I realize that it “only” solves about a third of the contest problems, making it similar to a mediocre human programmer. If this is what academia perceives software development to be then it's no wonder we have software that is so disconnected from the human problems it aims to solve. Programmers don't routinely [re-]invent complex algorithms. We parse complex and contradictory requirements from humans, and compile them into (hopefully) simple solutions on a computer. The solutions to "competition programming" problems are a Google search away. If you want to take it up as a professional sport then, sure, AI might just replace you (as it already has done with many other mind sports such as chess).
furyofantares 5y ago
> Reviewing and fixing code is harder than writing it. Please explain why I'm wrong about that (it's certainly true for me, but maybe most people don't feel that way?) At the very least, I don't think it being true for you (or me) means much. Presumably like me, you learned primarily to write code and have a lot of experience with it, and have gotten better and better at that with time. I assume that if I had spent as much time and effort learning to efficiently review and fix code as I have spent learning to write it that it would be much easier. I certainly know of artists who have shipped video games they authored alone, who wouldn't call themselves programmers or know where to start programming something from scratch, who got by with a high level engine and copy and pasting starting points from the internet. (I would call them programmers, but their style is much closer to review and edit than my own.)
kgeist 5y ago
>The problem is that doing so is, I expect, harder for the human than writing the code in the first place. Programming is mostly about writing boilerplate code using well-known architectural patterns and technologies, nothing extraordinary but which takes time (at least in my experience). If I can describe a project in a few abstract words, and the AI generates the rest, it can considerably improve my productivity, and I don't think it's going to be harder to review than code written by a junior dev who also makes mistakes and doesn't usually get it right on the first try anyway (given the AI is pretrained to know what kind of architecture we prefer). I can envision a future where programmers are basically AI operators who iterate on the requirements with the client/stakeholders and let AI do the rest. It looks like we're almost there (with GitHub Copilot and all), and I think it's enough to "revolutionize" the industry, because it changes the way we approach problems and makes us far more productive with less effort.
- rakejake 5y ago
  If a junior dev writes truly head-scratching code, you could ping that person and ask why they wrote this line a certain way, as opposed to a more straight-forward way. Correct me if I'm wrong but you can't ask an ML model to do that (yet).
  - kgeist 5y ago
    True. But is it really important "why"? I think what's more important is whether we can correct AI's output in a way that makes it learn to avoid writing similar head-scratching code in the future.
    - rakejake 5y ago
      Unless the AI can fix the bug itself, a human is going to have to explore the AI-generated code. The AI cannot help you here and you are going to have to figure out the "why" all by yourself. It will become incredibly easy to generate code with prompts but fixing any problem is going to be a pain. The "Tech Debt" would be astronomical.
  - 5y ago
rakejake 5y ago
I agree that reviewing code is much much harder than writing something that works. Hasn't anyone here written what they thought was a smart, compact solution or feature, only to have it ripped to shreds during a code review? I have worked in large decade-old codebases and sometimes the code is truly puzzling. Usually, an old-timer helps out when you run into a block and explains design decisions taken by people who moved on. This is the crucial factor that determines how long a task takes. A task could be resolved in a few hours with the help of a senior engineer, which otherwise could take days to weeks. All said, I think simply generating the code will not be enough to replace programmers. Now, if an AI can generate code AND explain queries about logic, syntax, etc, the game is over.
- gbasin 5y ago
  Explainability for an NLP model is well within reach. I predict OpenAI will have something like this in the next 1-2 years
  - altvali 5y ago
    Could you explain the reasoning behind your claim?
killerstorm 5y ago
AlphaCode is a piece of academic research. It's about demonstrating a possibility of using a particular type of a model to solve a particular type of tasks. It's not about making a practical tool. People can certainly take this approach and build tools. That might take different shapes and forms. There are many ways these models can be specialized, fine-tuned, combined with other approaches, etc. For example, somebody can try to apply them to a narrow use case, e.g. generate front-end code in React/JS and CSS from a backend API. They can fine-tune it on best example of React code, add a way to signal uncertainty so that specification can be clarified, etc. Nobody expects these models to be able to write an entire OS kernel any time soon. But a lot of types of programming are far more regular, repetitive and can be verified a lot easier.
microtherion 5y ago
> have the machine generate code and have the human review it The best characterization of such massive language models I've seen is "world class bullshitters": they are very good at producing output that superficially looks plausible, but neither know nor care whether what they're saying has any relationship to the truth. This is virtually guaranteed to make such code reviews very frustrating, and of course AlphaCode & co have no capability of explaining why they wrote a certain line. I can't see this having much of a role in a high quality code base, but I suspect the sort of management which currently would offshore coding to the lowest bidder would be quite enamored of it.
- mirceal 5y ago
  now you need an ai that reviews code. problem solved!
krajzeg 5y ago
You raise an interesting point about the difficulty of code writing versus code review, and I think the point generalizes nicely to many other areas where AI is currently rapidly progressing. For example, self-driving cars are also currently in this weird place where they're prone to making lots of errors, so a simple task for the human (driving) is replaced with a much more taxing one (keeping an eye on an AI driver and being ready to react at any moment), if we want to do it safely. Whether any of those applications will be able to progress past that weird space into "no/minimal supervision needed" remains an open question.

ogogmad 5y ago

I think using these tools might become a science or an art form in its own right. You'll have to give these tools the input they need to produce the most useful answers to you. In the short term at least, this is not going to take away your need to think. But it might change how you think, and it might make you more productive when your problem aligns well with these tools.

omnicognate 5y ago
Potentially, but that doesn't address the problem of replacing writing code with the (harder) process of reading, verifying and fixing it. However well I engineer my inputs I'll still have to review, verify and fix the outputs.
- derekja 5y ago
  This problem has been (albeit imperfectly) addressed in speech recognition. When error corrections are made through the the UI the engine can learn from those corrections. Presumably over time the corrections needed in alphacode will become more semantic than syntactical. But you're right, correcting subtly flawed code or text is way harder than writing from scratch.
  - falcor84 5y ago
    One of the most difficult problems in software development in general is coming up with the right test cases that cover the real-world domain's needs. If you have an AI like this at your disposal, that you can throw test cases at and it will give you the stupidest piece of code that makes them pass, then at the least you have something that helps you iterate on your test cases a lot more effectively, which would be a great boon.
mark_l_watson 5y ago
I think that you are correct, we will see AI used as a copilot for artists and content creators. I have had access to the OpenAI APIs for GPT-3 since last summer, and in addition to using this for NLP tasks, I have also experimented with using GPT-3 to help me work on a sci-fi short book that I have been occasionally writing for years. It has been useful for expanding sections that I have been stuck on. I feel that I should copyright my story as being authored by The Internet since every content creator whose text, programming listings, etc. that goes into training these language models has in some sense been a contributor to my short story.
- meken 5y ago

zamalek 5y ago

Agreed, and also: > Post: I realize that it “only” solves about a third of the contest problems, making it similar to a mediocre human programmer. If this is what academia perceives software development to be then it's no wonder we have software that is so disconnected from the human problems it aims to solve. Programmers don't routinely [re-]invent complex algorithms. We parse complex and contradictory requirements from humans, and compile them into (hopefully) simple solutions on a computer. The solutions to "competition programming" problems are a Google search away. If you want to take it up as a professional sport then, sure, AI might just replace you (as it already has done with many other mind sports such as chess).

furyofantares 5y ago

> Reviewing and fixing code is harder than writing it. Please explain why I'm wrong about that (it's certainly true for me, but maybe most people don't feel that way?) At the very least, I don't think it being true for you (or me) means much. Presumably like me, you learned primarily to write code and have a lot of experience with it, and have gotten better and better at that with time. I assume that if I had spent as much time and effort learning to efficiently review and fix code as I have spent learning to write it that it would be much easier. I certainly know of artists who have shipped video games they authored alone, who wouldn't call themselves programmers or know where to start programming something from scratch, who got by with a high level engine and copy and pasting starting points from the internet. (I would call them programmers, but their style is much closer to review and edit than my own.)

kgeist 5y ago

>The problem is that doing so is, I expect, harder for the human than writing the code in the first place. Programming is mostly about writing boilerplate code using well-known architectural patterns and technologies, nothing extraordinary but which takes time (at least in my experience). If I can describe a project in a few abstract words, and the AI generates the rest, it can considerably improve my productivity, and I don't think it's going to be harder to review than code written by a junior dev who also makes mistakes and doesn't usually get it right on the first try anyway (given the AI is pretrained to know what kind of architecture we prefer). I can envision a future where programmers are basically AI operators who iterate on the requirements with the client/stakeholders and let AI do the rest. It looks like we're almost there (with GitHub Copilot and all), and I think it's enough to "revolutionize" the industry, because it changes the way we approach problems and makes us far more productive with less effort.

rakejake 5y ago
If a junior dev writes truly head-scratching code, you could ping that person and ask why they wrote this line a certain way, as opposed to a more straight-forward way. Correct me if I'm wrong but you can't ask an ML model to do that (yet).
- kgeist 5y ago
  True. But is it really important "why"? I think what's more important is whether we can correct AI's output in a way that makes it learn to avoid writing similar head-scratching code in the future.
  - rakejake 5y ago
    Unless the AI can fix the bug itself, a human is going to have to explore the AI-generated code. The AI cannot help you here and you are going to have to figure out the "why" all by yourself. It will become incredibly easy to generate code with prompts but fixing any problem is going to be a pain. The "Tech Debt" would be astronomical.
- 5y ago

rakejake 5y ago

I agree that reviewing code is much much harder than writing something that works. Hasn't anyone here written what they thought was a smart, compact solution or feature, only to have it ripped to shreds during a code review? I have worked in large decade-old codebases and sometimes the code is truly puzzling. Usually, an old-timer helps out when you run into a block and explains design decisions taken by people who moved on. This is the crucial factor that determines how long a task takes. A task could be resolved in a few hours with the help of a senior engineer, which otherwise could take days to weeks. All said, I think simply generating the code will not be enough to replace programmers. Now, if an AI can generate code AND explain queries about logic, syntax, etc, the game is over.

gbasin 5y ago
Explainability for an NLP model is well within reach. I predict OpenAI will have something like this in the next 1-2 years
- altvali 5y ago
  Could you explain the reasoning behind your claim?

killerstorm 5y ago

AlphaCode is a piece of academic research. It's about demonstrating a possibility of using a particular type of a model to solve a particular type of tasks. It's not about making a practical tool. People can certainly take this approach and build tools. That might take different shapes and forms. There are many ways these models can be specialized, fine-tuned, combined with other approaches, etc. For example, somebody can try to apply them to a narrow use case, e.g. generate front-end code in React/JS and CSS from a backend API. They can fine-tune it on best example of React code, add a way to signal uncertainty so that specification can be clarified, etc. Nobody expects these models to be able to write an entire OS kernel any time soon. But a lot of types of programming are far more regular, repetitive and can be verified a lot easier.

microtherion 5y ago

> have the machine generate code and have the human review it The best characterization of such massive language models I've seen is "world class bullshitters": they are very good at producing output that superficially looks plausible, but neither know nor care whether what they're saying has any relationship to the truth. This is virtually guaranteed to make such code reviews very frustrating, and of course AlphaCode & co have no capability of explaining why they wrote a certain line. I can't see this having much of a role in a high quality code base, but I suspect the sort of management which currently would offshore coding to the lowest bidder would be quite enamored of it.

mirceal 5y ago
now you need an ai that reviews code. problem solved!

krajzeg 5y ago

You raise an interesting point about the difficulty of code writing versus code review, and I think the point generalizes nicely to many other areas where AI is currently rapidly progressing. For example, self-driving cars are also currently in this weird place where they're prone to making lots of errors, so a simple task for the human (driving) is replaced with a much more taxing one (keeping an eye on an AI driver and being ready to react at any moment), if we want to do it safely. Whether any of those applications will be able to progress past that weird space into "no/minimal supervision needed" remains an open question.