4 ms·

LLMs can currently auto-complete a medium complexity function, and with some hand-holding can just about manage a class with basic state. I'm not talking about

LLMs can currently auto-complete a medium complexity function, and with some hand-holding can just about manage a class with basic state. I'm not talking about a text-book binary tree implementation, they can obviously manage those or problems that boil down to those, because they have clear patterns to them and have been repeated again and again, but that's not most code. I think they need to be roughly 3 orders of magnitude better than they currently are to meaningfully replace engineers. Managing architecture across many classes is a clear step change up from one class (or your language abstraction of choice). Managing state, persistence, consistency concerns, are another significant step-change in ability – not at the level of 3 interdependent variables, but more like 100s of data structures across asynchronous code. And lastly, debugging all of this and understanding how the system works as a whole is another significant step change. It's not obvious to me that a large _language_ model has the ability to understand these abstractions. I suspect LLMs will therefore max-out for programming applications (and others) "soon", and that we'll need another technological leap to get there.

mlboss 4y ago
This does not match my experience with ChatGPT 4. Just now I added a new api endpoint complete with database changes with unit test. Everything took around 1 hour with back and forth communication. This is at the level of developer with 5 year experience.
jstummbillig 4y ago
> to meaningfully replace engineers When engineers are producing any significant increase in productivity by use of any tool, this tool is statistically replacing engineers. And, with what is currently happening in AI (going by the self reported numbers that I picked up) the former seems to be true to a wild degree.
- jeremyjh 4y ago
  Yes and this is true of most use-cases of ChatGPT. It makes experts more productive but it cannot reliably produce expert quality work in the hands of a non-expert.
- travisjungroth 4y ago
  Not always. Compilers didn't replace programmers. An increase in supply can cause a drop in price that causes an increase in demand: https://en.wikipedia.org/wiki/Induced_demand https://en.wikipedia.org/wiki/Induced_demand
  - danpalmer 4y ago
    Exactly. Better auto-complete didn't either, and I think we're still very much in the category of better auto-complete, rather than writing complete software from scratch.
  - coldtea 4y ago
    In this case we were already in a decreased demand phase, due to lack of money in the economy. And unlike the era of the arrival of compilers, we're not in an rapidly-expanging-computer-market stage either. Only the "replace programmers with AI" segments are growing.
    - quonn 4y ago
      How are they growing? Nobody is replacing anyone so far. And a short term business cycle is not very relevant either. By the time practical tools are read the cycle may have inverted.
      - coldtea 4y ago
        >How are they growing? Increasingly more tools aimed at that are produced, and they are increasingly adopted. From CoPilot to specialized tools for end users to do what programmers (and other jobs like graphic designers) did for them. >And a short term business cycle is not very relevant either. By the time practical tools are read the cycle may have inverted. Who said it's short term?
        travisjungroth 4y ago
        You commented about phases, stages, and lack of money in the economy. Sounds like short term cycles.
- rileymat2 4y ago
  Not if there are projects that become economically feasible at higher productivity numbers. For example, I have a friend who owns pub that wanted a membership loyalty program customized to his business. He had a lot of great ideas, but I told him that this project is too big for the benefit he will get out of it; he would need to get 4 or 5 other bar owners together with similar needs. I believe there is a lot of software to be built if we radically change productivity. I believe This will keep the engineering demand up, and people employed.
- ttyprintk 4y ago
  There’s replacing engineer hours and there’s replacing engineering. I thing GP is talking about a 1000x capable AI replacing the division.
zamnos 4y ago
It's not obvious (to me, at least), that a large language model would ever have any of the characteristics of ChatGPT. GPT-2 is neat to play around with, but it's just so far away from ChatGPT(-3.5 or -4) that I didn't see it coming. Nor did I see a mere denoising algorithm coming up with StableDiffusion, but here we are. Given fine-tuning of the model, and improvements allowing for a much larger context window size, I think it'll be possible for LLMs to be able to turn a class inside out and then refactor its code and translate it into another language, while generating unit tests that prove it all still works correctly. How much faster could programming language design iterate if legacy code compatibility wasn't an issue?
- quonn 4y ago
  Probably not much. In fact I believe that these LLMs should favor simpler languages that are more verbose, because verbosity shouldn‘t matter that much anymore.

mlboss 4y ago

This does not match my experience with ChatGPT 4. Just now I added a new api endpoint complete with database changes with unit test. Everything took around 1 hour with back and forth communication. This is at the level of developer with 5 year experience.

jstummbillig 4y ago

> to meaningfully replace engineers When engineers are producing any significant increase in productivity by use of any tool, this tool is statistically replacing engineers. And, with what is currently happening in AI (going by the self reported numbers that I picked up) the former seems to be true to a wild degree.

jeremyjh 4y ago
Yes and this is true of most use-cases of ChatGPT. It makes experts more productive but it cannot reliably produce expert quality work in the hands of a non-expert.
travisjungroth 4y ago
Not always. Compilers didn't replace programmers. An increase in supply can cause a drop in price that causes an increase in demand: https://en.wikipedia.org/wiki/Induced_demand https://en.wikipedia.org/wiki/Induced_demand
- danpalmer 4y ago
  Exactly. Better auto-complete didn't either, and I think we're still very much in the category of better auto-complete, rather than writing complete software from scratch.
- coldtea 4y ago
  In this case we were already in a decreased demand phase, due to lack of money in the economy. And unlike the era of the arrival of compilers, we're not in an rapidly-expanging-computer-market stage either. Only the "replace programmers with AI" segments are growing.
  - quonn 4y ago
    How are they growing? Nobody is replacing anyone so far. And a short term business cycle is not very relevant either. By the time practical tools are read the cycle may have inverted.
    - coldtea 4y ago
      >How are they growing? Increasingly more tools aimed at that are produced, and they are increasingly adopted. From CoPilot to specialized tools for end users to do what programmers (and other jobs like graphic designers) did for them. >And a short term business cycle is not very relevant either. By the time practical tools are read the cycle may have inverted. Who said it's short term?
      - travisjungroth 4y ago
        You commented about phases, stages, and lack of money in the economy. Sounds like short term cycles.
rileymat2 4y ago
Not if there are projects that become economically feasible at higher productivity numbers. For example, I have a friend who owns pub that wanted a membership loyalty program customized to his business. He had a lot of great ideas, but I told him that this project is too big for the benefit he will get out of it; he would need to get 4 or 5 other bar owners together with similar needs. I believe there is a lot of software to be built if we radically change productivity. I believe This will keep the engineering demand up, and people employed.
ttyprintk 4y ago
There’s replacing engineer hours and there’s replacing engineering. I thing GP is talking about a 1000x capable AI replacing the division.

zamnos 4y ago

It's not obvious (to me, at least), that a large language model would ever have any of the characteristics of ChatGPT. GPT-2 is neat to play around with, but it's just so far away from ChatGPT(-3.5 or -4) that I didn't see it coming. Nor did I see a mere denoising algorithm coming up with StableDiffusion, but here we are. Given fine-tuning of the model, and improvements allowing for a much larger context window size, I think it'll be possible for LLMs to be able to turn a class inside out and then refactor its code and translate it into another language, while generating unit tests that prove it all still works correctly. How much faster could programming language design iterate if legacy code compatibility wasn't an issue?

quonn 4y ago
Probably not much. In fact I believe that these LLMs should favor simpler languages that are more verbose, because verbosity shouldn‘t matter that much anymore.