11 ms·

I use LLM-based autocomplete in my IDE, and it’s not taking away my job unless/until it improves by multiple orders of magnitude. It’s good at filling in boiler

by medler 3y ago

I use LLM-based autocomplete in my IDE, and it’s not taking away my job unless/until it improves by multiple orders of magnitude. It’s good at filling in boilerplate, but even for that I have to carefully check its output because it can make little errors even when I feel like what I want should be obvious. The article is absolutely correct in saying you have to be critical of its output. I would say it improves my productivity by maybe 5%, which is an incredible achievement. I’m already getting to where coding without it feels very tedious.

AlotOfReading 3y ago
I've also been using an LLM autocomplete for a few months, and yeah, it's pretty nice. My spouse was able to use it to write an Easter egg into a game while I was doing housework the other day.
firebirdn99 3y ago
Agreed 100%. It's helpful at filling out some functions maybe if you name them correctly, and boiler plate code. Eventually, they will get better, because these things get orders better with orders more scale. Society has to do something about all the jobs at that point, but we'll hopefully get a sense of how close/far is that, with ChatGPT 5, and the next versions coming up.
- Filligree 3y ago
  The biggest benefit, I’ve found, is it makes me comment my code. If I can make the AI understand what I want, then it turns out that three months later I’ll also be able to understand the code.
  - moonchrome 3y ago
    That's the worst part about generative AI IMO - it makes writing new code faster - it barely helps with editing existing code. So when someone eventually updates the code and forgets to update the comments I wouldn't be surprised if the misleading comments made AI hallucinate.
    - coliveira 3y ago
      I believe that AI will get so good at creating new code that a lot of existing libraries will be let unused. What is the point of using lots of libraries if AI can generate the code we need directly? The AI will be the library itself, and the generated code will embed the knowledge about doing lots of things for which we used libraries.
      - pydry 3y ago
        >What is the point of using lots of libraries if AI can generate the code we need directly? Theyve been debugged.
        vasili111 3y ago
        And also have documentation.
        throwuwu 3y ago
        At some point the models will produce code with a lower error rate than existing libraries.
        funnymony 3y ago
        How do you ensure quality? Review and testing. Reviewing is easier when there is less code (i.e. libraries are in use)
        throwuwu 3y ago
        It’s crazy how many people miss this. GPT models can review code too! They can also write and run tests. Once the context window is big enough to fit the whole code base into it they will be better at review than you are. Eventually we’ll have fine tuned models that are experts in any subject you can think of, the only barrier is data and a lot of recent research is showing that that can be machine generated too.
  - hyperbovine 3y ago
    What's the point of comments then. Just feed it back in in three months.
yayitswei 3y ago
I agree with 5%. That said, I've found rubber duck debugging to be an exceptionally effective use case for ChatGPT. Often it will surprise me by pinpointing the solution outright, but I'll always be making progress by clarifying my own thinking.
- hn_user82179 3y ago
  Fascinating! Can I ask how you use ChatGPT for debugging? are the bugs you've used it with more high level, "this is what's happening" kind of things? Or could you give an example?
  - dustymcp 3y ago
    just give it the log output or error code and your code then in alot of cases it comes up with decent solutions
  - yayitswei 3y ago
    It's similar to how you would describe a problem to a coworker on Slack. I give it some context, then I state the problem or paste in the error message/stacktrace. I might also list steps that I've taken already. Then I follow ChatGPT's suggestions to troubleshoot. Sometimes I need to supplement with my own ideas, but usually that's enough to iteratively bisect the issue.
- kromem 3y ago
  Yeah, it's an amazing rubber duck. Even in the IDE I'll sometimes just write comments like (arbitrary example out of thin air): // Q: Should we use a for loop or a while loop here? // A: It doesn't always have a great answer, but as you say, it almost always helps my own thinking about it, which is often much more valuable.
jmiskovic 3y ago
"Very tedious without it" doesn't sound like just 5% improvement? I've started developing in a new language and I can hardly do any work without the LLM assistance, the friction is just too high. Even when auto-competitions are completely wrong they still get the ball rolling, it's so much easier to fix the nicely formatted code than to write from scratch. In my case the improvement is vast, a difference from slacking off and actually being productive.
arrowsmith 3y ago
I find it increases my productivity about 5-10% when working with the technologies I'm the most familiar with and use regularly (Elixir, Phoenix, JavaScript, general web dev.) But when I'm doing something unfamiliar and new, it's more like 90%. It's incredible. Recently at work, for example, I've been setting up a bunch of stuff with some new technologies and libraries that I'd never really used before. Without ChatGPT I'd have spent hours if not days poring through tedious documentation and outdated tutorials while trying to hack something together in an agonising process of trial and error. But ChatGPT gave me a fantastic proof-of-concept app that has everything I needed to get started. It's been enormously helpful and I'm convinced it saved me days of work. This technology is miraculous. As for my job security... well, I think I'm safe for now; ChatGPT sped me up in this instance but the generated app still needs a skilled programmer to edit it, test it and deploy it. On the other hand I am slightly concerned that ChatGPT will destroy my side income from selling programming courses... so if you're a Rails developer who wants to learn Elixir and Phoenix, please check out my course Phoenix on Rails before we're both replaced by robots: PhoenixOnRails.com (Sorry for the self promotion but the code ELIXIRFORUM will give a $10 discount.)
- wouldbecouldbe 3y ago
  The thing is the hallucinations, I also wasted few hours trying to work on solutions with GPT where it just kept making up parameters and random functions.
  - yieldcrv 3y ago
    Yeah it can give you something that works out of the box, but fixing it requires even more effort Better to ask it for a bunch of small things and piece them together
    - mattkevan 3y ago
      Yes. Start small and build up. I’ve found it to be very forgetful and have to work function-by-function, giving it the current code as part of the next prompt. Otherwise it randomly changes class names, invents new bits that weren’t there before or forgets entire chunks of functionality. It’s a good discipline as I have to work out exactly what I want to achieve first and then build it up piece by piece. A great way to learn a new framework or language. It also sometimes picks convoluted ways of doing things, so regularly asking whether there’s a simpler way of doing things can be useful.
dave333 3y ago
Best way to check LLM output is to make it write its own tests and do TDD. Obviously someone has to check the tests but that is a 1% of the effort problem.
- dumpsterdiver 3y ago
  One percent? Are you really suggesting with a straight face that generated code could provide the other ninety-nine percent? If not, say what you actually mean. Don't bullshit us with trash numbers.
  - dave333 3y ago
    Let me guess, ENTJ right?
    - dumpsterdiver 3y ago
      No, I was just drinking at the time (so for a bit I may have been an alcohol induced ENTJ)... My reply was a little rude. I apologize for that. That wasn't a productive way for me to express my disagreement, and I should have chosen my words more thoughtfully.
yieldcrv 3y ago
It writes my unit tests super fast, and my method comments Its hard to say if it improves my productivity because I just wouldn’t have done those things But for the overall applications I think its improved a lot because we can implement best practices more consistently and catch regressions due to the aforementioned unit tests and documentation
jiggawatts 3y ago
The only widely available LLM-based autocomplete is GitHub Copilot, which is based on GPT 3. Notably, it's not GPT 3.5, it's 3.0, which is pretty stupid as far as the state of the art goes. The upcoming Copilot X will be based on GPT 4, which has "sparks of AGI". In my experience there is no comparison. GPT 3 is barely good enough for some trivial tab-complete tasks. GPT 4 can do quite complex tasks like generating documentation, useful tests, finding obscure bugs, etc...
- fnordpiglet 3y ago
  LLMs no matter how clever have no agency or creativity or ability to innovate, anticipate beyond what they’re prompted, etc. It’s crucial to realize that LLM chat interfaces disguise the fact they’re still completing a prompt. This isn’t AGI as AGI requires agency. GPT4/5 or whatever successor might be a key building block, and I suspect we’ve already discovered the missing elements in classical AI and the challenge will be integration, constraint, feedback, etc, but nothing will make LLMs alone AGI. That shouldn’t be surprising. Our brains are composed of many models, some heuristic, some optimizers, some solvers, some constrainers, and some generative. The answer won’t be a single magic thing operating in a black box. It’ll be an ensemble. We already see this effort beginning with plugins and things like langchain. This is the path forward.
  - jiggawatts 3y ago
    > “No agency or creativity or ability to innovate, anticipate beyond what they’re prompted, etc.” Sadly, you’ve just described the majority of the developers I’ve had to work with recently. Most have no agency, write boilerplate code with no creativity, need their hand held every step of they way, and won’t do anything they’re not explicitly ordered to do. You probably work in an SV startup with a highly skilled workforce. Out there in the real world there are armies of low-skill H1Bs and outsourcers that will soon be replaced with automation. It’s a recurring theme in economics. Outsource to low cost labour, insource with automation, repeat.
    - fnordpiglet 3y ago
fnordpiglet 3y ago
Given AI (today) has no direct agency and can’t create anything unless directly prompted, and engineering is largely domain discovery and resolving of unforeseen edges in a domain, I don’t think we are going to see a time where generative AI alone is able to be more than an assistant. It’ll likely improve, but given it only can react to what it is given/told/fed and inherently can’t innovate or create or discover, despite the illusion that it might be from the position of the users ignorance of details the AI can produce, it’ll be an increasingly powerful adjunct to increasingly capable engineers. The problems we solve will be more interesting, we will produce better software faster, but I’ve never seen the world as lacking in problems to solve but rather capacity to solve them well or quickly enough given the iterative time it takes to develop software. I think this current trend of generative AI will help improve that situation, but will likely make software engineers even more in demand as the possible uses of software become more ubiquitous as the per unit cost of development goes down.
gexla 3y ago
I think the real problem is going to be increased volatility in the work market. You get a chaotic situation in which the bullet that strikes you is the one you would have never guessed. For example, it could be that short term, the increased productivity squeezes workers in every industry and the concern becomes increased competition. You aren't getting replaced by AI, you're getting replaced by someone who out-competed you. The market may adjust over the longer term, or it may just continue to be volatile as the rate of change accelerates. In that case, we can't fix the work market, and we instead have to address the need for people to feed themselves another way.
kromem 3y ago
Oh, it will improve by several orders of magnitude. But even then, it's not 'replacing' you. It's just going to let you spend less time on BS and more time on the things that are your maximal value contributions to a project. If I had a dozen junior or mid level devs you could hand work off to, would that save you time? Would you kick back and not review what they were doing, particularly around business critical parts of the software? The conversation around AI has become obscenely binary, pulling from (now obsolete) SciFi influences to cast it as humans vs machines. But it's a false dichotomy. Collaborative efforts are almost certainly where this is going, and 100% human or 100% AI will both be significantly inferior to a mix of both.
- jabradoodle 3y ago
  For sure it will still mostly make sense to have a division of labour where you have people who are focused on building software. The question is if generative AI is powerful enough to reduce the number of programmers needed to achieve a task, without creating enough opportunities to replace those programmers. Before we are all replaced there could be a moment where demand for software engineers is 10x less.
  - Gigachad 3y ago
    Society would simply demand more capable and complex software. Specialized industrial applications that currently look like windows 98 java apps would be expected to be as polished as iOS.
    - jabradoodle 3y ago
      I don't think there is some natural law that dictates we will need enough new software that we will always increase demand in the face of efficiency gains. For industrial applications in particular they need to be functional and operable, not shiny.

by medler 3y ago

AlotOfReading 3y ago

I've also been using an LLM autocomplete for a few months, and yeah, it's pretty nice. My spouse was able to use it to write an Easter egg into a game while I was doing housework the other day.

firebirdn99 3y ago

Agreed 100%. It's helpful at filling out some functions maybe if you name them correctly, and boiler plate code. Eventually, they will get better, because these things get orders better with orders more scale. Society has to do something about all the jobs at that point, but we'll hopefully get a sense of how close/far is that, with ChatGPT 5, and the next versions coming up.

Filligree 3y ago
The biggest benefit, I’ve found, is it makes me comment my code. If I can make the AI understand what I want, then it turns out that three months later I’ll also be able to understand the code.
- moonchrome 3y ago
  That's the worst part about generative AI IMO - it makes writing new code faster - it barely helps with editing existing code. So when someone eventually updates the code and forgets to update the comments I wouldn't be surprised if the misleading comments made AI hallucinate.
  - coliveira 3y ago
    I believe that AI will get so good at creating new code that a lot of existing libraries will be let unused. What is the point of using lots of libraries if AI can generate the code we need directly? The AI will be the library itself, and the generated code will embed the knowledge about doing lots of things for which we used libraries.
    - pydry 3y ago
      >What is the point of using lots of libraries if AI can generate the code we need directly? Theyve been debugged.
      - vasili111 3y ago
        And also have documentation.
      - throwuwu 3y ago
        At some point the models will produce code with a lower error rate than existing libraries.
        funnymony 3y ago
        How do you ensure quality? Review and testing. Reviewing is easier when there is less code (i.e. libraries are in use)
        throwuwu 3y ago
        It’s crazy how many people miss this. GPT models can review code too! They can also write and run tests. Once the context window is big enough to fit the whole code base into it they will be better at review than you are. Eventually we’ll have fine tuned models that are experts in any subject you can think of, the only barrier is data and a lot of recent research is showing that that can be machine generated too.
- hyperbovine 3y ago
  What's the point of comments then. Just feed it back in in three months.

yayitswei 3y ago

I agree with 5%. That said, I've found rubber duck debugging to be an exceptionally effective use case for ChatGPT. Often it will surprise me by pinpointing the solution outright, but I'll always be making progress by clarifying my own thinking.

hn_user82179 3y ago
Fascinating! Can I ask how you use ChatGPT for debugging? are the bugs you've used it with more high level, "this is what's happening" kind of things? Or could you give an example?
- dustymcp 3y ago
  just give it the log output or error code and your code then in alot of cases it comes up with decent solutions
- yayitswei 3y ago
  It's similar to how you would describe a problem to a coworker on Slack. I give it some context, then I state the problem or paste in the error message/stacktrace. I might also list steps that I've taken already. Then I follow ChatGPT's suggestions to troubleshoot. Sometimes I need to supplement with my own ideas, but usually that's enough to iteratively bisect the issue.
kromem 3y ago
Yeah, it's an amazing rubber duck. Even in the IDE I'll sometimes just write comments like (arbitrary example out of thin air): // Q: Should we use a for loop or a while loop here? // A: It doesn't always have a great answer, but as you say, it almost always helps my own thinking about it, which is often much more valuable.

jmiskovic 3y ago

"Very tedious without it" doesn't sound like just 5% improvement? I've started developing in a new language and I can hardly do any work without the LLM assistance, the friction is just too high. Even when auto-competitions are completely wrong they still get the ball rolling, it's so much easier to fix the nicely formatted code than to write from scratch. In my case the improvement is vast, a difference from slacking off and actually being productive.

arrowsmith 3y ago

I find it increases my productivity about 5-10% when working with the technologies I'm the most familiar with and use regularly (Elixir, Phoenix, JavaScript, general web dev.) But when I'm doing something unfamiliar and new, it's more like 90%. It's incredible. Recently at work, for example, I've been setting up a bunch of stuff with some new technologies and libraries that I'd never really used before. Without ChatGPT I'd have spent hours if not days poring through tedious documentation and outdated tutorials while trying to hack something together in an agonising process of trial and error. But ChatGPT gave me a fantastic proof-of-concept app that has everything I needed to get started. It's been enormously helpful and I'm convinced it saved me days of work. This technology is miraculous. As for my job security... well, I think I'm safe for now; ChatGPT sped me up in this instance but the generated app still needs a skilled programmer to edit it, test it and deploy it. On the other hand I am slightly concerned that ChatGPT will destroy my side income from selling programming courses... so if you're a Rails developer who wants to learn Elixir and Phoenix, please check out my course Phoenix on Rails before we're both replaced by robots: PhoenixOnRails.com (Sorry for the self promotion but the code ELIXIRFORUM will give a $10 discount.)

wouldbecouldbe 3y ago
The thing is the hallucinations, I also wasted few hours trying to work on solutions with GPT where it just kept making up parameters and random functions.
- yieldcrv 3y ago
  Yeah it can give you something that works out of the box, but fixing it requires even more effort Better to ask it for a bunch of small things and piece them together
  - mattkevan 3y ago
    Yes. Start small and build up. I’ve found it to be very forgetful and have to work function-by-function, giving it the current code as part of the next prompt. Otherwise it randomly changes class names, invents new bits that weren’t there before or forgets entire chunks of functionality. It’s a good discipline as I have to work out exactly what I want to achieve first and then build it up piece by piece. A great way to learn a new framework or language. It also sometimes picks convoluted ways of doing things, so regularly asking whether there’s a simpler way of doing things can be useful.

dave333 3y ago

Best way to check LLM output is to make it write its own tests and do TDD. Obviously someone has to check the tests but that is a 1% of the effort problem.

dumpsterdiver 3y ago
One percent? Are you really suggesting with a straight face that generated code could provide the other ninety-nine percent? If not, say what you actually mean. Don't bullshit us with trash numbers.
- dave333 3y ago
  Let me guess, ENTJ right?
  - dumpsterdiver 3y ago
    No, I was just drinking at the time (so for a bit I may have been an alcohol induced ENTJ)... My reply was a little rude. I apologize for that. That wasn't a productive way for me to express my disagreement, and I should have chosen my words more thoughtfully.

yieldcrv 3y ago

It writes my unit tests super fast, and my method comments Its hard to say if it improves my productivity because I just wouldn’t have done those things But for the overall applications I think its improved a lot because we can implement best practices more consistently and catch regressions due to the aforementioned unit tests and documentation

jiggawatts 3y ago

The only widely available LLM-based autocomplete is GitHub Copilot, which is based on GPT 3. Notably, it's not GPT 3.5, it's 3.0, which is pretty stupid as far as the state of the art goes. The upcoming Copilot X will be based on GPT 4, which has "sparks of AGI". In my experience there is no comparison. GPT 3 is barely good enough for some trivial tab-complete tasks. GPT 4 can do quite complex tasks like generating documentation, useful tests, finding obscure bugs, etc...

fnordpiglet 3y ago
LLMs no matter how clever have no agency or creativity or ability to innovate, anticipate beyond what they’re prompted, etc. It’s crucial to realize that LLM chat interfaces disguise the fact they’re still completing a prompt. This isn’t AGI as AGI requires agency. GPT4/5 or whatever successor might be a key building block, and I suspect we’ve already discovered the missing elements in classical AI and the challenge will be integration, constraint, feedback, etc, but nothing will make LLMs alone AGI. That shouldn’t be surprising. Our brains are composed of many models, some heuristic, some optimizers, some solvers, some constrainers, and some generative. The answer won’t be a single magic thing operating in a black box. It’ll be an ensemble. We already see this effort beginning with plugins and things like langchain. This is the path forward.
- jiggawatts 3y ago
  > “No agency or creativity or ability to innovate, anticipate beyond what they’re prompted, etc.” Sadly, you’ve just described the majority of the developers I’ve had to work with recently. Most have no agency, write boilerplate code with no creativity, need their hand held every step of they way, and won’t do anything they’re not explicitly ordered to do. You probably work in an SV startup with a highly skilled workforce. Out there in the real world there are armies of low-skill H1Bs and outsourcers that will soon be replaced with automation. It’s a recurring theme in economics. Outsource to low cost labour, insource with automation, repeat.
  - fnordpiglet 3y ago

fnordpiglet 3y ago

Given AI (today) has no direct agency and can’t create anything unless directly prompted, and engineering is largely domain discovery and resolving of unforeseen edges in a domain, I don’t think we are going to see a time where generative AI alone is able to be more than an assistant. It’ll likely improve, but given it only can react to what it is given/told/fed and inherently can’t innovate or create or discover, despite the illusion that it might be from the position of the users ignorance of details the AI can produce, it’ll be an increasingly powerful adjunct to increasingly capable engineers. The problems we solve will be more interesting, we will produce better software faster, but I’ve never seen the world as lacking in problems to solve but rather capacity to solve them well or quickly enough given the iterative time it takes to develop software. I think this current trend of generative AI will help improve that situation, but will likely make software engineers even more in demand as the possible uses of software become more ubiquitous as the per unit cost of development goes down.

gexla 3y ago

I think the real problem is going to be increased volatility in the work market. You get a chaotic situation in which the bullet that strikes you is the one you would have never guessed. For example, it could be that short term, the increased productivity squeezes workers in every industry and the concern becomes increased competition. You aren't getting replaced by AI, you're getting replaced by someone who out-competed you. The market may adjust over the longer term, or it may just continue to be volatile as the rate of change accelerates. In that case, we can't fix the work market, and we instead have to address the need for people to feed themselves another way.

kromem 3y ago

Oh, it will improve by several orders of magnitude. But even then, it's not 'replacing' you. It's just going to let you spend less time on BS and more time on the things that are your maximal value contributions to a project. If I had a dozen junior or mid level devs you could hand work off to, would that save you time? Would you kick back and not review what they were doing, particularly around business critical parts of the software? The conversation around AI has become obscenely binary, pulling from (now obsolete) SciFi influences to cast it as humans vs machines. But it's a false dichotomy. Collaborative efforts are almost certainly where this is going, and 100% human or 100% AI will both be significantly inferior to a mix of both.

jabradoodle 3y ago
For sure it will still mostly make sense to have a division of labour where you have people who are focused on building software. The question is if generative AI is powerful enough to reduce the number of programmers needed to achieve a task, without creating enough opportunities to replace those programmers. Before we are all replaced there could be a moment where demand for software engineers is 10x less.
- Gigachad 3y ago
  Society would simply demand more capable and complex software. Specialized industrial applications that currently look like windows 98 java apps would be expected to be as polished as iOS.
  - jabradoodle 3y ago
    I don't think there is some natural law that dictates we will need enough new software that we will always increase demand in the face of efficiency gains. For industrial applications in particular they need to be functional and operable, not shiny.