4 ms·

> Can an average programmer using GPT4 as a tool solve Advent of Code faster than an equally-skilled programmer without an LLM? I mean, yes, obviously? Any too

by qsort 3y ago

> Can an average programmer using GPT4 as a tool solve Advent of Code faster than an equally-skilled programmer without an LLM? I mean, yes, obviously? Any tool, even if it had near-zero marginal utility must necessarily improve performance, as you can simply elect not to use it. The disagreement is not in the direction, it's in the magnitude. Therefore the real test is: "Can an average programmer solve AoC faster with GPT4 and without syntax coloring or with syntax coloring and without GPT4?", or "GPT4 in C versus no-GPT4 in Python", or "GPT4 on a crappy laptop vs no-GPT4 on a high-end workstation" and so on.

Closi 3y ago
Why are your tests changing multiple variables? A simpler test would be keeping the development environment the same and then adding GPT4 to see if there is a statistically significant and meaningful speed increase (of a decent magnitude). I'm not looking for a 1% speed improvement, I'm looking for a >50% speed improvement. Maybe I should have stated 'significant' speed improvement in the initial post.
- qsort 3y ago
  It's not meant to be a scientifically rigorous test, but the idea is this. It seems like you're accepting that (at least in its current state, barring unknowable future discoveries) this is a technology that complements developers, making them more productive. If this is the case, then the question is how relevant this improvement is in quantitative terms. The cumulative improvements in developer productivity since the days of punching cards have easily been several hundred percent. A thought experiment to figure out how relatively impactful this is would be to compare it to other technologies and see which gives the greatest boost. My prior belief: somewhere around Intellisense level of useful, but not significantly more than that.
  - Closi 3y ago
    Yes, I think the current state of technology (just starting) is a technology that complements developers. Although my personal belief is this is more likely going to be an improvement similar to Assembly -> C (even if the LLM component doesn't improve, but assuming that the tooling does improve). Personally I think there is going to be a new 'higher, higher level' of programming paradigms that are about to be invented that are supported by the ability of LLM's to write code - still augmenting humans, but making them many times more productive.
    - bcrosby95 3y ago
      I think the opposite - LLMs will cause programming language development to become stagnant because the language that compliments an LLM the best is the language that has the most code out there in the wild. So the "LLM language" already exists - its Javascript, Java, C#, etc.
      - Closi 3y ago
        I'm not talking about a language that an LLM outputs. I'm talking about a higher-level language that outputs Javascript, Java, C# etc (but presumably some sort of lower level byte-code in the future). A language where intent and clarity is more important than syntax or implementation details. Where menu_options = [process['name'] for process in processes if process['active']] is more clearly expressed as menu_options = list of the active process names (or something similar)
chefandy 3y ago
> Any tool, even if it had near-zero marginal utility must necessarily improve performance, as you can simply elect not to use it. That might be true for any tool with consistent, predictable output that you thoroughly understand. Someone could easily lose a few hours dickering around with chatgpt only to realize they’re better off just starting it from scratch.
ska 3y ago
> Any tool, even if it had near-zero marginal utility must necessarily improve performance, as you can simply elect not to use it. No opinion on the specific tradeoff here, but in general it's not obvious to me that your statement is true. Using tools involves opportunity cost, so sometimes "not electing to us it" can on average be a net win, no? There is a cognitive and time load to using it at all, right?
jiggawatts 3y ago
This reminds me of a study where they compared the time taken to solve a problem comparable to a two-star AoC problem in difficulty across different dev environments, such as languages and IDEs. There are a lot of anecdotes about how "it's the programmer, not the language" or "any language can be productive", etc... The actual results were that many popular languages are more than an order of magnitude slower to achieve time-to-correct-solution than others. From memory, the fastest was F# with about 20 minutes, then Python and C# with about 40 minutes for both, and then C/C++/Java were hours to even days!
- zdimension 3y ago
  Do you have a link to that study? I'd be very interested in the full results!

by qsort 3y ago

Closi 3y ago

Why are your tests changing multiple variables? A simpler test would be keeping the development environment the same and then adding GPT4 to see if there is a statistically significant and meaningful speed increase (of a decent magnitude). I'm not looking for a 1% speed improvement, I'm looking for a >50% speed improvement. Maybe I should have stated 'significant' speed improvement in the initial post.

qsort 3y ago
It's not meant to be a scientifically rigorous test, but the idea is this. It seems like you're accepting that (at least in its current state, barring unknowable future discoveries) this is a technology that complements developers, making them more productive. If this is the case, then the question is how relevant this improvement is in quantitative terms. The cumulative improvements in developer productivity since the days of punching cards have easily been several hundred percent. A thought experiment to figure out how relatively impactful this is would be to compare it to other technologies and see which gives the greatest boost. My prior belief: somewhere around Intellisense level of useful, but not significantly more than that.
- Closi 3y ago
  Yes, I think the current state of technology (just starting) is a technology that complements developers. Although my personal belief is this is more likely going to be an improvement similar to Assembly -> C (even if the LLM component doesn't improve, but assuming that the tooling does improve). Personally I think there is going to be a new 'higher, higher level' of programming paradigms that are about to be invented that are supported by the ability of LLM's to write code - still augmenting humans, but making them many times more productive.
  - bcrosby95 3y ago
    I think the opposite - LLMs will cause programming language development to become stagnant because the language that compliments an LLM the best is the language that has the most code out there in the wild. So the "LLM language" already exists - its Javascript, Java, C#, etc.
    - Closi 3y ago
      I'm not talking about a language that an LLM outputs. I'm talking about a higher-level language that outputs Javascript, Java, C# etc (but presumably some sort of lower level byte-code in the future). A language where intent and clarity is more important than syntax or implementation details. Where menu_options = [process['name'] for process in processes if process['active']] is more clearly expressed as menu_options = list of the active process names (or something similar)

chefandy 3y ago

> Any tool, even if it had near-zero marginal utility must necessarily improve performance, as you can simply elect not to use it. That might be true for any tool with consistent, predictable output that you thoroughly understand. Someone could easily lose a few hours dickering around with chatgpt only to realize they’re better off just starting it from scratch.

ska 3y ago

> Any tool, even if it had near-zero marginal utility must necessarily improve performance, as you can simply elect not to use it. No opinion on the specific tradeoff here, but in general it's not obvious to me that your statement is true. Using tools involves opportunity cost, so sometimes "not electing to us it" can on average be a net win, no? There is a cognitive and time load to using it at all, right?

jiggawatts 3y ago

This reminds me of a study where they compared the time taken to solve a problem comparable to a two-star AoC problem in difficulty across different dev environments, such as languages and IDEs. There are a lot of anecdotes about how "it's the programmer, not the language" or "any language can be productive", etc... The actual results were that many popular languages are more than an order of magnitude slower to achieve time-to-correct-solution than others. From memory, the fastest was F# with about 20 minutes, then Python and C# with about 40 minutes for both, and then C/C++/Java were hours to even days!

zdimension 3y ago
Do you have a link to that study? I'd be very interested in the full results!