4 ms·

> I think it's as close to the perfect benchmark as you can get. Depends what you are benchmarking for... If you are benchmarking the ability of the solution t

by Closi 3y ago

> I think it's as close to the perfect benchmark as you can get. Depends what you are benchmarking for... If you are benchmarking the ability of the solution to solve LEETCODE challenges, that is different to the ability of GPT4 to assist everyday programmers knock out business logic or diagnose bugs. My experience of GPT4 is that it's significantly better at the latter than GPT3.5. Additionally, the real test is for me is "Can an average programmer using GPT4 as a tool solve Advent of Code faster than an equally-skilled programmer without an LLM?".

nirvael 3y ago
>diagnose bugs Literally says in the article that GPT's main drawback is that it can't debug in the same way a human can.
- okasaki 3y ago
  It can if you use the python interpreter.
- Closi 3y ago
  Fixing bugs in code it writes itself is very different to diagnosing (i.e. identifying, not fixing) bugs in my own code. When stuck I often paste code into ChatGPT and ask why it doesn't work, and it will often help me quickly identify the error and propose a fix.
  - tnel77 3y ago
    I have done the same thing. “I am seeing the following behavior, but I expected <X>. What am I missing?” That line has often quickly solved some bugs that would have otherwise taken some time to debug. Very handy!
    - DaiPlusPlus 3y ago
      Can you share some examples of this? I haven’t had much luck with ChatGPT correctly identifying issues because (in my case, at least) they stem from other parts of a large codebase, and (last time I checked) I couldn’t paste more than a few kilobytes of code into ChatGPT. One example are bugs caused by precondition violations, which ChatGPT can’t diagnose without also being given the code to all of the incoming call-sites, which means you end-up solving the problem yourself before you’ve even explained the issue to ChatGPT - so (to me, at least) my use of ChatGPT is more akin to rubber-duck-debugging[1] than anything else. [1]: https://en.wikipedia.org/wiki/Rubber_duck_debugging https://en.wikipedia.org/wiki/Rubber_duck_debugging
  - blibble 3y ago
    > Fixing bugs in code it writes itself is very different to diagnosing (i.e. identifying, not fixing) bugs in my own code. yeah, we run the random number generator again and hopefully this time less buggy code pops out
- airstrike 3y ago
  Consider for a moment the possibility that TFA is wrong
qsort 3y ago
> Can an average programmer using GPT4 as a tool solve Advent of Code faster than an equally-skilled programmer without an LLM? I mean, yes, obviously? Any tool, even if it had near-zero marginal utility must necessarily improve performance, as you can simply elect not to use it. The disagreement is not in the direction, it's in the magnitude. Therefore the real test is: "Can an average programmer solve AoC faster with GPT4 and without syntax coloring or with syntax coloring and without GPT4?", or "GPT4 in C versus no-GPT4 in Python", or "GPT4 on a crappy laptop vs no-GPT4 on a high-end workstation" and so on.
- Closi 3y ago
  Why are your tests changing multiple variables? A simpler test would be keeping the development environment the same and then adding GPT4 to see if there is a statistically significant and meaningful speed increase (of a decent magnitude). I'm not looking for a 1% speed improvement, I'm looking for a >50% speed improvement. Maybe I should have stated 'significant' speed improvement in the initial post.
  - qsort 3y ago
    It's not meant to be a scientifically rigorous test, but the idea is this. It seems like you're accepting that (at least in its current state, barring unknowable future discoveries) this is a technology that complements developers, making them more productive. If this is the case, then the question is how relevant this improvement is in quantitative terms. The cumulative improvements in developer productivity since the days of punching cards have easily been several hundred percent. A thought experiment to figure out how relatively impactful this is would be to compare it to other technologies and see which gives the greatest boost. My prior belief: somewhere around Intellisense level of useful, but not significantly more than that.
    - Closi 3y ago
      Yes, I think the current state of technology (just starting) is a technology that complements developers. Although my personal belief is this is more likely going to be an improvement similar to Assembly -> C (even if the LLM component doesn't improve, but assuming that the tooling does improve). Personally I think there is going to be a new 'higher, higher level' of programming paradigms that are about to be invented that are supported by the ability of LLM's to write code - still augmenting humans, but making them many times more productive.

by Closi 3y ago

nirvael 3y ago

>diagnose bugs Literally says in the article that GPT's main drawback is that it can't debug in the same way a human can.

okasaki 3y ago
It can if you use the python interpreter.
Closi 3y ago
Fixing bugs in code it writes itself is very different to diagnosing (i.e. identifying, not fixing) bugs in my own code. When stuck I often paste code into ChatGPT and ask why it doesn't work, and it will often help me quickly identify the error and propose a fix.
- tnel77 3y ago
  I have done the same thing. “I am seeing the following behavior, but I expected <X>. What am I missing?” That line has often quickly solved some bugs that would have otherwise taken some time to debug. Very handy!
  - DaiPlusPlus 3y ago
    Can you share some examples of this? I haven’t had much luck with ChatGPT correctly identifying issues because (in my case, at least) they stem from other parts of a large codebase, and (last time I checked) I couldn’t paste more than a few kilobytes of code into ChatGPT. One example are bugs caused by precondition violations, which ChatGPT can’t diagnose without also being given the code to all of the incoming call-sites, which means you end-up solving the problem yourself before you’ve even explained the issue to ChatGPT - so (to me, at least) my use of ChatGPT is more akin to rubber-duck-debugging[1] than anything else. [1]: https://en.wikipedia.org/wiki/Rubber_duck_debugging https://en.wikipedia.org/wiki/Rubber_duck_debugging
- blibble 3y ago
  > Fixing bugs in code it writes itself is very different to diagnosing (i.e. identifying, not fixing) bugs in my own code. yeah, we run the random number generator again and hopefully this time less buggy code pops out
airstrike 3y ago
Consider for a moment the possibility that TFA is wrong

qsort 3y ago

> Can an average programmer using GPT4 as a tool solve Advent of Code faster than an equally-skilled programmer without an LLM? I mean, yes, obviously? Any tool, even if it had near-zero marginal utility must necessarily improve performance, as you can simply elect not to use it. The disagreement is not in the direction, it's in the magnitude. Therefore the real test is: "Can an average programmer solve AoC faster with GPT4 and without syntax coloring or with syntax coloring and without GPT4?", or "GPT4 in C versus no-GPT4 in Python", or "GPT4 on a crappy laptop vs no-GPT4 on a high-end workstation" and so on.

Closi 3y ago
Why are your tests changing multiple variables? A simpler test would be keeping the development environment the same and then adding GPT4 to see if there is a statistically significant and meaningful speed increase (of a decent magnitude). I'm not looking for a 1% speed improvement, I'm looking for a >50% speed improvement. Maybe I should have stated 'significant' speed improvement in the initial post.
- qsort 3y ago
  It's not meant to be a scientifically rigorous test, but the idea is this. It seems like you're accepting that (at least in its current state, barring unknowable future discoveries) this is a technology that complements developers, making them more productive. If this is the case, then the question is how relevant this improvement is in quantitative terms. The cumulative improvements in developer productivity since the days of punching cards have easily been several hundred percent. A thought experiment to figure out how relatively impactful this is would be to compare it to other technologies and see which gives the greatest boost. My prior belief: somewhere around Intellisense level of useful, but not significantly more than that.
  - Closi 3y ago
    Yes, I think the current state of technology (just starting) is a technology that complements developers. Although my personal belief is this is more likely going to be an improvement similar to Assembly -> C (even if the LLM component doesn't improve, but assuming that the tooling does improve). Personally I think there is going to be a new 'higher, higher level' of programming paradigms that are about to be invented that are supported by the ability of LLM's to write code - still augmenting humans, but making them many times more productive.