3 ms·

Interesting study, but unfortunately already outdated. I don't think anybody uses ChatGPT 3.5 for coding anymore. From my personal experience, Claude 3.5 or GP

Interesting study, but unfortunately already outdated. I don't think anybody uses ChatGPT 3.5 for coding anymore. From my personal experience, Claude 3.5 or GPT-4o work best. They're more coding assistants, not really capable of writing anything more than very simple programs on their own. They make a lot of mistakes, and you need to know how to debug code they produce. Claude is my favorite but it just randomly added a division by 100,000 to a line of code for no discernible reason. According to Claude it was "an oversight on [Claude's] part".

trashface 2y ago
I was using 3.5 until quite about a month ago. Now I'm using 4o. 4o is better but its not a huge difference. I was surprised as I expected it to be a huge improvement. I haven't tried the regular "4" model yet, I've heard it used to be quite good but maybe got worse. In any case, these models are good for simple stuff as far as I can tell, but can't do anything hard or off the beaten path. They can give me ideas for solving hard issues, but any code they generate is killed by hallucinations and general lack of anything resembling contextual knowledge. It can be quite obstinate too, even if I explain I'm trying to solve an unusual problem, and so need out of the box "thinking", 4o will try to force me back to some mainstream solution that I can't use. I also use Amazon's Q and that is good for simple/repetitive automation but often generates extremely wacky stuff otherwise. They are all apparently better at Python than anything else. I don't use that so maybe that's an issue.
gtirloni 2y ago
Exactly what the article is saying.
- calibas 2y ago
  Maybe my reading comprehension is awful, but I don't see it mentioned anywhere in the article that the "ChatGPT" from the study is the worst at coding of the 3 models that people commonly use. It seems relevant to mention that the "ChatGPT" in the article isn't the one most of us are using for coding.
  - gtirloni 2y ago
    It's not that relevant because there isn't a huge inherent difference between both. It's more of an evolutionary difference based on training data and small improvemens. The article mentions the model doesn't work well on things that it has not seen in its training data, which makes the whole argument of AGI spontaneously appearing given more data a bit unrealistic. We can discuss percentages in this or that task but the article points to the root cause which I think is inescapable.
    - p1esk 2y ago
      There’s a huge difference between 3.5 and 4. 3.5 is pretty much useless. 4 is very decent, it saved me countless hours at work.
    - calibas 2y ago
      > The article mentions the model doesn't work well on things that it has not seen in its training data, which makes the whole argument of AGI spontaneously appearing given more data a bit unrealistic. It seems like we're talking about two different things. If you understand what ChatGPT is, and what it can and can't do, then there's no problem. It's a useful tool. If you think ChatGPT is AGI or it's going to replace actual programmers soon, then there's a problem. To me, it sounds like the issue is mostly unrealistic expectations.
algo_trader 2y ago
Are you using code-completion-in-the-IDE ? which tool ?

trashface 2y ago

I was using 3.5 until quite about a month ago. Now I'm using 4o. 4o is better but its not a huge difference. I was surprised as I expected it to be a huge improvement. I haven't tried the regular "4" model yet, I've heard it used to be quite good but maybe got worse. In any case, these models are good for simple stuff as far as I can tell, but can't do anything hard or off the beaten path. They can give me ideas for solving hard issues, but any code they generate is killed by hallucinations and general lack of anything resembling contextual knowledge. It can be quite obstinate too, even if I explain I'm trying to solve an unusual problem, and so need out of the box "thinking", 4o will try to force me back to some mainstream solution that I can't use. I also use Amazon's Q and that is good for simple/repetitive automation but often generates extremely wacky stuff otherwise. They are all apparently better at Python than anything else. I don't use that so maybe that's an issue.

gtirloni 2y ago

Exactly what the article is saying.

calibas 2y ago
Maybe my reading comprehension is awful, but I don't see it mentioned anywhere in the article that the "ChatGPT" from the study is the worst at coding of the 3 models that people commonly use. It seems relevant to mention that the "ChatGPT" in the article isn't the one most of us are using for coding.
- gtirloni 2y ago
  It's not that relevant because there isn't a huge inherent difference between both. It's more of an evolutionary difference based on training data and small improvemens. The article mentions the model doesn't work well on things that it has not seen in its training data, which makes the whole argument of AGI spontaneously appearing given more data a bit unrealistic. We can discuss percentages in this or that task but the article points to the root cause which I think is inescapable.
  - p1esk 2y ago
    There’s a huge difference between 3.5 and 4. 3.5 is pretty much useless. 4 is very decent, it saved me countless hours at work.
  - calibas 2y ago
    > The article mentions the model doesn't work well on things that it has not seen in its training data, which makes the whole argument of AGI spontaneously appearing given more data a bit unrealistic. It seems like we're talking about two different things. If you understand what ChatGPT is, and what it can and can't do, then there's no problem. It's a useful tool. If you think ChatGPT is AGI or it's going to replace actual programmers soon, then there's a problem. To me, it sounds like the issue is mostly unrealistic expectations.

algo_trader 2y ago

Are you using code-completion-in-the-IDE ? which tool ?