3 ms·

Yet another garbage study where “researchers” wasted months of their lives evaluating GPT 3.5 instead of paying $20 to evaluate GPT 4. It’s like drinking the c

Yet another garbage study where “researchers” wasted months of their lives evaluating GPT 3.5 instead of paying $20 to evaluate GPT 4. It’s like drinking the complementary table water at a restaurant and then leaving a one-star review saying “tasteless and flat”. I seriously can’t believe this is maybe the tenth such “paper” I’ve seen making headlines. It brings shame on not only the institution that produced it, but the journalists that failed to notice and published the clickbait headline without even reading the paper.

europeanNyan 3y ago
It seems like it's only important to bring out a paper to fill a quota somewhere. The researchers probably don't care because no one (e.g. a perspective employer) is going to be actually reading the paper, just the amount of papers they published. That's why linking to something on Arxiv trying to undeniably prove a point ("Look, researchers confirm my point!") most of the time makes me cringe. Plus, I've mostly given up on journalists. I get clickbait, that's why titles are there for. But clickbait doesn't have to equal badly researched. They could have just as well gotten clicks if they wrote an article "Researchers fail to prove ChatGPT is bad at coding because of this lazy mistake". But they mostly either have no idea of the field they are reporting about or are pushing an agenda. Research? Ain't nobody got time for that.
ChatGTP 3y ago
I'd like to see the stats on how many people use 3.5 over 4 though, I bet you that number is much higher than you think. The price of ChatGPT is being what I'm personally willing to pay for it so if I do use it, it's 3.5 for me. Maybe the logic is to study the most popular product? Would make sense to me.
- jiggawatts 3y ago
  Many more people drink tap water than eat dinner at a MICHELIN Starred restaurant, but if you're a food critic writing about the latest fancy restaurant, you had better pay for some food and write about that. This is especially painful considering the ludicrous pace of advancement in AI. You really have to aim for where the puck will be, not where the puck is. Within a year GPT4 will be out-of-date. Fundamentally, the entire point of this "research paper" is to compare ChatGPT with Stack Overflow answers, coming to the conclusion that people prefer SO. Yeah, well, meanwhile Stack Overflow usage has dropped off a cliff since ChatGPT become generally available: https://observablehq.com/@ayhanfuat/the-fall-of-stack-overflow https://observablehq.com/@ayhanfuat/the-fall-of-stack-overfl... I certainly prefer to ask ChatGPT basic coding questions because I get an answer immediately with no argument.
  - pests 3y ago
    But if all that existed at the restaurant was water? The food or better drinks werent released yet.
  - ChatGTP 3y ago
    This analogy should be in a gallery, good day to you sir.
- monooso 3y ago
  If that were the case, I would expect to see an explicit reference to the CGPT version in the introduction, and an explanation as to why the authors chose it for the study. Instead, the only version information appears to be buried in section 3.1.2 ("ChatGPT 3.5 Turbo API is used").
  - pests 3y ago
    They used 3.5 because that is all that was available at the time.
HWR_14 3y ago
It takes time to create and publish a study.
- tasogare 3y ago
  [dead]
nonameiguess 3y ago
Jesus Christ, dude. To know they used 3.5, you must have read the methods section, in which case you can also see they conducted the data collection in March 2023, the same month GPT-4 was released. Bad timing, I guess, but your rant reduces to telling researchers to just scrap whatever work they did and not bother if an update happens to be released to whatever they were studying before they publish.
deleted 3y ago
[deleted]
raylad 3y ago
GPT-4 has gotten so much worse over the last few months, and GPT-3.5 has gotten better. I still think GPT-4 has an edge but its frustratingly dumb and loses context so often that I'm not sure how much of an edge it really has any more.

europeanNyan 3y ago

It seems like it's only important to bring out a paper to fill a quota somewhere. The researchers probably don't care because no one (e.g. a perspective employer) is going to be actually reading the paper, just the amount of papers they published. That's why linking to something on Arxiv trying to undeniably prove a point ("Look, researchers confirm my point!") most of the time makes me cringe. Plus, I've mostly given up on journalists. I get clickbait, that's why titles are there for. But clickbait doesn't have to equal badly researched. They could have just as well gotten clicks if they wrote an article "Researchers fail to prove ChatGPT is bad at coding because of this lazy mistake". But they mostly either have no idea of the field they are reporting about or are pushing an agenda. Research? Ain't nobody got time for that.

ChatGTP 3y ago

I'd like to see the stats on how many people use 3.5 over 4 though, I bet you that number is much higher than you think. The price of ChatGPT is being what I'm personally willing to pay for it so if I do use it, it's 3.5 for me. Maybe the logic is to study the most popular product? Would make sense to me.

jiggawatts 3y ago
Many more people drink tap water than eat dinner at a MICHELIN Starred restaurant, but if you're a food critic writing about the latest fancy restaurant, you had better pay for some food and write about that. This is especially painful considering the ludicrous pace of advancement in AI. You really have to aim for where the puck will be, not where the puck is. Within a year GPT4 will be out-of-date. Fundamentally, the entire point of this "research paper" is to compare ChatGPT with Stack Overflow answers, coming to the conclusion that people prefer SO. Yeah, well, meanwhile Stack Overflow usage has dropped off a cliff since ChatGPT become generally available: https://observablehq.com/@ayhanfuat/the-fall-of-stack-overflow https://observablehq.com/@ayhanfuat/the-fall-of-stack-overfl... I certainly prefer to ask ChatGPT basic coding questions because I get an answer immediately with no argument.
- pests 3y ago
  But if all that existed at the restaurant was water? The food or better drinks werent released yet.
- ChatGTP 3y ago
  This analogy should be in a gallery, good day to you sir.
monooso 3y ago
If that were the case, I would expect to see an explicit reference to the CGPT version in the introduction, and an explanation as to why the authors chose it for the study. Instead, the only version information appears to be buried in section 3.1.2 ("ChatGPT 3.5 Turbo API is used").
- pests 3y ago
  They used 3.5 because that is all that was available at the time.

HWR_14 3y ago

It takes time to create and publish a study.

tasogare 3y ago
[dead]

nonameiguess 3y ago

Jesus Christ, dude. To know they used 3.5, you must have read the methods section, in which case you can also see they conducted the data collection in March 2023, the same month GPT-4 was released. Bad timing, I guess, but your rant reduces to telling researchers to just scrap whatever work they did and not bother if an update happens to be released to whatever they were studying before they publish.

deleted 3y ago

[deleted]

raylad 3y ago

GPT-4 has gotten so much worse over the last few months, and GPT-3.5 has gotten better. I still think GPT-4 has an edge but its frustratingly dumb and loses context so often that I'm not sure how much of an edge it really has any more.