4 ms·

>Thus, in this study, we take the state-of-the-art ChatGPT (the default version of GPT-3.5), the recent popular product, as the representative of LLMs for evalu

>Thus, in this study, we take the state-of-the-art ChatGPT (the default version of GPT-3.5), the recent popular product, as the representative of LLMs for evaluation. Are they really publishing a paper based on GPT3.5 in July 2024? I am not sure these results are relevant in any way today. Edit: Just for reference. The best model for coding today (according to most benchmarks) is Claude-3.5-Sonnet which is freely accessible. Also GPT-4o is freely accessible and is still vastly better than GPT-3.5. The lm sys arena coding leaderboard (https://chat.lmsys.org/?leaderboard https://chat.lmsys.org/?leaderboard) lists sonnet-3.5 and gpt-4o jointly on #1 and GPT-3.5-Turbo on #35. You can freely download and run LLMs locally on your machine that are significantly better than GPT-3.5, for example Mistral Codestral. There is really no reason to accept any results on GPT3.5 for relevant today. This is as if you were complaining that a computer from the 00ies is not running <recent operating system> well.

deskr 2y ago
Scientific studies take a while from inception to publishing.
- cpldcpu 2y ago
  That is why the ML/AI community is usually shortcutting this process by publishing preprints on Arxiv. A publication based on an LLM that was state-of-the art only until March of 2023 cannot be justified by long review times. Edit: To be fair, it seems their preprint was first submitted in August 2023 and the IEEE article that is based on the paper was a bit slow...
  - p1esk 2y ago
    4 months is plenty of time to rerun experiments using GPT4.
  - deskr 2y ago
    But this isn't a preprint and this isn't on Arxiv.
jazzyjackson 2y ago
Yes, the version that the vast majority of people are using since it is free when you go to chatgpt.com
- GaggiX 2y ago
  GPT-4o is the free version you will mostly use on chatgpt.com
  - jazzyjackson 2y ago
    Huh, I was fooled by marketing copy, the upgrade button says my free account has "Access to GPT-3.5, Limited Access to GPT-4o" but you're right, 4o is the default
hypeatei 2y ago
That is still the default, non-paid version of ChatGPT today, no?
- GaggiX 2y ago
  It's not, the default one is GPT-4o.
  - hypeatei 2y ago
    GPT-4o is "limited" for non-paid accounts, though. So you're at the mercy of OpenAI if you get it or not.

deskr 2y ago

Scientific studies take a while from inception to publishing.

cpldcpu 2y ago
That is why the ML/AI community is usually shortcutting this process by publishing preprints on Arxiv. A publication based on an LLM that was state-of-the art only until March of 2023 cannot be justified by long review times. Edit: To be fair, it seems their preprint was first submitted in August 2023 and the IEEE article that is based on the paper was a bit slow...
- p1esk 2y ago
  4 months is plenty of time to rerun experiments using GPT4.
- deskr 2y ago
  But this isn't a preprint and this isn't on Arxiv.

jazzyjackson 2y ago

Yes, the version that the vast majority of people are using since it is free when you go to chatgpt.com

GaggiX 2y ago
GPT-4o is the free version you will mostly use on chatgpt.com
- jazzyjackson 2y ago
  Huh, I was fooled by marketing copy, the upgrade button says my free account has "Access to GPT-3.5, Limited Access to GPT-4o" but you're right, 4o is the default

hypeatei 2y ago

That is still the default, non-paid version of ChatGPT today, no?

GaggiX 2y ago
It's not, the default one is GPT-4o.
- hypeatei 2y ago
  GPT-4o is "limited" for non-paid accounts, though. So you're at the mercy of OpenAI if you get it or not.