5 ms·
ChatGPT's odds of getting code questions correct are worse than a coin flip
- tkgally 3y agoFrom the paper [1]: > For each of the 517 SO questions, the first two authors manually used the SO question's title, body, and tags to form one question prompt and fed that to the Chat Interface of ChatGPT. The generated answers by ChatGPT are then stored in CSV files. For the additional 2000 SO questions, ChatGPT 3.5 Turbo API is used. I would be interested in seeing the results for GPT-4. For a paper released in August 2023, it’s strange that GPT-4 isn’t even mentioned. [1] https://arxiv.org/abs/2308.02312 https://arxiv.org/abs/2308.02312
- tanseydavid 3y ago>> it’s strange that GPT-4 isn’t even mentioned Except for when you look at the results. ;)
- sam0x17 3y agoIsn't this a bit silly since chat gpt _was trained_ on SO questions, or are these all questions asked since the training cutoff...
- mechanical_bear 3y agoThis has definitely not been my experience, nor my colleagues’ experiences.
- tete 3y agoIt also says that programmer essentially prefer wrong answers.
- interstice 3y agoI prefer easy to fix answers
- oceanplexian 3y agoI think 30% is still impressive. Imagine if we treated search engines this way.. “30% of the time someone googled a software problem and clicked the first link it was the wrong answer”. I don’t think anyone would be shocked by such a groundbreaking finding.
- vba616 3y agoWell...as it happens, I have a recent example of Google getting something trivial wrong and ChatGPT getting it right. But oddly that doesn't change my opinion of ChatGPT. The only reason I know that ChatGPT was right is because I blindly followed the first Googled suggestion. If I started with ChatGPT, then I would have to verify the answer somehow. If I use Google, I should verify the answer. But ChatGPT cannot do that job. Wikipedia is starting to feel as useless to me as ChatGPT. I don't know if anyone is actually replacing the articles with AI generated fiction at scale yet, or whether it's still an artisan, human-centered craft but the more I click on references, the more I find they are uncheckable, even if it's ambiguous whether they are fraudulent. People write rants on a topic and cite a single book without page numbers five times. Or, of course, dead links to websites without archive.org.
- lcuff 3y agoTwo thoughts: (1) ChatGPT answers questions immediately (I'm paying for premium service so I don't get shut out), which is a big win over Stack Overflow where I recently waited for two or three days before getting a partially-correct answer. (2) I often am using ChatGPT (in my Ruby world) to find gems (ruby packages) or language features that address my issue. Even if the whole answer is wrong, it often provides me with very helpful tidbits of information at a huge time saving.
- sam0x17 3y agoOn more than one occasion, the hallucinated solution has had better dev ux than the real solution and I've convinced whatever package to add support for a similar syntax.
- 1B05H1N 3y agoIt really helps me get a better idea of what I should be doing. Like any answer from anyone, you have to understand and peer review. I usually end up rewriting ~50% of anything I get out of chatgpt.
- sam0x17 3y agoIf you told me in 2005 there would be something non-human that can get a coding question right with near-coin-flip accuracy I'd say you're crazy.
- jychang 3y agoWouldn’t a coin be able to do that pretty well in 2005?
- flangola7 3y agoI would expect a coin to do it 0% of the time.
- sam0x17 3y agoUnless the question was "write a program that simulates a coin flip"
- sam0x17 3y agoTrue/false answers to programming questions don't build apps.
- captainbland 3y agoOnly if it was choosing between one of two answers to begin with. The analogy is a bit weird here because there's far more than two potential answers it could give but it still manages to pick the right one almost half the time.
- vba616 3y agoThe free version of ChatGPT, which is now calling itself 3.5, has a seemingly consistent type of answer whenever I ask it how to code something I don't know how to do. It outputs some boilerplate code, and inserts a method with comments saying "insert logic here" - entirely sidestepping the request. You can say that's remarkably human-like, but if it was a human cow-orker, you would not keep them on the payroll. It is a cheat, like ELIZA, like SHRDLU, like a useless employee. I asked ChatGPT-3.5 for the usual rejoinder to complainers: "If you're looking for an enhanced experience, you might want to consider trying out the paid version called ChatGPT-4. It offers even more capabilities and improvements. Feel free to explore the options available to find the best fit for your needs!""
- Kerb_ 3y agoChatGPT has helped me install plugins on my Minecraft server and given me configuring commands with near 100% accuracy, and the only mistake it made was using an old command that is one letter different now. This was ChatGPT-4, running the Code Interpreter.
- happytiger 3y agoThis is really more about social factors anyways… what’s the study actually trying to hypothesize and test?
- RecycledEle 3y agoI can remember a few time when a computer program getting 48% of coding questions right was science fiction.