6 ms·
ChatGPT Isn't as Good at Coding as We Thought
- typon 3y ago52% correct seems awfully high - I am guessing it's inflated because of the easiness of the questions on StackOverflow. In my experience, at least in my domain, I will be lucky it it can generate an answer that is more than 5 lines of code without a single error. Still incredibly useful - just have to handhold it and watch it very carefully when going through a solution.
- fragmede 3y agoGPT4 can run code, so you can ask it to run the code it generated and iterate until it works. Does that make it one shot? Either way, its limitations are becoming apparent, but it's still quite the leap forwards for writing code.
- Our_Benefactors 3y agoI don’t think it runs code, it’s an illusion and it’s actually inferring what the output would be but not actually executing any statements
- jiggawatts 3y agoYet another garbage study where “researchers” wasted months of their lives evaluating GPT 3.5 instead of paying $20 to evaluate GPT 4. It’s like drinking the complementary table water at a restaurant and then leaving a one-star review saying “tasteless and flat”. I seriously can’t believe this is maybe the tenth such “paper” I’ve seen making headlines. It brings shame on not only the institution that produced it, but the journalists that failed to notice and published the clickbait headline without even reading the paper.
- europeanNyan 3y agoIt seems like it's only important to bring out a paper to fill a quota somewhere. The researchers probably don't care because no one (e.g. a perspective employer) is going to be actually reading the paper, just the amount of papers they published. That's why linking to something on Arxiv trying to undeniably prove a point ("Look, researchers confirm my point!") most of the time makes me cringe. Plus, I've mostly given up on journalists. I get clickbait, that's why titles are there for. But clickbait doesn't have to equal badly researched. They could have just as well gotten clicks if they wrote an article "Researchers fail to prove ChatGPT is bad at coding because of this lazy mistake". But they mostly either have no idea of the field they are reporting about or are pushing an agenda. Research? Ain't nobody got time for that.
- ChatGTP 3y agoI'd like to see the stats on how many people use 3.5 over 4 though, I bet you that number is much higher than you think. The price of ChatGPT is being what I'm personally willing to pay for it so if I do use it, it's 3.5 for me. Maybe the logic is to study the most popular product? Would make sense to me.
- jiggawatts 3y agoMany more people drink tap water than eat dinner at a MICHELIN Starred restaurant, but if you're a food critic writing about the latest fancy restaurant, you had better pay for some food and write about that. This is especially painful considering the ludicrous pace of advancement in AI. You really have to aim for where the puck will be, not where the puck is. Within a year GPT4 will be out-of-date. Fundamentally, the entire point of this "research paper" is to compare ChatGPT with Stack Overflow answers, coming to the conclusion that people prefer SO. Yeah, well, meanwhile Stack Overflow usage has dropped off a cliff since ChatGPT become generally available: https://observablehq.com/@ayhanfuat/the-fall-of-stack-overflow https://observablehq.com/@ayhanfuat/the-fall-of-stack-overfl... I certainly prefer to ask ChatGPT basic coding questions because I get an answer immediately with no argument.
- monooso 3y agoIf that were the case, I would expect to see an explicit reference to the CGPT version in the introduction, and an explanation as to why the authors chose it for the study. Instead, the only version information appears to be buried in section 3.1.2 ("ChatGPT 3.5 Turbo API is used").
- HWR_14 3y agoIt takes time to create and publish a study.
- tasogare 3y ago[dead]
- nonameiguess 3y agoJesus Christ, dude. To know they used 3.5, you must have read the methods section, in which case you can also see they conducted the data collection in March 2023, the same month GPT-4 was released. Bad timing, I guess, but your rant reduces to telling researchers to just scrap whatever work they did and not bother if an update happens to be released to whatever they were studying before they publish.
- deleted 3y ago[deleted]
- raylad 3y agoGPT-4 has gotten so much worse over the last few months, and GPT-3.5 has gotten better. I still think GPT-4 has an edge but its frustratingly dumb and loses context so often that I'm not sure how much of an edge it really has any more.
- jhoelzel 3y agoAnd it does not need to be. Personally i do not use it to generate novel code that will solve my problems, but when i point it at a github repo with the github plugin enabled, it will answer the questions i have about the code pretty much on the nose. its not your new AI Friend steve, its a LLM. And one with a knowledge cutoff in 2021 too
- PoignardAzur 3y ago> but when i point it at a github repo with the github plugin enabled, it will answer the questions i have about the code pretty much on the nose. Holy crap, you can do that? I need to learn about ChatGPT plugins.
- jhoelzel 3y agoyes its in the "plugin store" and named "ask the code"
- mitchellpkt 3y agoNote that at time of writing the “ask the code” plug-in does well with describing repo structure, but hallucinates file contents (I tested it on one of my own repos)
- jhoelzel 3y agomaybe i have random successes or my questions are simpler than yours, but it works for me: https://chat.openai.com/share/fbd14b6c-0ba0-473f-9e94-db97c69fc040 https://chat.openai.com/share/fbd14b6c-0ba0-473f-9e94-db97c6...
- namaria 3y agoI don't understand why generating language, or code, would be a good thing. Language, and code, is a tool for communication. If you treat it as fodder or merchandise, you loose the main use: transport meaning between different people. I fail to understand why we would want automatically generated language at all, except for the fact that you can sometimes make money off of it. Considering we're where we are as a civilization because people could make money off of poisoning the Earth and causing catastrophic ecosystem and climate change for profit, I think we should examine our motives more closely.
- omarfarooq 3y agoWould you be so kind as to demonstrate how generated language and code does not transport meaning between different people.
- SanderNL 3y agoI'd say the symbols themselves are meaningless without their ground truths, whatever that is. We can call this "experience" or "thinking". The results of these opaque processes are somehow amenable to be encoded into symbols which are themselves deprived of any significance. Which I guess makes them usable as such in the first place. Extreme example: if a random algorithm picked out a few words from a dictionary and somehow cobbled a sentence together using those words, would you say "meaning" has been transported? The "generation" of these symbols should be the result of a process equivalent to whatever we are doing when we "experience" or "cognate" or whatever, otherwise the results will be only very superficially useful.
- bheadmaster 3y ago> I'd say the symbols themselves are meaningless without their ground truths, whatever that is. We can call this "experience" or "thinking". So you're basically dismissing the whole field of genetic algorithms?
- SanderNL 3y ago
- soultrees 3y agoLately it’s even worse. And any AI agent that relies on open AI has gone down in quality too (phind). I find it’s more of a struggle to have the AI write code in the first pass, even when I explicitly say something like “I want you to write a function that does this {action}” and on average it takes about 3 messages back and forth before it actually writes it. Whereas before it would start writing it immediately, even though it was almost always wrong, I could tweak the result after. Now though, the stupid context memory is filled up with useless junk words not related to the actual function I want to write, since we spent multiple messages going back and forth.
- tbiehn 3y agoA tip for your context window problem: prefer ‘editing’ a previous message to add whatever clarification is required - rather than having a ‘chat.’ Definitely helps keep it on the rails.
- robbywashere_ 3y agoFair. But garbage in garbage out. The prompts are in human language. Of which the models use to derive output. Human language is obviously not a rigid mode of instruction. Therefore; the models behave exactly as you'd guess. The output is not purely predictable nor rigid. This is by design. Providing a prompt, and/or know how to properly prompt is key.
- guluarte 3y agoLLM are autocomplete with context, I don't know what they were expecting
- tbiehn 3y agoThis article shines a light on exactly why LLMs /arent/ just autocomplete with context. There’s real computation going on in there. https://arstechnica.com/science/2023/07/a-jargon-free-explanation-of-how-ai-large-language-models-work/ https://arstechnica.com/science/2023/07/a-jargon-free-explan...