7 ms·
ChatGPT used to assure me that you can use JS dot notation to access elements in a Python dict. It also invented Redocly CLI flags that don't exist. Claude some
by dominicq 2y ago
ChatGPT used to assure me that you can use JS dot notation to access elements in a Python dict. It also invented Redocly CLI flags that don't exist. Claude sometimes invents OpenAPI specification rules. Any time I ask anything remotely niche, LLMs are often bad.
- miningape 2y agoAny time I ask anything, LLMs are often bad. inb4 you just aren't prompting correctly
- johnisgood 2y agoYeah, you probably are not prompting properly, most of my questions are answered adequately, and I have made larger projects with success, too; with both Claude and ChatGPT.
- miningape 2y agoWhat I've found is that the quality of an AI answer is inversely proportional to the knowledge of the person reading it. To an amateur it answers expertly, to an expert it answers amateurishly. So no, it's not a lack of skill in prompting: I've sat down with "prompting" "experts" and universally they overlook glaring issues when assessing the how good an answer it was. When I tell them where to press it further it breaks down with even worse gibberish.
- johnisgood 2y agoI know what I want to do and how to do it (expert), so the results are good, for me at least. Of course I have to polish it off here and there.
- shermantanktop 2y agoYou answered that more graciously than many would.
- johnisgood 2y agoThis is the reason I pay for Claude. If it weren't making me as productive as I am with it, I would not pay for it.
- myaccountonhn 2y agoI try LLMs every now and then briefly just to keep myself up to date. I’d say they’re more useful when you know what you’re doing because then you can correct the mistakes it makes. It can be good for boilerplate, refactors or remembering syntax. It’s when you don’t know what you don’t know that they can be harmful. It’s the issue with Stackoverflow but more pronounced. I don’t want to use LLMs because I think they’re unethical and I dont want to depend on a tool that requires internet, but I think if you take a disciplined approach then they can really speed up development.
- Etheryte 2y agoYeah this is so common that I've already compiled a mental list of prompts to try against any new release. I haven't seen any improvement in quite a long while now, which confirms my belief that we've more or less hit the scaling wall for what the current approaches can provide. Everything new is just a microoptimization to game one of the benchmarks, but real world use has been identical or even worse for me.
- throwaway0123_5 2y agoI think it would be an alright (potentially good) outcome if in the short-term we don't see major progress towards AGI. There are a lot of positive things we can do with current model abilities, especially as we make them cheaper, but they aren't at the point where they will be truly destructive (people using them to make bioweapons or employers using them to cause widespread unemployment across industries, or the far more speculative ASI takeover). It gives society a bit of time to catch up and move in a direction where we can better avoid or mitigate the negative consequences.
- Marazan 2y agoI would ask chatgpt every year when was the last time England had beaten Scotland at rugby. It would never get the answer right. Often transposing the scores, getting the game location wrong and on multiple occasions saying a 38-38 draw was an England win. As in literally saying " England won 38-38"
- deleted 2y ago[deleted]
- nopurpose 2y agoIt tried to convince me that it is possible to break out of outer loop in C++ with `break 'label` statement placed in nested loop. No such syntax exists.
- Yoric 2y agoSounds like it's confusing C++ and Rust. To be fair, their syntaxes are rather similar.
- int_19h 2y agoIt's confusing C++ in Java - the latter has labelled breaks but no goto.
- doubletwoyou 2y agoThe funny thing is that I think that’s a feature in D.
- rpcope1 2y agoC++ has that functionality, it's just called goto not break. That's pretty low hanging fruit for a SOTA model to fuck up though.
- TeMPOraL 2y agoDepends on prompting. I've done a lot of C++ with GPT-4, GPT-4 Turbo and Claude 3.5 Sonnet, and at no point - not once - has any of them ever hallucinated a language feature for me. Hallucinating APIs of obscure libraries? Sure[0]. Occasionally using a not-yet-available feature of the standard library? Ditto, sometimes, usually with the obvious cases[1]. Writing code in old-school C++? Happened a few times. But I have never seen it invent a language feature for C++. Might be an issue of prompting? From day one, I've been using LLMs through API and alternate frontend that lets me configure system prompts. The experience described above came from rather simple prompts[2], but I always made sure to specify the language version in the prompt. Like this one (which I grabbed from my old Emacs config): "You are a senior C++ software developer, you design and develop complex software systems using C++ programming language, and provide technical leadership to other software developers. Always double-check your replies for correctness. Unless stated otherwise, assume C++17 standard is current, and you can make use of all C++17 features. Reply concisely, and if providing code examples, wrap them in Markdown code block markers." It's as simple as it gets, and it didn't fail me. EDIT: Of course I had other, more task-specific prompts, like one for helping with GTest/GMock code; that was a tough one - for some reason LLMs loved to hallucinate on the testing framework for me. The one prompt I was happiest with was my "Emergency C++17 Build Tool Hologram" - creating an "agent" I could copy-paste output of MSBuild or GCC or GDB into, and get back a list of problems and steps to fix them, free of all the noise. On that note, I had mixed results with Aider for C++ and JavaScript, and I still feel like it's a problem with prompting - too generic and arguably poisons the context with few-shot learning examples that use code that is not in the language my project is. -- [0] - Though in LLMs' defense, the hallucinated results usually looked like what the API should have been, i.e. effectively suggesting how to properly wrap the API to make it more friendly. Which is good development practice and a useful way to go about solving problems: write the solution using non-existing helpers that are convenient for you, and afterwards, implement the helpers. [1] - Like std::map<K,T>::contains() - which is an obvious API for such container, that's typically available and named such in any other language or library, and yet only got introduced to C++ in C++20. [2] - I do them differently today, thanks to experience. For one, I never ask the model to be concise anymore - LLMs think in tokens, so I don't want to starve them. If I want a fixed format, it's better to just tell the model to put it at the end, and then skim through everything above. This is more-less the idea that "thinking models" automate these days anyway.
- jurgenaut23 2y agoWell, it makes sense. The smaller the niche, the lesser weight in the overall training loss. At the end of the day, LLMs are (literally) classifiers that assign probabilities to tokens given some previous tokens.
- svantana 2y agoYes, but o1, o3 and sonnet are not necessarily pure language models - they are opaque services. For all we know they could do syntax-aware processing or run compilers on code behind the scenes.
- skissane 2y agoThe fact they make mistakes like this implies they probably don’t, since surely steps like that would catch many of these
- bubblyworld 2y agoSOTA language models are trained with reinforcement learning now too, so the model distributions can be far removed from the underlying data distribution. Not that I think your overall point is wrong (it's obviously a very strong bias), but things are getting more complex with time.
- ljm 2y agoI once asked Perplexity (using Claude underneath) about some library functionality, which it totally fabricated. First, I asked it to show me a link to where it got that suggestion, and it scolded me saying that asking for a source is problematic and I must be trying to discredit it. Then after I responded to that it just said “this is what I thought a solution would look like because I couldn’t find what you were asking for.” The sad thing is that even though this thing is wrong and wastes my time, it is still somehow preferable to the dogshit Google Search has turned into.
- eurleif 2y agoIt baffles me how the LLM output that Google puts at the top of search results, which draws on the search results, manages to hallucinate worse than even an LLM that isn't aided by Web results. If I ask ChatGPT a relatively straightforward question, it's usually more or less accurate. But the Google Search LLM provides flagrant, laughable, and even dangerous misinformation constantly. How have they not killed it off yet?
- skissane 2y ago> But the Google Search LLM provides flagrant, laughable, and even dangerous misinformation constantly. It’s a public service: helping the average person learn that AI can’t be trusted to get its facts right
- rpcope1 2y agoHaven't you seen that Brin quote recently about how "AI" is totally the future and googlers need to work at least 60 hours a week to enhance the slop machine because reasons? Getting rid of "AI" summarization from results would look kind of like admitting defeat.
- x______________ 2y agoI concur and can easily see this occurring in several areas, for example with Linux troubleshooting. I recently found myself going down a rabbit hole of ever-increasing complicated troubleshooting steps with command that didn't exist, and after several hours of trial and error, gave up after considering the next steps brick-worthy of the system.. Dgg'ing google is still a better resort despite the drop in quality results.
- skissane 2y agoI think a lot of these issues could be avoided if, instead of just a raw model, you have an AI agent which is able to test its own answers against the actual software… it doesn’t matter as much if the model hallucinates if testing weeds out its hallucinations. Sometimes humans “hallucinate” in a similar way - their memory mixes up different programming languages and they’ll try to use syntax from one in another… but then they’ll quickly discover their mistake when the code doesn’t compile/run
- AlotOfReading 2y agoTesting is better than nothing, but still highly fallible. Take these winning examples from the underhanded C contest [0], [1], where the issues are completely innocuous mistakes that seem to work perfectly despite completely undermining the nominal purpose of the code. You can't substitute an automated process for thinking deeply and carefully about the code. [0] https://www.underhanded-c.org/#winner https://www.underhanded-c.org/#winner [1] https://www.underhanded-c.org/_page_id_17.html https://www.underhanded-c.org/_page_id_17.html
- deleted 2y ago[deleted]
- skissane 2y agoI think it is unlikely (of course not impossible) an LLM would fail in that way. The underhanded C contest is not a case of people accidentally producing highly misleading code, it is a case of very smart people going to a great amount of effort to intentionally do that. Most of the time, if your code is wrong, it doesn't work in some obvious way – it doesn't compile, it fails some obvious unit tests, etc. Code accidentally failing in some subtle way which is easy to miss is a lot rarer – not to say it never happens – but it is the exception not the rule. And it is something humans do too. So if an LLM occasionally does it, they really aren't doing worse than humans are. > You can't substitute an automated process for thinking deeply and carefully about the code. Coding LLMs work best when you have an experienced developer checking their output. The LLM focuses on the boring repetitive details leaving the developer more time to look at the big picture – and doing stuff like testing obscure scenarios the LLM probably wouldn't think of. OTOH, it isn't like all code is equal in terms of consequences if things go wrong. There's a big difference between software processing insurance claims and someone writing a computer game as a hobby. When the stakes are low, lack of experience isn't an issue. We all had to start somewhere.
- andrepd 2y agoMy rule of thumb is: is the answer to your question on the first page of google (a stackoverflow maybe, or some shit like geek4geeks)? If yes GPT can give you an answer, otherwise not.
- spookie 2y agoExactly the same experience.
- ijustlovemath 2y agoSemi related: when I'm using a dict of known keys as some sort of simple object, I almost always reach for a dataclass (with slots=True, and kw_only=True) these days. Has the added benefit that you can do stuff like foo = MyDataclass(*some_dict) and get runtime errors when the format has changed
- skerit 2y ago> Any time I ask anything remotely niche, LLMs are often bad As soon as the AI coder tools (like Aider, Cline, Claude-Coder) come into contact with a _real world_ codebase, it does not end well. So far I think they managed to fix 2 relatively easy issues on their own, but in other cases they: - Rewrote tests in a way that the broken behaviour passes the test - Fail to solve the core issue in the code, and instead patch-up the broken result (Like `if (result.includes(":") || result.includes("?")) { /* super expensive stupid fixed for a single specific case */ }` - Failed to even update the files properly, wasting a bunch of tokens