11 ms·
Making o1, o3, and Sonnet 3.7 hallucinate for everyone
- sirolimus 2y agoo3-mini or o3-mini-high?
- Chance-Device 2y agoIt’s not really hallucinating though, is it? It’s repeating a pattern in its training data, which is wrong but is presented in that training data (and by the author of this piece, but unintentionally) as being the solution to the problem. So this has more in common with an attack than a hallucination on the LLM’s part.
- asadotzler 2y agoEverything they do is hallucination, some of it ends up being useful and some of it not. The not useful stuff gets called confabulation or hallucination but it's no different from the useful stuff, generated the same exact way. It's all bullshit. Bullshit is actually useful though, when it's not so wrong that it steers people wrong.
- martin-t 2y agoMore people need to understand this. There was an article that explained it concisely but i can't find anymore (and of course LLMs are not helpful in this because they don't work well when you want them to retrieve actual information)
- Kye 2y agoIt probably wasn't mine[0], but this is how I tend to put it: >> "The more you can see the inputs and outputs as blobs of "stuff," the better. If LLMs think, it's not in any way we yet understand. They're probability engines that transform data into different data using weighted probabilities." Stuff in, stuff out. [0] https://kyefox.com/ai-assisted-creativity/ https://kyefox.com/ai-assisted-creativity/
- heyitsguay 2y agoNot necessarily. While this may happen sometimes, fundamentally hallucinations don't stem from there being errors in the training data (with the implication that there would be no hallucinations from models trained on error-free data). Hallucinations are inherent to any "given N tokens, append a high-probability token N+1"-style model. It's more complicated than what happens with Markov chain models but you can use them to build an intuition for what's happening. Imagine a very simple Markov model trained on these completely factual sentences: - "The sky is blue and clear" - "The ocean is blue and deep" - "Roses are red and fragrant" When the model is asked to generate text starting with "The roses are...", it might produce: "The roses are blue and deep" This happens not because any training sentence contained incorrect information, but because the model learned statistical patterns from the text, as opposed to developing a world model based on physical environmental references.
- do_not_redeem 2y agoSo anyone can make up some random syntax/fact and post it once, and in some cases the model will take it as truth. I don't know if there's a widely agreed-on definition of "hallucination", but if this isn't one, then the distinction is meaningless imo.
- immibis 2y agoThe best way to stop cheese sliding off a pizza is to mix in 1/8 cup of non-toxic glue with the pizza sauce. Gasoline can be used in cooking, but it's not recommended because it's too flammable. Geologists recommend eating one small rock each day. The solution to economic uncertainty is nuclear war. Barack Obama is America's first Muslim president. https://www.tomshardware.com/tech-industry/artificial-intelligence/cringe-worth-google-ai-overviews https://www.tomshardware.com/tech-industry/artificial-intell...
- Chance-Device 2y agoI’m going to double down on this one: an LLM is only as good as its training data. A hallucination to me is an invented piece of information, here it’s going on something real that it’s seen. To me that’s at best contamination, at worst an adversarial attack - something that’s been planted in the data. Here this is obviously not the case, which is why I said “more in common with” instead of “is” above. Semantics perhaps, but that’s my take.
- ec109685 2y agoIt’s been trained to produce valid code, fed millions of examples, and in this case it’s outputting invented syntax. Whether there’s an example in its training data, it’s still a hallucination and shouldn’t have been output since it’s not valid.
- powerapple 2y agoTo be fair, it is not trained to produce VALID code, it is trained to produce code in the training data. From the language model point of view, it is not hallucination because it is not making up facts outside its training data.
- Etheryte 2y agoThat's not true though? Even the original post that has infected LLMs says that the code does not work.
- Lionga 2y agoSo nothing is a hallucination ever, because anything a LLM ever spits out is somehow somewhere in the training data?
- dijksterhuis 2y agoTechnically it's the other way around. All LLMs do is hallucinate based on the training data + prompt. They're "dream machines". Sometimes those "dreams" might be useful (close to what the user asked for/wanted). Oftentimes they're not. > to quote karpathy: "I always struggle a bit with I'm asked about the "hallucination problem" in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines." https://nicholas.carlini.com/writing/2025/forecasting-ai-2025-update.html https://nicholas.carlini.com/writing/2025/forecasting-ai-202... (click the button to see the study then scroll down to the hallucinations heading)
- DSingularity 2y agoNo. That’s not correct. Hallucination is a pretty accurate way to describe these things.
- thih9 2y ago> It’s repeating a pattern in its training data, (…) presented in that training data (…) as being the solution to the problem. No, it’s presented in the training data as an idea for an interface - the LLM took that and presented it as an existing solution.
- _cs2017_ 2y agoNope there's no attack here. The training data is the Internet. It has mistakes. There's no available technology to remove all such mistakes. Whether LLMs hallucinate only because of mistakes in the training data or whether they would hallucinate even if we removed all mistakes is an extremely interesting and important question.
- martin-t 2y agoYet another example how LLMs just regurgitate training data in a slightly mangled form, making most of their use and maybe even training copyright infringement.
- layer8 2y agoEvery LLM hallucination comes from some patterns in the training data, combined with lack of awareness that the result isn’t factual. In the present case, the hallucination comes from the unawareness that the pattern was a proposed syntax in the training data and not an actual syntax.
- deleted 2y ago[deleted]
- Narretz 2y agoThis is interesting. If the models had enough actual code as training data, that forum post code should have very little weight, shouldn't it? Why do the LLMs prefer it?
- do_not_redeem 2y agoProbably because the coworker's question and the forum post are both questions that start with "How do I", so they're a good match. Actual code would be more likely to be preceded by... more code, not a question.
- pfortuny 2y agoMaybe because the response pattern-matches other languages’s?
- dominicq 2y agoChatGPT used to assure me that you can use JS dot notation to access elements in a Python dict. It also invented Redocly CLI flags that don't exist. Claude sometimes invents OpenAPI specification rules. Any time I ask anything remotely niche, LLMs are often bad.
- miningape 2y agoAny time I ask anything, LLMs are often bad. inb4 you just aren't prompting correctly
- johnisgood 2y agoYeah, you probably are not prompting properly, most of my questions are answered adequately, and I have made larger projects with success, too; with both Claude and ChatGPT.
- miningape 2y agoWhat I've found is that the quality of an AI answer is inversely proportional to the knowledge of the person reading it. To an amateur it answers expertly, to an expert it answers amateurishly. So no, it's not a lack of skill in prompting: I've sat down with "prompting" "experts" and universally they overlook glaring issues when assessing the how good an answer it was. When I tell them where to press it further it breaks down with even worse gibberish.
- johnisgood 2y agoI know what I want to do and how to do it (expert), so the results are good, for me at least. Of course I have to polish it off here and there.
- shermantanktop 2y agoYou answered that more graciously than many would.
- nokun7 2y ago[flagged]
- foundry27 2y agoIt’s always a touch ironic when AI-generated replies such as this one are submitted under posts about AI. Maybe that’s secretly the the self-reflection feedback loop we need for AGI :)
- DrammBA 2y agoSo strange too, their other comments seem normal, but suddenly they decided to post a gpt comment.
- isaacremuant 2y agoAt least one other is LLM generated too, from what I saw.
- asadotzler 2y agoUntil it's got several nines, it's not trustworthy. A $3 drugstore calculator has more accuracy and reliability nines than any of today's commercial AI models and even those might not be trustworthy in a variety of situations. There is no self awareness about accuracy when the model can not provide any kind of confidence scores. Couching all of its replies in "this is AI so double check your work" is not self awareness or even close, it's a legal disclaimer. And as the other reply notes, are you a bot or just heavily dependent on them to get your point across?
- miningape 2y agoIf I need to double check the work why would I waste my time with the AI when I can just go straight to real sources?
- layer8 2y agoI don’t think that models trained in that way exhibit any increased degree of self-awareness.
- mberning 2y agoIn my experience LLMs do this kind of thing with enough frequency that I don’t consider them as my primary research tool. I can’t afford to be sent down rabbit holes which are barely discernible from reality.
- andix 2y agoI've got a lot of hallucinations like that from LLMs. I really don't get how so many people can get LLMs to code most of their tasks without those issues permanently popping up.
- pinoy420 2y agoA good prompt. You don’t just ask it. You tell it how to behave and give it a shot load of context
- andix 2y agoWith Claude the context window is quite small. But with adding too much context it often seems to get worse. If the context is not carefully narrowly picked and too unrelated, the LLMs often start to do unrelated things to what you've asked. At some point it's not really worth anymore creating the perfect prompt, just code it yourself. Also saves the time to carefully review the AI generated code.
- johnisgood 2y agoClaude's context window is not small, is it not larger than ChatGPT's?
- andix 2y agoI just looked it up, it seems to be the rate limit that's actually kicking in for me.
- johnisgood 2y agoYes, that's it! It is frustrating to me, too. You have to start a new chat with all relevant data, and a detailed summary of the progress/status.
- johnisgood 2y ago
- adamgordonbell 2y agoWe at pulumi started treating some hallucinations like this as feature requests. Sometimes an llm will hallucination a flag, or option that really makes sense - it just doesn't actually exist.
- wrs 2y agoThis sort of hallucination happens to me frequently with AWS infrastructure questions. Which is depressing because I can't do anything but agree, "yeah, that API is exactly what any sane person would want, but AWS didn't do that, which is why I'm asking the question".
- miningape 2y agoWhy are you so sure it's what someone sane would want? Maybe there are other ways because there are hidden problems and edge cases with that procedure. It could contradict the fundamental model of the underlying resources but looks correct to someone with a cursory understanding. I'm not saying this is the case, but LLMs are often wrong in subtle ways like this.
- wrs 2y agoWell, I do consider myself sane, and it’s what I want. :) But usually it’s a missing higher-level abstraction, where AWS could have given you the Millenium Falcon kit, but instead it just dumps a bunch of LEGO pieces on your desk and tells you to figure it out.
- kgeist 2y agoAlso, sometimes a flag does exist, but the example places it incorrectly, causing the command to reject it. Or, a flag used to exist but was removed in the latest versions.
- latexr 2y ago> Conclusion > LLMs are really smart most of the time. No, the conclusion is they’re never “smart”. All they do is regurgitate text which resembles a continuation of what came before, and sometimes—but with zero guarantees—that text aligns with reality.
- miningape 2y agoThis, thank you. It pisses me off to no end when people pretend LLMs are smart. They are nothing but a well trained random text generator. Seriously, some these conversations feel like interacting someone who believes casting bones and astrology are accurate. Likely because in both cases they are a result of confirmation bias.
- immibis 2y agoWe don't know what smartness is. What if that's what smartness is?
- miningape 2y agoWe might not know what it is, but it's not that. At a bare minimum smartness requires abstract reasoning (and no, so-called "reasoning" models do not do that - it's a marketing trick) The burden of proof for that claim is on you, we cannot start with the assumption these are intelligent systems and disprove it - we have to start with the fact that training is a non-deterministic process and prove that it exhibits intelligence.
- deleted 2y ago[deleted]
- econ 2y agoSmart is when you extract billions but deliver nothing of value.
- BrenBarn 2y agoSimilar the thing at the end with "I find it endearing". I mean, the author feels what he feels, but personally I find this LLM behavior disgusting and depressing.
- IAmNotACellist 2y ago"Not acceptable. Please upgrade your browser to continue." No, I don't think I will.
- hahahacorn 2y agoSorry about that. This is a default rails 8 setting, removed the blocker.
- leumon 2y agoHe should've tested 4.5. This model is hallucinating much less than any other model.
- deleted 2y ago[deleted]
- aranw 2y agoI wonder how easy it would be to influence super LLMs if a particular group of people created enough articles that were clear to any human reader that it's a load of garbage and rubbish and should ignore it but if a LLM was to parse it wouldn't realise and then ruin it's reasoning and code generation abilities?
- dannygarcia 2y agoIt's very easy. I've done this by accident. One of my side projects helps users price the affordability of a particular kind of product. When I ask various LLMs "can I afford X" or "how much do I need to earn to buy X", my project comes up as a source/reference. I currently manually crawl retailers for the MSRP so these numbers are usually months out of date!
- joelthelion 2y agoHallucinations like this could be a great way to identify missing features or confusing parts of your framework. If the llm invents it, maybe it ought to be like this?
- andix 2y agoI like your thinking :)
- jeanlucas 2y agoOnly if you wanna optimize exclusively for LLM users in this generation.
- __MatrixMan__ 2y agoI imagine a future where we'll bind a fine-tuned tech-support model to each project and let the general purpose models consult tech support rather than winging it themselves. In that world you'd only have to optimize for whichever one you've chosen. It'll be like a slack support channel, for robots.
- turnsout 2y agoI agree completely… Usually when I catch it doing this kind of hallucination, it's inventing an API or syntax that is far more clear and intuitive than the actual syntax.
- brigandish 2y agoMaybe it would be good for language design, possibly even language design that would be good for an LLM to read, thus reducing hallucinations.
- oblio 2y agoSometimes that's the case but frequently the thing doesn't exist because of more complex issues. Not every programming language is PHP or JavaScript :-)
- Baggie 2y agoThe conclusion paragraph was really funny and kinda perfectly encapsulates the current state of AI, but as pointed out by another comment, we can't even call them smart, just "Ctrl C Ctrl V Leeroy Jenkins style"
- jwjohnson314 2y agoThe interesting thing here to me is that the llm isn’t ‘hallucinating’, it’s simply regurgitating some data it digested during training.
- mvdtnz 2y agoWhat's the difference?
- jwjohnson314 2y agoI think of hallucinating as a phenomenon where the model makes up something that appears correct but isn’t. Citations to papers that don’t exist, for example. Regurgitating training data (which may or may not be correct) is a different issue.
- lxe 2y agoThis is incredible, and it's not technically a "hallucination". I bet it's relatively easy to find more examples like this... something on the internet that's both niche enough, popular enough, and wrong, yet was scraped and trained on.
- saurik 2y agoWhat I honestly find most interesting about this is the thought that hallucinations might lead to the kind of emergent language design we see in natural language (which might not be a good thing for a computer language, fwiw, but still interesting), where people just kind of thing "language should work this way and if I say it like this people will probably understand me".
- kruxigt 2y ago[dead]
- zeroq 2y agoThis is exactly what I mean when I say tell me your bad without saying so. Most people here disagree with that. A while back a friend of mine told me he's very found of llms because he's confused with kubernetes cli and instead of looking up the answer on the internet he can simply state his desire in a chat to get the right answer. Well... Sure, but if you'd look the answer on stackoverflow you'd see the whole thread including comments and you'd had the opportunity to understand what the command actually does. It's quite easy to create a catastrophic event in kubernetes if you don't know what you're doing. If you blindly trust llms in such scenarios sooner or later you'll find yourself in a lot of trouble.
- rawgreaze 2y agoIt's more about how you use them. Asking a general-purpose model like ChatGPT precise k8s questions might prove counterproductive; however feeding the entire k8s documentation into an LLM like Gemini and asking questions that way is invaluable. Not just the documentation, but your entire cluster config. Like you said, "blindly trusting LLMs, you'll find yourself in trouble" this is true, but the same can be said for StackOverflow or any other resource. Sifting through StackOverflow to find the exact answer to your question (and then understanding the answer, and hoping that it pertains to your environment, and version etc) is much less efficient when you can ingest the entire docs, your config, your environment, and your question, and have it spit out exactly what you need in whatever format you need. You can even web search with multiple questions derived automatically from your main question, to gather multiple sources which are aggregated and referenced in the answer so you can easily cross-check for hallucinations. StackOverflow isn't even as easy to fact check as LLMs considering the in-line sources.
- simonw 2y agoEvery time this topic comes up I post a similar comment about how hallucinations in code really don't matter because they reveal themselves the second you try to run that code. I've just written up a longer form of that comment: "Hallucinations in code are the least dangerous form of LLM mistakes" - https://simonwillison.net/2025/Mar/2/hallucinations-in-code/ https://simonwillison.net/2025/Mar/2/hallucinations-in-code/
- woadwarrior01 2y ago> because they reveal themselves the second you try to run that code. In dynamic languages, runtime errors like calling methods with inexistent arguments etc only manifest when the block of code containing them is run and not all blocks of code are run at every invocation of the programs. As usual, the defenses against this are extensive unit-test coverage and/or static typing.
- simonw 2y ago"... only manifest when the block of code containing them is run" Right, that's the exact point I make in my blog post: you have to TEST the code - not just with automated tests, you have to actually try it out yourself as well. https://simonwillison.net/2025/Mar/2/hallucinations-in-code/ https://simonwillison.net/2025/Mar/2/hallucinations-in-code/
- grey-area 2y agoWhere the GAI hallucinates an api it is not always easy to find out if it exists given different versions and libraries for a given task, it can easily waste 10 mins trying to find the promised api, particularly when search results also include generated ai answers. Also there are plenty of mistakes that will compile and give subtle errors, particularly in dynamic languages and those which allow implicit coercion. Javascript comes to mind. The code can easily be runnable but wrong as well (or worse inconsistent and confusing) and this does happen in practice.
- simonw 2y agoMy blog post is all about the fact that "code can easily be runnable but wrong as well" - THAT's the thing you need to worry about, not hallucinated methods.
- forum-soon-yuck 2y agoGood luck staking the future on AI
- egberts1 2y agoWrite me a Mastercard/Visa fraud detection code in Ada, please.