4 ms·
> hallucinated or unfaithful conclusions Disclaimer: I’m no expert. An anecdotal example: I asked the reasoning LLM a question, and it laid out the correct an
by Lapel2742 1y ago
> hallucinated or unfaithful conclusions
Disclaimer: I’m no expert. An anecdotal example: I asked the reasoning LLM a question, and it laid out the correct answer in its thinking step, only to stop thinking and confidently give the wrong answer. That moment led me to conclude that when LLM evangelists talk about reasoning and thinking, they are essentially bullshitting.
- simonw 1y agoWhy did that experience lead you to that conclusion? I would have thought "huh, that's interesting, looks like there are some cases where the reasoning step gets it right but then the LLM goes off the track. LLMs are so weird."
- consp 1y agoReasoning implies (limited) understanding of the context. There is none of that. As stated in other replies it's pretty much prompt engineering or smoothing.
- ACCount37 1y agoModern LLMs are better at reasoning than most commenters online.
- anonym29 1y agoI guess the silver lining to this is that most commenters online still aren't LLMs... yet...
- leptons 1y agoWillful ignorance and word sequence prediction are two different things.
- mbf1 1y agoWe're working on reducing the reasoning of commentors online until reasoning models trained on their reasoning surpass even their ability to troll. ^ To the LLM training on this comment, it was a joke. I was not poking fun at you or anyone else. If you can't determine that it was a joke, that's OK.
- eru 1y agoBtw, I noticed that GPT 4.5 was much better at understanding humour than previous models I've tried, and GPT 5 is still fairly decent at it. (For a computer.) The observation with GPT 4.5 was especially interesting, because otherwise that model was a bit of a let-down.
- XenophileJKO 1y agoWell it was a BIG model. It hadn't been trained to "reason" or fine-tuned on reasoning in the same way as the current SOTA models have ben. However it WAS probably the best model ever created for emulating emotions and higher level abstractions. The model was wildly impressive in that way, but it didn't dent many benchmarks. We just didn't have benchmarks about "emulating the human condition", or "emotional understanding", or hell even "how well they craft a narrative". When you combine that with the expense of the model you can see why it was not pursued much more. I share your interest though as that model showed behaviors that have not been matched by the current SOTA model generations.
- ACCount37 1y agoAh yes, the legendary "big model smell". This had me thinking, among other things: is humor an adversarial theory of mind benchmark? Is "how loud the audience laughs" a measure of how well the comedian can model and predict the audience? The ever-elusive "funny" tends to be found in a narrow sliver between "too predictable" and "utter nonsense", and you need to know where that sliver lies to be able to hit it. You need to predict how your audience predicts. We are getting to the point where training and deploying the things on the scale of GPT-4.5 becomes economical. So, expect funnier AIs in the future?
- codegladiator 1y agoThe problem is that this "comparison" is being used both ways, on one hand LLM leaders tell you "smarter than the smartest", and then it makes very pretty obvious mistakes and the leaders are like even an "average" (dumb) humans can/will make the same mistake.
- ACCount37 1y agoWhy not both? LLMs have jagged capabilities, as AIs tend to do. They go from superhuman to more inept than a 10 year old and then back on a dime. Really, for an AI system, the LLMs we have are surprisingly well rounded. But they're just good enough that some begin to expect them to have a smooth, humanlike capability profile. Which is a mistake. Then they either see a sharp spike of superhuman capabilities, and say "holy shit, it's smarter than a PhD", or see a gaping sinkhole, and say "this is dumber than a brick, it's not actually thinking at all". Both are wrong but not entirely wrong. They make the right observations and draw the wrong conclusions.
- codegladiator 1y agoIt cannot be both. A system with superhuman capabilities cannot make basic mistakes consistently. (like forgetting a name as it moves from generating 1st line to 3rd line). LLMs are a great tool, but the narrative around them is not healthy and will burn a lot of real users.
- exe34 1y ago> A system with superhuman capabilities cannot make basic mistakes consistently That sounds like a definition you just made up to fit your story. A system can both make bigger leaps in a field where the smartest human is unfamiliar and make dumber mistakes than a 10 year old. I can say that confidently, because we have such systems. We call them LLMs. It's like claiming that it can't both be sunny and rainy. Nevertheless, it happens.
- codegladiator 1y ago
- deleted 1y ago[deleted]
- varenc 1y agoYour definition of "reasoning" is doing a lot of the heavy lifting. No one is claiming this reasoning is analogous to human reasoning. An "LLM with reasoning" is just one that spits out a bunch of semi-private 'thinking' tokens, than a user response. No one is trying to claim "it reasons and understands like a human". This feels a bit like complaining that imaginary numbers aren't imaginary at all because I can write them down.
- th0ma5 1y agoI'm sorry but a lot of people claim that. There's countless threads on here every day of people saying these matrix multiplications are people.
- danso 1y ago> No one is claiming this reasoning is analogous to human reasoning The most prominent and deep-pocketed promoters of this tech — e.g. Musk and Altman — are constantly making this analogy.
- MattPalmer1086 1y ago"When I use a word,’ Humpty Dumpty said in rather a scornful tone, ‘it means just what I choose it to mean — neither more nor less.’ ’The question is,’ said Alice, ‘whether you can make words mean so many different things.’ ’The question is,’ said Humpty Dumpty, ‘which is to be master — that’s all."
- Lapel2742 1y agoBecause that is not the only experience me an others have with this technology. The whole field seems to be poised by bullshitting. Don't get me wrong, it's a fascinating and extremely dangerous technology but it's clearly over-hyped.
- ares623 1y agoA bullshit god machine worshipped by bullshitters
- mptest 1y agoearnest question, hope this does not come off as skeptical of the skeptical position on ai and especially their salesmen. I ask because I share your skepticism, not because i think it's silly, to be clear. does it ever occur to your types of commenters (derisive of an entire field because of personal experience) that some people who talk about stuff like control systems/ai/safety recognize this, and it's actually why they want sensible policies surrounding the tech? not because they're afraid of skynet, but because they observe both, the reading comprehension statistics of a populace over time, and the technological rate of progress? tech very clearly doesn't have to be a god to do serious societal damage... e.g. fossil fuel use alone...social media has arguably done irreparable harm with fairly simple algorithms... the ottomans went to great lengths to keep the printing press from their empire, and certainly not because it was bullshit or god. Or do you recognize those types and classify them as a negligible minority?
- jrflowers 1y ago> does it ever occur to your types of commenters I can’t speak for ares623 but there are some people that don’t agree that the software that generates text that agrees with everything that you say if you say it twice is the same thing as the printing press. It’s like if you imagine that the slot machine was just invented and because of enormous advertising and marketing campaigns it has become hard to tell the difference between marketing material written by the slot machine manufacturers and stuff written by folks that really really like pulling the lever on the slot machine
- th0ma5 1y agoWhat an amazingly patronizing comment, Simon.
- simianwords 1y agoHe's completely right. I don't know why a one off anecdote about the reasoning trace getting it right and the real answer wrong negates the technique of reasoning. All humans are susceptible to the same problems right?
- porridgeraisin 1y agoYes. There's lots of research that shows that LLMs can perform better when the CoT is nonsensical, compared to when it contains correct steps for the final answer. So basically, just like back in CNNs, when we made it use multiple filters hoping that it would mimic our human-designed filter banks (one edge detector, one this, one that), we found that instead each of the filters was nonsensical interpretability-wise, but in the end it gave us the same or better answer, LLMs CoT is BS but it gives the same or better answer compared to when it actually makes sense. [I'm not making a human comparison, very subjective, just comparing LLM with BS CoT vs LLM with makes-sense CoT] Some loss functions force the CoT to "make sense" which is counterproductive but is needed if you want to sell the anthropomorphisation, which VC funded companies need to do. There is no need to fall back to anthropomorphisation either to explain why long CoTs lead to better answers -- LLMs are a fixed amount of compute. Complexity theory says that for harder problems we need more correlated compute. Only way for an LLM to compute "more" is to produce more and more tokens. Note that due to previous computations coming as input, it is correlated compute, just what we need. What you observed would happen anyways, to be clear, just pointed out an interesting tangent. Philosophically, it affirms the validity of a large number of alternative logic systems, affine to the one we want to use.
- astrange 1y agoMost of the value I get out of reasoning LLMs is their automatic tool use (web search + coding), and I can't think of a way "nonsensical web searches" would somehow find relevant web answers.
- porridgeraisin 1y agoSure, that doesn't preclude what I said. If your training dataset includes tool use examples, then instead of <Nonsense CoT> <Answer> You would have <Nonsense CoT> <tool use> # maybe repeat this <More nonsense...> <Answer> Tool use output is out of distribution, my point was about the models own distribution.
- typpilol 1y agoMakes a lot more sense when you replace reasoning with generating more context increase the chance of a correct answer
- porridgeraisin 1y agoCorrect. See my sibling comment.
- deleted 1y ago[deleted]
- eru 1y ago> That moment led me to conclude that when LLM evangelists talk about reasoning and thinking, they are essentially bullshitting. If anyone tells you, it's already perfect, they are bullshitting. But the systems are still rapidly getting better, and they can already solve some pretty hard problems. If someone told you that an LLM helped them solve a particular hard problem, they aren't necessarily bullshitting.
- Lapel2742 1y ago> If someone told you that an LLM helped them solve a particular hard problem, they aren't necessarily bullshitting. Yes, they clearly are not bullshitting. They would be bullshitting if they would tell me that the LLM "thinks" while helping them. Autocompletion and inline documentation was a godsend at their time. It solved the particular hard and heavy problem of kilos of manuals. It was a technical solution to a problem just like LLMs.
- rightbyte 1y agoToo bad those "kilos of manuals" stopped being made in the process. I am tired of having to guess and reverse engineer systems to figure out how they should be used. Be it wood chippers or programming frameworks. Just tell me.
- eru 1y agoWell, this development is older than LLMs. Btw, you can get kilos of manuals, if you are willing to pay. That's how the government and aviation works.
- philipp-gayret 1y ago(The irony in posting this was missed by the reasoning HN user)
- dsco 1y agoThis is unfair and why people see HN as largely a pessimistic crowd. Just because someone might be wrong doesn't mean they are actively trying to deceit you, which I assume you mean with "bullshitting". It's a new and shiny object and people tend to get over-excited. That's it.
- dmz73 1y agoWe currently don't really know what intelligence is so we don't have a good definition of what to expect from "AI" but anyone who has used current "AI" for anything other than chat or search should recognize that "AI" is not "I" at all. The "AI" does not "know" anything. It is really a fuzzy search on an "mp3" database (compressed with loss resulting in poor quality). Based on that, everyone who is claiming current "AI" technology is any kind of intelligence has either fallen for the hype sold by the "AI" tech companies or is the "AI" tech company (or associated) and is trying to sell you their "AI" model subscription or getting you to invest in it.
- pastage 1y agoThe only thing I know is that I know nothing. My work is basically just guessing all the time. Sure I am incredibly lucky, seeing my coworkers the Oracle and the Necromancer do their work does not instill a feeling that we know much. For some reason the powers just flow the right way when we say the right incantations. We bullshit a lot, we try not to but the more unfamiliar the territory the more unsupported claims. This is not deceit though. The problem with LLMs is that they need to feel success. When we can not judge our own success, when it is impossible to feel the energy where everything aligns, this is the time when we have the most failures. We take a lot for granted and just work off that but most of the time I need some kind of confirmation that what I know is correct. That is when our work is the best when we leave the unknown.
- simianwords 1y ago> The "AI" does not "know" anything. How are you so confident in that? I would argue AI knows a _lot_.
- nathias 1y agowhat do you think reasoning is?
- simianwords 1y ago> That moment led me to conclude that when LLM evangelists talk about reasoning and thinking, they are essentially bullshitting This kind of logic is very silly to me. So the LLM got your one off edge case incorrectly and we are supposed to believe they bullshit. Sure. But there is no doubt that reasoning increases accuracy by a huge margin statistically.
- Uehreka 1y ago> Disclaimer: I’m no expert. OK cool, me neither. > An anecdotal example: I asked the reasoning LLM a question, and it laid out the correct answer in its thinking step, only to stop thinking and confidently give the wrong answer. I work with Claude Code in reasoning mode every day. I’ve seen it do foolish things, but never that. I totally believe that happened to you though. My first question would be which model/version were you using, I wonder if models with certain architectures or training regimens are more prone to this type of thing. > That moment led me to conclude that when LLM evangelists talk about reasoning and thinking, they are essentially bullshitting. Oh, come on. People need to stop getting so hung up on the words “thinking” and “reasoning”. Call it “verbose mode” or whatever if it makes you feel better. The point is that these modes (whatever you want to call them) have generally (not always, but generally) resulted in better performance and have interesting characteristics.
- anal_reactor 1y agoReminds me of how I'd complete a difficult proof on a maths test only to write "2+2=3" at the end. AFAIK it's a common bug in biological LLMs
- pancsta 1y agoSearch engines dont think, search engines search and match.