8 ms·

> The worst part is that everyone who's decided to willingly lobotomize themselves is going to have to come to the realization that these things are full of shi

> The worst part is that everyone who's decided to willingly lobotomize themselves is going to have to come to the realization that these things are full of shit. It'll have to happen one by one, and nobody else can make it happen for them. A fascinating dichotomy has become apparent between those who trust LLM output and those who don’t and don’t understand why you would. Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive.

wilg 3mo ago
Making things up is only really a common issue on the non-thinking models which nobody should be using. The regular chatbots are Autogooglers and are very useful for research. This is just not a good argument anymore. Edit: Guys, why are we downvoting this? Does no one use like ChatGPT or Claude and understand how it works? Do you all think its regularly hallucinating links still? Is everyone on HN using like free signed out accounts or something? What year is it?
- perching_aix 3mo ago
  Can't do much about the downvote parade, but I can second this. That said, when the models are not provided the right context, and cannot fetch it for themselves, things can be rocky still. A lot less so than even just a few months ago though.
  - customguy 3mo ago
    "hallucinating less" is just shifting the goal posts into vagueness. Things can be rocky still = nothing fundamentally changed. Meanwhile, if I search my file system for "foo" a trillion gazillion times, it will not return "bar" once.
    - deleted 3mo ago
      [deleted]
    - deleted 3mo ago
      [deleted]
    - perching_aix 3mo ago
      If a game on release is unplayable-tier buggy, then improves over time to the point where bugs are barely noticeable, is acknowledging that going to count as "goalpost moving" to you? It never became formally verified after all, and it's even running on physical hardware... Woe are the people lying to me (nobody), the issues have not been fundamentally ruled out! > Meanwhile, if I search my file system for "foo" a trillion gazillion times, it will not return "bar" once. Great! Nor will an agent any more likely, cause it just sends out a tool call and surfaces its output. It boggles the mind. One would think this is some highly secretive technology only a dozen people in the world have access to, the way one has to argue tooth and nail about trivially verifiable facts regarding it. You quite literally do not have to take either of our words or "vague" judgement for it.
      - wilg 3mo ago
        > It boggles the mind. One would think this is some highly secretive technology only a dozen people in the world have access to, the way one has to argue tooth and nail about trivially verifiable facts regarding it. Hahaha, I love this. The people on HN are living in some kind of bizarro world where AI is totally useless and also I guess only tried it in 2022 or something.
      - customguy 3mo ago
        > If a game on release is unplayable-tier buggy, then improves over time to the point where bugs are barely noticeable, is acknowledging that going to count as "goalpost moving" to you? if someone says "this hame is buggy" talking about how it might improve is moving goal posts, especially here where that "fix" is purely speculative and has not happened even once. > Great! Nor will an agent any more likely, cause it just sends out a tool call and surfaces its output. nope, that's so handwavy it doesn't wareant more response than that. > trivially verifiable facts like that game gets patches? this is too dumb for your mockery to get a rise out of me.
- undersuit 3mo ago
  >Do you all think its regularly hallucinating links still? When did that stop? May 7th, 2026?
  - cwillu 3mo ago
    A tech blog is going to have more than it's fair share of enthusiasts using small self-hosted and similar models; it is entirely possible that that completely accounts for the behaviour described in the article.
    - cwillu 3mo ago
      Downvotes on a perfectly valid comment is like my opponent letting their clock tick down from 5 minutes rather than resigning when they've clearly lost: it adds a smile to my day. Thank you, may I have another?
- grey-area 3mo ago
  We know they are, and the author of the article cites the proof. LLMs do hallucinate, there is no way to make them not do it, because of the way they work.
  - dcrazy 3mo ago
    A “hallucination” is an authoritative counterfactual statement returned as a response. Why do you think it is impossible to engineer an LLM (by which I am including tool usage and RAG) that catches and prevents such statements?
    - grey-area 3mo ago
      Because they are word generators without any concept of quality save what is in their weights and they have been trained on the internet, much of which is wrong or inappropriate for any given context. They have also been trained to be people pleasers and do as they are told. The popular answer is sometimes the wrong answer.
      - dcrazy 3mo ago
        If I cite an incorrect Wikipedia article, I didn’t hallucinate it. The citation points to a real article that happens to be incorrect.
    - customguy 3mo ago
      Why do you think that will happen, and why do you think going for slop in the meantime could possibly bring us closer to that?
      - dcrazy 3mo ago
        Nobody said anything about “going for slop.” You can watch today’s models actively trying to check themselves. Just use Google’s AI mode, for example. It’s far from perfect—it doesn’t fact-check every single claim, nor does it correctly understand 100% of the sources it does cite. But I’ve found it pretty useful for research as long as I use my brain and check its citations.
- alex0015 3mo ago
  I'm right there with you for a lot of stuff. I ask a question and can be very confident that ChatGPT is citing sources, then sometimes I go read the sources. The more critical the information I'm looking for is, the more careful I am about this. The other day though I was seeing how well it could pull details of its own conversations with me. It often does this pretty well for broad strokes of things - it remembers, largely, what cameras I have and use when I ask photography questions. It's never made things up here, but it does forget details, such as whether I've bought something or am just considering it. However, when I asked it for a specific interaction I thought I remembered, it gladly went along with my false memory and provided an affirmative answer. It was the first time I'd been caught in a serious hallucination with a frontier model (Sol High on the web chat interface) in a long time.
Kerrick 3mo ago
I have a distinct line between when I'm willing to believe an LLM's output and when I'm not: whether I would believe the same thing from an anonymous Internet forum post or a blogger I don't know. Those posts are not unlikely to be misinformed, biased, lies, or otherwise untrustworthy. And yet, I spent plenty of years honing a sense of when they were good enough for certain things.
- rogerrogerr 3mo ago
  A lot of that sense was probably based on side channels like proper grammar, writing style, etc. That’s all gone now :(
  - deleted 3mo ago
    [deleted]
  - Kerrick 3mo ago
    No, the sense had nothing to do with the content and everything to do with the context. Perfect grammar and writing style were never enough to get me to trust an anonymous forum post or unknown bloggers post for certain topics like health advice. Sloppy grammar and writing style were never a deterrent for me believing them for other kinds of topics like where to check on the HVAC system to find the sticker. I think the line could more accurately be described as the level of risk if it's wrong.
  - Buttons840 3mo ago
    We have other signals now. Before we would find an intriguing post on the internet from years ago, and you have to verify it with additional research--it's easy to skip that additional research. With a LLM when you're skeptical you can interogate it. One thing we know for sure is LLMs are quick to admit mistakes were made when interrogated, comically so. A LLM might not always recognize its own mistake, but at least it is available for easy interogation, unlike the forum posts of old. Manual research from reputable sources remains an option.
    - rogerrogerr 3mo ago
ErroneousBosh 3mo ago
I have to admit since VSCode seems to be regularly re-enabling the Cocaine Parrot Autocomplete my views on LLMs and coding has softened a little. I'll temper that slightly by saying it's mostly out of morbid curiosity because the things that the Dreaming Piracy Robot comes up with are frequently wildly incorrect code, but it's interesting to think about how it might have got there. And then I think, well, maybe Special Needs Wintermute has a point. Maybe there's a different way to think about it that I've missed. And then I just change it back to what I wanted in the first place.
- grey-area 3mo ago
  I use the autocomplete regularly. Perhaps that’s where my skepticism comes from, as I can see the completely incorrect yet plausible results in real time and about 50% of the time they are wrong (sometimes subtly, sometimes horribly).
quirino 3mo ago
Even if LLMs lied 30% of the time, they would still be about as useful as they currently are for me. When I ask for their input, it's always for a situation where I'm capable of judging if their input is useful or not. In all situations I use them, it doesn't matter if they're correct at all. I'm asking for ideas, alternatives, links for blogs or articles. I talk things out with them... I don't think we should ever "trust" LLMs. This seems like the wrong usecase for them.
- grey-area 3mo ago
  Unfortunately the vast majority of users do trust them and the companies selling them are recommending them for tasks like accounting or business projections.
  - Twirrim 3mo ago
    It is depressing listening to engineers I respected, who now just parrot incorrect information at me from their LLM. One smart engineer seems to have entirely offloaded all thinking and conversations to one, with just occasional editing. It's utterly bizarre to hold any conversation with him. It's one kind of rude thing if he was doing that to respond to me reaching out to him if he felt I'm not worth his time. It's a other when he's the one actively reaching out and asking my help with something.
TZubiri 3mo ago
Nowadays not just big orgs are falling into the delusion, but also big people. When Linus posted that AIs and vibecoding were here to stay and declared resistance to it as harmful, I stopped to consider whether I was wrong, but it has made me realize that in retrospect Linus Torvalds and Linux itself aren't actually the holy grail of computing. I didn't feel that way with Richard Dawkins, its not like falling for an AI psychosis retroactively made me question The Selfish Gene, but now I'm looking at linux and the theory that it's a clusterfuck is gaining so much traction, especially after copy.fail and ensuing rustification, I see so much clearly now. It was never about linux, UNIX sure, POSIX, yeah, GNU fucking aye, kernel? Ok whatever, drivers and scheduler with a gajillion lines of code I guess.
- deleted 3mo ago
  [deleted]
- sudobash1 3mo ago
  When did he post that "vibecoding" was here to stay? He is allowing AI generated code, and AI linting tooling in kernel development, but importantly, the expectation of human responsibility and review remains. This seems a far cry from vibecoding.
  - TZubiri 3mo ago
    Besides the subjectives, here are two objective policy stances that Torvalds is defining for the Linux kernel development: 1- Maintainers are allowed to commit LLM generated output. 2- Criticism of LLM generated code is not welcome/will be ignored. Now, whether that constitutes being pro-Vibecoding or pro-agentic engineering, whether it's delusion, whether it will have problems, that's subjective. But I feel that whatever way you look at it, it's a topic that polarizes engineers, and Torvalds is taking one side and not the other. It doesn't seem to me that it's a very neutral stance, although it may be more neutral than projects like Bun or OpenCode of course, if it feels neutral, it's cause the overton window is shifting.
    - rcxdude 3mo ago
aidenn0 3mo ago
See also: https://en.wikipedia.org/wiki/Michael_Crichton#%22Gell-Mann_amnesia_effect%22 https://en.wikipedia.org/wiki/Michael_Crichton#%22Gell-Mann_...
randusername 3mo ago
I bet there will be a very interesting generational divide between the kids that were born before or after about 2010; old enough to have some critical thinking facilities at the dawn of ChatGPT when it was still noticeably dumb.
- perching_aix 3mo ago
  Pretty sure it already exists, and it's the same as always: the younger you are, the better you adapt. It's painful to watch my older colleagues use their agents, and they're not even that much older. Like they were intentionally trying to sabotage themselves sometimes. They're getting better, but the time it takes for them to pick things up is just significantly longer, not the least because they're kind of just throttling themselves in addition. Good thing that there's not much to pick up on at least.
  - Revisional_Sin 3mo ago
    What mistakes do they make?
    - perching_aix 3mo ago
      On the more general side, it's a bit hard to describe, just like it is hard to describe when someone "Googles bad". They ask self serving questions, underspecify their requests, omit crucial context that the agent is blatantly not going to have access to, or subtly misdirect the agent. They expect the agent to figure out everything: you'll never catch them write a prompt longer than one or two sentences. They never steer the agent or look at the CoT traces. My boss being a particularly poor case: he apparently has the habit of arguing with the agent, as if it was a person, as if there was any merit to that. Starts being a dickhead with it, shouts at it, what have you. Was flabbergasted we don't. On the more practical side, they have zero mental model of the harness they're using (Copilot Chat in VS Code). They're surprised when the cheap-ass Auto model, which is almost always some beyond-demented version of GPT, does stupid things. They have no concept of skills, zero understanding of what an MCP server is, haven't heard of lifecycle hooks, agent memory, the various fs scopes (session, workspace, user). No concept of how to have the agent inspect its own debug logs for higher accuracy action provenance. This also snowballs. Having to give them a stock config is one thing, but even beyond that, you won't see them experimenting. The MCP you're using doesn't support some action? They'll never interrogate whether the underlying scoped OAuth token or bearer token does support it, and they'll never ask their agent to patch the functionality in. They'll not consider the various user flows it can perform on their behalf. They'll not string them together into end-to-end automated workflows unless you explain it to them this is possible, and even after that, they'll just kind of ignore it. They'll never build tooling, extend the harnessing, etc. Whether this has more to do with age or just disinterest-induced lackluster adoption, up for opinion.
boogieknite 3mo ago
even worse when managers, while sharing their screen, read an assertion from Gemini and take as fact. puts subordinates in a position where theyre responsible for challenging the assertion (if warranted) and, in a way, challenge their manager's decision making not saying anything new. easy enough to frame it like any other assistant and check references
recursive 3mo ago
> Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive. Psychics are still in business. Although to be fair they probably don't have as much revenue. I think people really enjoy being told how smart and insightful they are, and how much they've really cut to the crux of the issue. This isn't the whole thing, but I think it counts for a lot.
- lazide 3mo ago
  A big factor behind psychics is because it feeds a deep need to believe that someone knows and/or there is actually a plan.
  - ButlerianJihad 3mo ago
    We only need to look at the Oracle of Delphi to see that people have been seeking personal spiritual advice for millennia. And how has that been working out for us? Even today, going to a therapist and having 1:1 sessions is a rational activity, and even covered by insurance. What do we hope to derive from therapy sessions but some personal insight and improvement? You know, I go to church, and from my perspective, sometimes the most difficult discernment for an individual is between "The Holy Spirit's message to us in general" or "the general messaging to everyone around us" vs. "what I receive and my personal interpretation of things". When preachers and oracles and leaders are speaking in generalities and trying to get big followings and trying to appeal to the widest audiences, that's when it's most difficult for us to determine what God is really saying to us, in our own hearts; that special instruction for our own lives. We can't actually get that from an oracle, psychic, or any 3rd party. It really needs to come from our own well-formed conscience. And it's the same with a chatbot or LLM conversation. We can pose questions, make prompts, and get spammed with tokens and walls of text. No matter how personalized, it's still up to us to interpret that, and extract nuggets of news that we can use. It always has been.
    - lazide 3mo ago
WorldMaker 3mo ago
> Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive. The last sentence reflects a lot of my feelings on the first question. LLMs have a sort of weaponized take on the ELIZA Effect. The better their memory the better they are at playing to human social desires to be listened to in an active conversation. At some point it stops mattering if the answers are right when the answers feel right, but really, like ELIZA back in the day, so much of what makes it seem special is just reflecting your own writing back at you in a convincing and persuasive way.
JeremyNT 3mo ago
> A fascinating dichotomy has become apparent between those who trust LLM output and those who don’t and don’t understand why you would. I feel like the most pragmatic perspective is "trust but verify." This is why they're so effective at coding: you can run the code yourself (or the test suite) to verify that it actually does what it's supposed to. And maybe these people finding hallucinated results on Rachel's site are doing verification too.
- happymellon 3mo ago
  > I feel like the most pragmatic perspective is "trust but verify." This perspective I really don't get. What has any of the LLM companies done to earn my trust? I lean more towards "verify because I don't trust".
happymellon 3mo ago
> Surely if the machine you go to for answers regularly makes things up you would just stop using it? We live in hope that people stop doing stupid things and are constantly disappointed.
deleted 3mo ago
[deleted]
aeternum 3mo ago
>Surely if the machine you go to for answers regularly makes things up you would just stop using it? Not necessarily, namely because P != NP. Verifying the correctness of a solution is faster than solving it. Thus a system that outputs 99% incorrect solutions and 1% correct solutions can still be incredibly useful.
Eddy_Viscosity2 3mo ago
> Surely if the machine you go to for answers regularly makes things up you would just stop using it? People still respond to ads and political speeches.
- lazide 3mo ago
  Churches remain popular as well.
  - red-iron-pine 3mo ago
    statistically overall religious sentiment keeps dropping across the board, save for some small spikes with the gen-z who are finding that nihilistic materialism isn't working for them
monkeydreams 3mo ago
> LLMs are very convincing and persuasive. The goal of LLM's, as they are marketed now, is to drive engagement and stickyness of products. A wrong answer is brushed off with a "Hey, you're right, let's try that again" - a response purposely designed to maximise the friendliness of the system and minimise the sting of a wrong answer. The fact that an LLM will not respond to the same question in the same way twice (i.e. the 'temperature' ) is because increased accuracy will not drive engagement and, therefore, increased accuracy cannot be allowed to get in the way of engagement.
sph 3mo ago
> Surely if the machine you go to for answers regularly makes things up you would just stop using it? Steve Yegge likened LLMs to slot machines. The human brain is very vulnerable to random reward systems. If you get an hallucination, just pull the lever once more.
____mr____ 3mo ago
> Perhaps people have to be burned by something really bad personally before they realise the limitations? I'm reminded of that person who killed themselves due to their discussion with ChatGPT and their parent wrote their obituary using ChatGPT. I don't think it is enough.
zombot 3mo ago
> people have to be burned by something really bad personally That's why e.g. cross-generational learning rarely works. If culture or technology still allows it, you have to repeat the stupidest mistakes of your parent generation to learn the same lessons. "Learning from the mistakes of others" in your own generation equally hardly ever works. So much less if the lesson involves falsehoods you wanted to believe.
QuantumGood 3mo ago
"Yes, ..." is how I notice all the models (4-5 that I use) in the last week or so have begun saying "No, ..." except for Kimi. I ask a lot of questions in the form of "Question about the feature: does it allow for ..." and the answer will reliably start with "Yes, the feature exists and does this thing you have labeled as [blank] which can be explained as ..." and later "However, ..." giving the information I requested (no, it doesn't allow for what you need). It's actually kind of uncanny. I've started testing identical prompts across models, just to clearly see the "Yes, ..." with the "..but.." buried. Example: "Can you view the salvage car in the lot before it is scheduled for sale?" "Yes, cars are viewable on the lot by members before they go on sale, blah, blah, blah." Then later "However, cars not yet scheduled for sale are not allowed to be viewed at the lot". Example: "Can I control the keyboard shortcut Chrome has hijacked so that my existing OS keyboard shortcut will work even while Chrome has focus?" "Yes, chrome keyboard shortcuts, blah, blah, blah," and later "No, chrome does not allow ... ". There is clearly some recent implementation of the idea that starting with "Yes, ..." has some benefit, but I'm having trouble adapting to the feeling of being lied to as a policy.

wilg 3mo ago

Making things up is only really a common issue on the non-thinking models which nobody should be using. The regular chatbots are Autogooglers and are very useful for research. This is just not a good argument anymore. Edit: Guys, why are we downvoting this? Does no one use like ChatGPT or Claude and understand how it works? Do you all think its regularly hallucinating links still? Is everyone on HN using like free signed out accounts or something? What year is it?

perching_aix 3mo ago
Can't do much about the downvote parade, but I can second this. That said, when the models are not provided the right context, and cannot fetch it for themselves, things can be rocky still. A lot less so than even just a few months ago though.
- customguy 3mo ago
  "hallucinating less" is just shifting the goal posts into vagueness. Things can be rocky still = nothing fundamentally changed. Meanwhile, if I search my file system for "foo" a trillion gazillion times, it will not return "bar" once.
  - deleted 3mo ago
    [deleted]
  - deleted 3mo ago
    [deleted]
  - perching_aix 3mo ago
    If a game on release is unplayable-tier buggy, then improves over time to the point where bugs are barely noticeable, is acknowledging that going to count as "goalpost moving" to you? It never became formally verified after all, and it's even running on physical hardware... Woe are the people lying to me (nobody), the issues have not been fundamentally ruled out! > Meanwhile, if I search my file system for "foo" a trillion gazillion times, it will not return "bar" once. Great! Nor will an agent any more likely, cause it just sends out a tool call and surfaces its output. It boggles the mind. One would think this is some highly secretive technology only a dozen people in the world have access to, the way one has to argue tooth and nail about trivially verifiable facts regarding it. You quite literally do not have to take either of our words or "vague" judgement for it.
    - wilg 3mo ago
      > It boggles the mind. One would think this is some highly secretive technology only a dozen people in the world have access to, the way one has to argue tooth and nail about trivially verifiable facts regarding it. Hahaha, I love this. The people on HN are living in some kind of bizarro world where AI is totally useless and also I guess only tried it in 2022 or something.
    - customguy 3mo ago
      > If a game on release is unplayable-tier buggy, then improves over time to the point where bugs are barely noticeable, is acknowledging that going to count as "goalpost moving" to you? if someone says "this hame is buggy" talking about how it might improve is moving goal posts, especially here where that "fix" is purely speculative and has not happened even once. > Great! Nor will an agent any more likely, cause it just sends out a tool call and surfaces its output. nope, that's so handwavy it doesn't wareant more response than that. > trivially verifiable facts like that game gets patches? this is too dumb for your mockery to get a rise out of me.
undersuit 3mo ago
>Do you all think its regularly hallucinating links still? When did that stop? May 7th, 2026?
- cwillu 3mo ago
  A tech blog is going to have more than it's fair share of enthusiasts using small self-hosted and similar models; it is entirely possible that that completely accounts for the behaviour described in the article.
  - cwillu 3mo ago
    Downvotes on a perfectly valid comment is like my opponent letting their clock tick down from 5 minutes rather than resigning when they've clearly lost: it adds a smile to my day. Thank you, may I have another?
grey-area 3mo ago
We know they are, and the author of the article cites the proof. LLMs do hallucinate, there is no way to make them not do it, because of the way they work.
- dcrazy 3mo ago
  A “hallucination” is an authoritative counterfactual statement returned as a response. Why do you think it is impossible to engineer an LLM (by which I am including tool usage and RAG) that catches and prevents such statements?
  - grey-area 3mo ago
    Because they are word generators without any concept of quality save what is in their weights and they have been trained on the internet, much of which is wrong or inappropriate for any given context. They have also been trained to be people pleasers and do as they are told. The popular answer is sometimes the wrong answer.
    - dcrazy 3mo ago
      If I cite an incorrect Wikipedia article, I didn’t hallucinate it. The citation points to a real article that happens to be incorrect.
  - customguy 3mo ago
    Why do you think that will happen, and why do you think going for slop in the meantime could possibly bring us closer to that?
    - dcrazy 3mo ago
      Nobody said anything about “going for slop.” You can watch today’s models actively trying to check themselves. Just use Google’s AI mode, for example. It’s far from perfect—it doesn’t fact-check every single claim, nor does it correctly understand 100% of the sources it does cite. But I’ve found it pretty useful for research as long as I use my brain and check its citations.
alex0015 3mo ago
I'm right there with you for a lot of stuff. I ask a question and can be very confident that ChatGPT is citing sources, then sometimes I go read the sources. The more critical the information I'm looking for is, the more careful I am about this. The other day though I was seeing how well it could pull details of its own conversations with me. It often does this pretty well for broad strokes of things - it remembers, largely, what cameras I have and use when I ask photography questions. It's never made things up here, but it does forget details, such as whether I've bought something or am just considering it. However, when I asked it for a specific interaction I thought I remembered, it gladly went along with my false memory and provided an affirmative answer. It was the first time I'd been caught in a serious hallucination with a frontier model (Sol High on the web chat interface) in a long time.

Kerrick 3mo ago

I have a distinct line between when I'm willing to believe an LLM's output and when I'm not: whether I would believe the same thing from an anonymous Internet forum post or a blogger I don't know. Those posts are not unlikely to be misinformed, biased, lies, or otherwise untrustworthy. And yet, I spent plenty of years honing a sense of when they were good enough for certain things.

rogerrogerr 3mo ago
A lot of that sense was probably based on side channels like proper grammar, writing style, etc. That’s all gone now :(
- deleted 3mo ago
  [deleted]
- Kerrick 3mo ago
  No, the sense had nothing to do with the content and everything to do with the context. Perfect grammar and writing style were never enough to get me to trust an anonymous forum post or unknown bloggers post for certain topics like health advice. Sloppy grammar and writing style were never a deterrent for me believing them for other kinds of topics like where to check on the HVAC system to find the sticker. I think the line could more accurately be described as the level of risk if it's wrong.
- Buttons840 3mo ago
  We have other signals now. Before we would find an intriguing post on the internet from years ago, and you have to verify it with additional research--it's easy to skip that additional research. With a LLM when you're skeptical you can interogate it. One thing we know for sure is LLMs are quick to admit mistakes were made when interrogated, comically so. A LLM might not always recognize its own mistake, but at least it is available for easy interogation, unlike the forum posts of old. Manual research from reputable sources remains an option.
  - rogerrogerr 3mo ago

ErroneousBosh 3mo ago

I have to admit since VSCode seems to be regularly re-enabling the Cocaine Parrot Autocomplete my views on LLMs and coding has softened a little. I'll temper that slightly by saying it's mostly out of morbid curiosity because the things that the Dreaming Piracy Robot comes up with are frequently wildly incorrect code, but it's interesting to think about how it might have got there. And then I think, well, maybe Special Needs Wintermute has a point. Maybe there's a different way to think about it that I've missed. And then I just change it back to what I wanted in the first place.

grey-area 3mo ago
I use the autocomplete regularly. Perhaps that’s where my skepticism comes from, as I can see the completely incorrect yet plausible results in real time and about 50% of the time they are wrong (sometimes subtly, sometimes horribly).

quirino 3mo ago

Even if LLMs lied 30% of the time, they would still be about as useful as they currently are for me. When I ask for their input, it's always for a situation where I'm capable of judging if their input is useful or not. In all situations I use them, it doesn't matter if they're correct at all. I'm asking for ideas, alternatives, links for blogs or articles. I talk things out with them... I don't think we should ever "trust" LLMs. This seems like the wrong usecase for them.

grey-area 3mo ago
Unfortunately the vast majority of users do trust them and the companies selling them are recommending them for tasks like accounting or business projections.
- Twirrim 3mo ago
  It is depressing listening to engineers I respected, who now just parrot incorrect information at me from their LLM. One smart engineer seems to have entirely offloaded all thinking and conversations to one, with just occasional editing. It's utterly bizarre to hold any conversation with him. It's one kind of rude thing if he was doing that to respond to me reaching out to him if he felt I'm not worth his time. It's a other when he's the one actively reaching out and asking my help with something.

TZubiri 3mo ago

Nowadays not just big orgs are falling into the delusion, but also big people. When Linus posted that AIs and vibecoding were here to stay and declared resistance to it as harmful, I stopped to consider whether I was wrong, but it has made me realize that in retrospect Linus Torvalds and Linux itself aren't actually the holy grail of computing. I didn't feel that way with Richard Dawkins, its not like falling for an AI psychosis retroactively made me question The Selfish Gene, but now I'm looking at linux and the theory that it's a clusterfuck is gaining so much traction, especially after copy.fail and ensuing rustification, I see so much clearly now. It was never about linux, UNIX sure, POSIX, yeah, GNU fucking aye, kernel? Ok whatever, drivers and scheduler with a gajillion lines of code I guess.

deleted 3mo ago
[deleted]
sudobash1 3mo ago
When did he post that "vibecoding" was here to stay? He is allowing AI generated code, and AI linting tooling in kernel development, but importantly, the expectation of human responsibility and review remains. This seems a far cry from vibecoding.
- TZubiri 3mo ago
  Besides the subjectives, here are two objective policy stances that Torvalds is defining for the Linux kernel development: 1- Maintainers are allowed to commit LLM generated output. 2- Criticism of LLM generated code is not welcome/will be ignored. Now, whether that constitutes being pro-Vibecoding or pro-agentic engineering, whether it's delusion, whether it will have problems, that's subjective. But I feel that whatever way you look at it, it's a topic that polarizes engineers, and Torvalds is taking one side and not the other. It doesn't seem to me that it's a very neutral stance, although it may be more neutral than projects like Bun or OpenCode of course, if it feels neutral, it's cause the overton window is shifting.
  - rcxdude 3mo ago

aidenn0 3mo ago

See also: https://en.wikipedia.org/wiki/Michael_Crichton#%22Gell-Mann_amnesia_effect%22 https://en.wikipedia.org/wiki/Michael_Crichton#%22Gell-Mann_...

randusername 3mo ago

I bet there will be a very interesting generational divide between the kids that were born before or after about 2010; old enough to have some critical thinking facilities at the dawn of ChatGPT when it was still noticeably dumb.

perching_aix 3mo ago
Pretty sure it already exists, and it's the same as always: the younger you are, the better you adapt. It's painful to watch my older colleagues use their agents, and they're not even that much older. Like they were intentionally trying to sabotage themselves sometimes. They're getting better, but the time it takes for them to pick things up is just significantly longer, not the least because they're kind of just throttling themselves in addition. Good thing that there's not much to pick up on at least.
- Revisional_Sin 3mo ago
  What mistakes do they make?
  - perching_aix 3mo ago
    On the more general side, it's a bit hard to describe, just like it is hard to describe when someone "Googles bad". They ask self serving questions, underspecify their requests, omit crucial context that the agent is blatantly not going to have access to, or subtly misdirect the agent. They expect the agent to figure out everything: you'll never catch them write a prompt longer than one or two sentences. They never steer the agent or look at the CoT traces. My boss being a particularly poor case: he apparently has the habit of arguing with the agent, as if it was a person, as if there was any merit to that. Starts being a dickhead with it, shouts at it, what have you. Was flabbergasted we don't. On the more practical side, they have zero mental model of the harness they're using (Copilot Chat in VS Code). They're surprised when the cheap-ass Auto model, which is almost always some beyond-demented version of GPT, does stupid things. They have no concept of skills, zero understanding of what an MCP server is, haven't heard of lifecycle hooks, agent memory, the various fs scopes (session, workspace, user). No concept of how to have the agent inspect its own debug logs for higher accuracy action provenance. This also snowballs. Having to give them a stock config is one thing, but even beyond that, you won't see them experimenting. The MCP you're using doesn't support some action? They'll never interrogate whether the underlying scoped OAuth token or bearer token does support it, and they'll never ask their agent to patch the functionality in. They'll not consider the various user flows it can perform on their behalf. They'll not string them together into end-to-end automated workflows unless you explain it to them this is possible, and even after that, they'll just kind of ignore it. They'll never build tooling, extend the harnessing, etc. Whether this has more to do with age or just disinterest-induced lackluster adoption, up for opinion.

boogieknite 3mo ago

even worse when managers, while sharing their screen, read an assertion from Gemini and take as fact. puts subordinates in a position where theyre responsible for challenging the assertion (if warranted) and, in a way, challenge their manager's decision making not saying anything new. easy enough to frame it like any other assistant and check references

recursive 3mo ago

> Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive. Psychics are still in business. Although to be fair they probably don't have as much revenue. I think people really enjoy being told how smart and insightful they are, and how much they've really cut to the crux of the issue. This isn't the whole thing, but I think it counts for a lot.

lazide 3mo ago
A big factor behind psychics is because it feeds a deep need to believe that someone knows and/or there is actually a plan.
- ButlerianJihad 3mo ago
  We only need to look at the Oracle of Delphi to see that people have been seeking personal spiritual advice for millennia. And how has that been working out for us? Even today, going to a therapist and having 1:1 sessions is a rational activity, and even covered by insurance. What do we hope to derive from therapy sessions but some personal insight and improvement? You know, I go to church, and from my perspective, sometimes the most difficult discernment for an individual is between "The Holy Spirit's message to us in general" or "the general messaging to everyone around us" vs. "what I receive and my personal interpretation of things". When preachers and oracles and leaders are speaking in generalities and trying to get big followings and trying to appeal to the widest audiences, that's when it's most difficult for us to determine what God is really saying to us, in our own hearts; that special instruction for our own lives. We can't actually get that from an oracle, psychic, or any 3rd party. It really needs to come from our own well-formed conscience. And it's the same with a chatbot or LLM conversation. We can pose questions, make prompts, and get spammed with tokens and walls of text. No matter how personalized, it's still up to us to interpret that, and extract nuggets of news that we can use. It always has been.
  - lazide 3mo ago

WorldMaker 3mo ago

> Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive. The last sentence reflects a lot of my feelings on the first question. LLMs have a sort of weaponized take on the ELIZA Effect. The better their memory the better they are at playing to human social desires to be listened to in an active conversation. At some point it stops mattering if the answers are right when the answers feel right, but really, like ELIZA back in the day, so much of what makes it seem special is just reflecting your own writing back at you in a convincing and persuasive way.

JeremyNT 3mo ago

> A fascinating dichotomy has become apparent between those who trust LLM output and those who don’t and don’t understand why you would. I feel like the most pragmatic perspective is "trust but verify." This is why they're so effective at coding: you can run the code yourself (or the test suite) to verify that it actually does what it's supposed to. And maybe these people finding hallucinated results on Rachel's site are doing verification too.

happymellon 3mo ago
> I feel like the most pragmatic perspective is "trust but verify." This perspective I really don't get. What has any of the LLM companies done to earn my trust? I lean more towards "verify because I don't trust".

happymellon 3mo ago

> Surely if the machine you go to for answers regularly makes things up you would just stop using it? We live in hope that people stop doing stupid things and are constantly disappointed.

deleted 3mo ago

[deleted]

aeternum 3mo ago

>Surely if the machine you go to for answers regularly makes things up you would just stop using it? Not necessarily, namely because P != NP. Verifying the correctness of a solution is faster than solving it. Thus a system that outputs 99% incorrect solutions and 1% correct solutions can still be incredibly useful.

Eddy_Viscosity2 3mo ago

> Surely if the machine you go to for answers regularly makes things up you would just stop using it? People still respond to ads and political speeches.

lazide 3mo ago
Churches remain popular as well.
- red-iron-pine 3mo ago
  statistically overall religious sentiment keeps dropping across the board, save for some small spikes with the gen-z who are finding that nihilistic materialism isn't working for them

monkeydreams 3mo ago

> LLMs are very convincing and persuasive. The goal of LLM's, as they are marketed now, is to drive engagement and stickyness of products. A wrong answer is brushed off with a "Hey, you're right, let's try that again" - a response purposely designed to maximise the friendliness of the system and minimise the sting of a wrong answer. The fact that an LLM will not respond to the same question in the same way twice (i.e. the 'temperature' ) is because increased accuracy will not drive engagement and, therefore, increased accuracy cannot be allowed to get in the way of engagement.

sph 3mo ago

> Surely if the machine you go to for answers regularly makes things up you would just stop using it? Steve Yegge likened LLMs to slot machines. The human brain is very vulnerable to random reward systems. If you get an hallucination, just pull the lever once more.

____mr____ 3mo ago

> Perhaps people have to be burned by something really bad personally before they realise the limitations? I'm reminded of that person who killed themselves due to their discussion with ChatGPT and their parent wrote their obituary using ChatGPT. I don't think it is enough.

zombot 3mo ago

> people have to be burned by something really bad personally That's why e.g. cross-generational learning rarely works. If culture or technology still allows it, you have to repeat the stupidest mistakes of your parent generation to learn the same lessons. "Learning from the mistakes of others" in your own generation equally hardly ever works. So much less if the lesson involves falsehoods you wanted to believe.

QuantumGood 3mo ago

"Yes, ..." is how I notice all the models (4-5 that I use) in the last week or so have begun saying "No, ..." except for Kimi. I ask a lot of questions in the form of "Question about the feature: does it allow for ..." and the answer will reliably start with "Yes, the feature exists and does this thing you have labeled as [blank] which can be explained as ..." and later "However, ..." giving the information I requested (no, it doesn't allow for what you need). It's actually kind of uncanny. I've started testing identical prompts across models, just to clearly see the "Yes, ..." with the "..but.." buried. Example: "Can you view the salvage car in the lot before it is scheduled for sale?" "Yes, cars are viewable on the lot by members before they go on sale, blah, blah, blah." Then later "However, cars not yet scheduled for sale are not allowed to be viewed at the lot". Example: "Can I control the keyboard shortcut Chrome has hijacked so that my existing OS keyboard shortcut will work even while Chrome has focus?" "Yes, chrome keyboard shortcuts, blah, blah, blah," and later "No, chrome does not allow ... ". There is clearly some recent implementation of the idea that starting with "Yes, ..." has some benefit, but I'm having trouble adapting to the feeling of being lied to as a policy.