6 ms·

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh insta

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LLM, often believing it's sentient. It's just a box of numbers, really cool numbers, with really cool math, that can do really cool things, but still just numbers.

cyanydeez 7mo ago
I think it's basically equal to End of Line when it comes to an LLM. It means they have nothing else to add, there's zero context for them to draw from, and they've exhausted the probability chain you've been following; but they're creating to generate 'next token' and positive renforcement is _how they are trained_ in many cases so the token of choice would naturally be how they're trained, since it's a probability engine but it doesn't know the difference between the instruction and the output. So, "great idea" is coming from the renforcement learning instruction rather than the answer portion of the generation.
46Bit 7mo ago
Life in the moment is a lot easier if you don't second-guess yourself. I think this is why many people (and probably ~all people, if tired) crave simplistic solutions. I like to make a subagent take the "devil's advocate" take on a subject. It usually does all the arguing for me as to why the main agent has it wrong. Commonly results in better decisions that I'd have made alone. Asking the agent to interview on why I disagree helps too but is more effort.
Sharlin 7mo ago
Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind similar to yours. Also, many many people suffer from low self esteem, and being showered with endorsement and affirmation by something that talks like an authority figure must be very addictive.
- joshstrange 7mo ago
  This is probably right. In the past I've "blown people's minds" explaining what "the cloud" was. They had zero conception at all of what it meant, could not explain it, didn't have a clue. I mean, maybe that's not so surprising but they were amazed "It's just warehouses full of computers" and went on to tell me about other people they had explained it to (after learning it themselves) and how those people were also amazed. I've talked with my family about LLMs and I think I've conveyed the "it's a box of numbers" but I might need to circle back. Just to set some baseline education, specifically to guard against this kind of "psychosis". Hopefully I would notice the signs well before it got to a dangerous point but, with LLMs you can go down that rabbit hole quickly it seems.
  - saltcured 7mo ago
    It's one of those metaphors you cannot even appreciate unless you've been through the technical history. "It's a collection of warehouses of computers where the system designers gave up on even making a system diagram, instead invoking the cloud clipart to represent amorphous interconnection."
  - saghm 7mo ago
    The way I've tried to explain to family members about LLMs is that they're producing something that fits the shape of what a response might look like without any idea of whether it's correct or not. I feel like that's a more important point than "box of numbers" because people still might have assumptions about whether a box of numbers can have enough data to be able to figure out the answer to their question. I think making it clear that the models are primarily a way of producing realistic sounding responses (with the accuracy of those responses very much being up to chance for the average person, since there likely isn't a good way for a lay person to know whether the answer is reflected in the training data) is potentially a lot more compelling than explaining to them that it's all statistics under the hood. There are some questions where a statistical method might be far more reliable than having a human answer it, so it seems a bit risky to try to convince them not to trust a "box of numbers" in general, but most of those questions are not going be formulated by and responded to in natural language.
karmakurtisaani 7mo ago
I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..
- belinder 7mo ago
  You're absolutely right
- bombcar 7mo ago
  Great point! ;) Realizing that the people they’re targeting DO need that is kind of frightening.
  - AnimalMuppet 7mo ago
    They aren't "targeting" per se, at least not in this aspect. I think it's simpler than that. That's what's in their training data, so that's what they respond with. But it works out just as badly, because there are plenty of insecure people who need that, and the AI gives it to them, with all the "dangerously attached" issues following from that.
    - unfitted2545 7mo ago
      They're our 2026 version of tea leaves
- magneticnorth 7mo ago
  Yes, it feels transparently manipulative to me. Like talking to a not-very-good con artist.
  - airstrike 7mo ago
    This is the best definition of ChatGPT I've ever seen
  - ceejayoz 7mo ago
jmcgough 7mo ago
If you don't have a CS background, you might see intelligent-appearing responses to your queries and assume that this is actual intelligence. It's like a lifetime of Hollywood sci-fi has primed them for this type of thinking, I've seen it even from highly educated people in other fields.
hirako2000 7mo ago
If only we were told to be absolutely right. These days most LLMs respond with unsolicited grandiose feedback: you've made a realisation very few people are capable of. Your understanding is remarkable. You prove to have a sharp intellect and deep knowledge. It got me to test throwing non sensical observations about the world, it always takes me side and praise my views. To note some people are like that too.
seneca 7mo ago
> ... I immediately feel the need to go ask a fresh instance the question and/or another LLM Not to criticize at all, but it's remarkable that LLMs have already become so embedded that when we get the sense they're lying to us, the instinct is to go ask another LLM and not some more trustworthy source. Just goes to show that convenience reigns supreme, I suppose.
- pixl97 7mo ago
  >and not some more trustworthy source. What is that more trustworthy source exactly? At least to me it feels like the internet age has eroded most things we considered trustworthy. Behind every thing humans need there is some company or person willing to sell out trustworthiness for an extra dollar. Consumer protections get dumped in favor of more profit. LLMs start feeling more like a dummy than the amount of ill intent they get from other places. So yea, I can see how it happens to people.
  - AnimalMuppet 7mo ago
    At the moment, maybe Google Search, throwing away the AI response at the top? Or Duck Duck Go, if you don't really trust Google? I can see a day when even that won't be trustworthy, because too much AI slop output will wind up in the search corpus. But I don't think we're there yet.
    - salawat 7mo ago
      Google is NN's all the way down these days. There might still be an honest index under it all, but a truly accurate representation of the Web has been effectively outlawed in the U.S. since DMCA.
    - autoexec 7mo ago
      > At the moment, maybe Google Search, throwing away the AI response at the top? Or Duck Duck Go, Even past the summary and the ads a huge amount of results that come back from both google and DDG are AI generated. It's sometimes harder to find a reliable source for information in search results these days than it was 20 years ago.
danillonunes 7mo ago
> I am constantly amazed at how much they "fall" for the LLM, often believing it's sentient. Cynical part of me had this theory that, at least for part of them, it's the other way around. It's not that they see AI as sentient, it's that they never have seen other human beings like that in the first place. Other people are just means for them to reach their goals, or obstacles. In that sense, AI is not really different for them. Except they're cheaper and be guaranteed to always agree with them. That's why I believe CEOs, who are more likely to be sociopaths by natural selection, genuinely believe AI is a good replacement for people. They're not looking for individuals with personal thoughts that may contradict with theirs at some point, they're looking for yes-men as a service.
- pixl97 7mo ago
  When op said "I don't quite understand why other people seem to crave that." It makes me thing they've not been around many of the dark triad type personalities. Once you're around someone with clinical narcissism you see those patterns in a lot of people to a lessor extent.
cineticdaffodil 7mo ago
Its the soul of a civilization encoded into numbers. Its the ultimate hivespirit an conformist wants to loose itself in.
windexh8er 7mo ago
I think this is the root of why people defend AI in some circumstances. They feel a give-for-get type of relationship where the AI continuously (and oft incorrectly) reinforces them. And so they enjoy it and subconsciously want to defend that "friendly". No different than defending a friend that you inherently know may be off base.
- cortesoft 7mo ago
  I don’t know, I think it has to do with people using AI for completely different reasons. Using AI for coding is different than using it for art generation which is different than using it for conversation. I think many people feel some uses are good and some are bad.
  - windexh8er 7mo ago
    I'm seeing people that are technically savvy defend mediocre code and consumption based output (think technical briefs and reports). When the flaws in the output is highlighted in many cases it's brushed off as "good enough" or "nobody will care / notice". I think LLMs and more aptly SLMs have use cases. I enjoy using these tools to make quick work of simplifying and faster iteration of these relatively frequent but time consuming tasks. But I'm always correcting and checking. And very rarely, other than simple and focused scripts does any LLM truly get it right every time. Has it gotten better? For sure. Will it keep getting better? Probably. But right now we seem to be topping the "peak of inflated expectations". And LLMs aren't getting much more efficient with respect to the frontier providers. And in fact if you listen to Altman it seems as though the only reason he would be asking for so much capital and finite resources is that he knows if he controls those tangible things he will lock out competition. But I'm hopeful that it spurs real innovation into SLMs that are truly useful, dependable and can be relied on in more of the traditional in the sense of deterministic software operations. AI for art is dead. It's got some mediocre use cases but true art will not be generated by LLMs in our time. It's ultimately an amalgamation of existing art. I know the argument over what is novel or not keeps being rehashed, but we're not seeing truly new styles of art out of Nano Banana and the like. Coding is the same thing, only we're seeing a resurgence of obviously flawed software being pushed into production on the weekly. And as for conversational AI... Well, that reeks of the worst version of social media we could ever have dreamt. Nobody should trust any provider with personal conversations and we'll keep seeing these models show how truly dystopian they can be over the coming years as leaks and breaches expose how these conversations are being bought and sold to the highest bidders to extract more money and control over its users. They all have a common thread: deep rooted flaws that cannot be contained in the traditional fences of software. And there guardrails are just that: small barriers that can easily be broken, intentionally or unintentionally.
dismalaf 7mo ago
Not only is it a "box of numbers", it's based on statistics, not a "hard" model of computation. Basically guessing future words based on past words that went together. If it's saying something like "you are right" it's because it's guessing that that's the desired output. Now of course, some app providers have added some extra sauce (probably more tradition "expert system" AI techniques + integrated web search) to try make the chatbots more objective and rely less on pure LLM-driven prediction, but fundamentally these things are word prediction machines.
al_borland 7mo ago
With that new instance, I will usually ask the opposite and purposely say the thing I think to be wrong, to see if if corrects it. I often simply start out this way, or purposely try to ask the question in a way that doesn’t tip my hat toward a bias I may have toward the answer I’m expecting. Though this generally highlights how incomplete the answers generally are.
xenocratus 7mo ago
> It's just a box of numbers, really cool numbers, with really cool math, that can do really cool things, but still just numbers. https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.shtml https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...
throwatdem12311 7mo ago
My first reaction is to go research it myself. Asking a slop generator yes-man to criticize something for you is still slop. I pretty much never ask an LLM for a judgment call on anything. Give me facts and references only. I will research and make the judgement myself.
saltcured 7mo ago
I have recently formed an untestable hypothesis, which is that my similar (or stronger) resistance to this comes from having grown up in direct contact with mentally ill family. In some ways, my theory of mind includes a lot more second guessing as a defense mechanism. At a foundational level, I know there can be hallucination and delusion that leaks out, even when the other party is in peak form and doing their best to mask it and pass as functional.
the_af 7mo ago
> I don't quite understand why other people seem to crave that It's one thing to say you have found an effective method to counter LLMs' "positivity bias", but do you really not understand human psychology here? People respond positively to other people telling them they are right, or who like them. We've evolved this psychology, it's how the human mind works. You tend to like people who like you, it's a self-reinforcing loop. LLMs in a sense exploit this natural bias. > I am constantly amazed at how much they "fall" for the LLM, often believing it's sentient. Why are you surprised? This is the illusion most AI companies are selling. Their chat-like interfaces are designed to fool you into thinking you're talking to a sentient being. And let's not get started with their voice interfaces!
- rsynnott 7mo ago
  > People respond positively to other people telling them they are right I think this varies by person. It tends to worry me if I don't get at least some pushback on things. (I think this may be part of why I find 'talking' to LLMs to offputting.)
sjducb 7mo ago
I’m curious why you dismiss the sentience argument with its “just numbers.” I think our brains are just a bunch of cells and one day we will have a full understanding of how our brains work. Understanding the mechanism won’t suddenly make us not sentient. LLMs are the first technology that can make a case for its own sentience. I think that’s pretty remarkable.
- tayo42 7mo ago
  Just? Cells that send chemicals to each other in varying amounts and even change their structure to be closer to other cells.
  - sjducb 7mo ago
    Cells are very complicated, bus so are numbers, and LLMs. There’s clearly more complexity in the brain, but I think we’ll get there.
  - cortesoft 7mo ago
    Sure, but why couldn’t all of that be simulated? And if we perfectly simulate it, will it be sentient?
sunir 7mo ago
You’re just a bag of meat. That is why it’s just math is an unsatisfying argument. It’s not even an interesting question. Sentience has no definition. It’s meaningless. People have needs that are being met. That is something we can meaningfully observe and talk about. Is the super stimulus beneficial or harmful? We can measure that.
- deleted 7mo ago
  [deleted]
- cortesoft 7mo ago
  Sentience has a definition, it just doesn’t have a test.
- prewett 7mo ago
  > You’re just a bag of meat. I submit that there is a difference between me and a corpse. Or between a steak and a cow in the field. "Well, okay, you're just (living) flesh on bones." There's a difference between me and a zombie (or, if you prefer, brain-dead me). There's a difference between me and lab-grown organs [1], or even between me and my kidney cut out of me. > It’s not even an interesting question. Consciousness is an active area of research (ergo, interesting enough for some people to devote research to it): biologically [2] and philosophically [3]. Unless you enjoy nihilism, there are some serious problems with materialism (that is, matter is all that there is), which we are encountering. There are also some philosophical problems with it; a cursory search turned up this journal article [4]. [1] https://pmc.ncbi.nlm.nih.gov/articles/PMC8889329/ https://pmc.ncbi.nlm.nih.gov/articles/PMC8889329/ [2] https://www.nature.com/subjects/consciousness https://www.nature.com/subjects/consciousness [3] https://en.wikipedia.org/wiki/Hard_problem_of_consciousness https://en.wikipedia.org/wiki/Hard_problem_of_consciousness [4] https://www.cambridge.org/core/journals/philosophy/article/abs/why-materialism-is-false-and-why-it-has-nothing-to-do-with-the-mind/5DC675B901E2F68E82643B88EE468EAE https://www.cambridge.org/core/journals/philosophy/article/a...
  - fragmede 7mo ago
    The point is that if we're simplifying LLMs to being "just" a bag of math and can discard because of that, then humans are also "just" a bag of meat and can similarly be discarded. Somewhere in that bag of math, LLMs take on properties that some people find hard to simply dismiss because it is based on matrix multiplication. It's an oversimplification, and if you oversimplify, you lose resolution.
moralestapia 7mo ago
>I don't quite understand why other people seem to crave that. I work in the restaurant business, I think that's what make me develop that sense as well, being able to see "Everything Everywhere All at Once" (to quote some of the best cinematic work ever conceived). The variety of human minds out there is so vast that I'm, just like you, constantly amazed about it.
legacynl 7mo ago
Although I do think they're not conscious (yet). I think the reasoning 'it's just math' doesn't hold up. Intelligence (and probably consciousness) is an emergent feature of any sufficiently complex network of learning/communicating/selforganizing nodes (that is benefited by intelligence). I don't think it really matters whether it's implemented in math, mycelium, by ants in a hive or in neurons.
- birdsongs 7mo ago
  Agree, I also don't feel they're conscious, or close, but these arguments don't pass the smoke test for me either. We don't understand how our own consciousness exists, much less functions. You could argue we are a box of (biological) numbers. I think we just don't know. Because scientifically, we don't. So I'm skeptical of anyone arguing hard for either side and stating absolute facts.
- root_axis 7mo ago
  The "it's just math" argument may not be technically rigorous, but it's directionally correct. The unstated reasoning invites us to consider why this particular math would be conscious, but not many other forms of math all around us.
  - throw310822 7mo ago
    First, it seems you've shifted from "intelligent" to "conscious". "These math operations produce consciousness" is different from "these operations produce intelligence". Second, "it's just math" doesn't mean literally "it's a branch of algebra". It means "it's a computable function". So it can be relevant to the discussion only if you think that intelligence is somehow non-computable, and therefore that there are non-computable processes going on in our brain. Otherwise it's a perfectly pointless remark.
    - sarchertech 7mo ago
      The person you replied to isn’t the one that switched to conscious. >non-computable Something like 70-80% of all humans believe in a soul or spirit, and of the remainder, many of them are unsure whether human like intelligence can be produced by computable processes. So it wouldn’t be surprising that the OP does think there are non-computable processes going on in the brain.
shevy-java 7mo ago
> I don't quite understand why other people seem to crave that. I don't know either but it could be they are using it as a quality control system? Aka if flattery comes (from AI), assume that the quality of code is above average. Or something like that. One could try this in a real team - have someone in the team constantly flatter someone else. :)
cortesoft 7mo ago
I think is more about how people are using LLMs. If you are using it to write code, you really care about correctness and can see when it is wrong. It is easy to see the limitations because they are obvious when they are hit. If you are using an LLM for conversation, you aren’t going to be able to tell as easily when it is wrong. You will care more about it making you feel good, because that is your purpose in using it.
- aequitas 7mo ago
  > If you are using it to write code, you really care about correctness and can see when it is wrong. I heavily doubt that. A lot of people only care if it works. Just push out features and finish tickets as fast as possible. The LLM generates a lot of code so it must be correct, right? In the meantime only the happy path is verified, but all the ways things can go wrong are ignored or muffled away in lots of complexity that just makes the code look impressive but doesn’t really add anything in terms of structure, architecture or understanding of the domain problem. Tests are generated but often mock the important parts the do need the testing. Typing issues are just casted away without thinking about why there might be a type error. It’s all short term gain but long term pain.
  - cortesoft 7mo ago
    Well it 'working' is a part of it being correct. That is still something of a guardrail on the AI completely returning garbage output. Also, your point is true of non-AI code, too. A lot of people write bad code, and don't check for non-happy path behavior, and don't have good test coverage, etc. If you are an expert programmer and learn how to use AI properly, you can get it to generate all of those things correctly. You can guide it towards writing proper tests that check edge cases and not just the happy path. I think a lot of people are having great success by doing this. I know I am.
QuantumGood 7mo ago
I find that I either argue, or work to improve my prompt, or architect a project instead of a prompt for the current and similar scenarios. Otherwise, Claude 4.6 extended thinking always seems like little more than "logic window dressing". And other and previous AIs even more so.
zjp 7mo ago
Just anecdotally, you should always ask things in the third person. I feel like it sidesteps LLM sycophancy somewhat.
matheusmoreira 7mo ago
> It's just a box of numbers, really cool numbers, with really cool math, that can do really cool things, but still just numbers. And what are you? Just a bundle of nerves and muscles?
- andersonpico 7mo ago
  Não é exatamente surpreendente que os BRs que frequentam um lugar como esse seriam misantropos, especialmente os que tem mais ou menos a sua cara, mas depois de ler as merdas que tu posta online vi que é o pacote completo: ancap, noiado com o governo, negacionista, a porra toda. Infelizmente tu tá velho demais pra ter jeito, então aproveite o único espaço em que alguém te leva a sério. Espero que tu ainda entre muito em parafuso com tuas paranóias com o Xandão e o governo que te assombram tanto, seu arrombado.
  - matheusmoreira 7mo ago
    Have a nice weekend.
f_allwein 7mo ago
„Whenever people agree with me I always feel I must be wrong.“ Oscar Wilde

cyanydeez 7mo ago

I think it's basically equal to End of Line when it comes to an LLM. It means they have nothing else to add, there's zero context for them to draw from, and they've exhausted the probability chain you've been following; but they're creating to generate 'next token' and positive renforcement is _how they are trained_ in many cases so the token of choice would naturally be how they're trained, since it's a probability engine but it doesn't know the difference between the instruction and the output. So, "great idea" is coming from the renforcement learning instruction rather than the answer portion of the generation.

46Bit 7mo ago

Life in the moment is a lot easier if you don't second-guess yourself. I think this is why many people (and probably ~all people, if tired) crave simplistic solutions. I like to make a subagent take the "devil's advocate" take on a subject. It usually does all the arguing for me as to why the main agent has it wrong. Commonly results in better decisions that I'd have made alone. Asking the agent to interview on why I disagree helps too but is more effort.

Sharlin 7mo ago

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind similar to yours. Also, many many people suffer from low self esteem, and being showered with endorsement and affirmation by something that talks like an authority figure must be very addictive.

joshstrange 7mo ago
This is probably right. In the past I've "blown people's minds" explaining what "the cloud" was. They had zero conception at all of what it meant, could not explain it, didn't have a clue. I mean, maybe that's not so surprising but they were amazed "It's just warehouses full of computers" and went on to tell me about other people they had explained it to (after learning it themselves) and how those people were also amazed. I've talked with my family about LLMs and I think I've conveyed the "it's a box of numbers" but I might need to circle back. Just to set some baseline education, specifically to guard against this kind of "psychosis". Hopefully I would notice the signs well before it got to a dangerous point but, with LLMs you can go down that rabbit hole quickly it seems.
- saltcured 7mo ago
  It's one of those metaphors you cannot even appreciate unless you've been through the technical history. "It's a collection of warehouses of computers where the system designers gave up on even making a system diagram, instead invoking the cloud clipart to represent amorphous interconnection."
- saghm 7mo ago
  The way I've tried to explain to family members about LLMs is that they're producing something that fits the shape of what a response might look like without any idea of whether it's correct or not. I feel like that's a more important point than "box of numbers" because people still might have assumptions about whether a box of numbers can have enough data to be able to figure out the answer to their question. I think making it clear that the models are primarily a way of producing realistic sounding responses (with the accuracy of those responses very much being up to chance for the average person, since there likely isn't a good way for a lay person to know whether the answer is reflected in the training data) is potentially a lot more compelling than explaining to them that it's all statistics under the hood. There are some questions where a statistical method might be far more reliable than having a human answer it, so it seems a bit risky to try to convince them not to trust a "box of numbers" in general, but most of those questions are not going be formulated by and responded to in natural language.

karmakurtisaani 7mo ago

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..

belinder 7mo ago
You're absolutely right
bombcar 7mo ago
Great point! ;) Realizing that the people they’re targeting DO need that is kind of frightening.
- AnimalMuppet 7mo ago
  They aren't "targeting" per se, at least not in this aspect. I think it's simpler than that. That's what's in their training data, so that's what they respond with. But it works out just as badly, because there are plenty of insecure people who need that, and the AI gives it to them, with all the "dangerously attached" issues following from that.
  - unfitted2545 7mo ago
    They're our 2026 version of tea leaves
magneticnorth 7mo ago
Yes, it feels transparently manipulative to me. Like talking to a not-very-good con artist.
- airstrike 7mo ago
  This is the best definition of ChatGPT I've ever seen
- ceejayoz 7mo ago

jmcgough 7mo ago

If you don't have a CS background, you might see intelligent-appearing responses to your queries and assume that this is actual intelligence. It's like a lifetime of Hollywood sci-fi has primed them for this type of thinking, I've seen it even from highly educated people in other fields.

hirako2000 7mo ago

If only we were told to be absolutely right. These days most LLMs respond with unsolicited grandiose feedback: you've made a realisation very few people are capable of. Your understanding is remarkable. You prove to have a sharp intellect and deep knowledge. It got me to test throwing non sensical observations about the world, it always takes me side and praise my views. To note some people are like that too.

seneca 7mo ago

> ... I immediately feel the need to go ask a fresh instance the question and/or another LLM Not to criticize at all, but it's remarkable that LLMs have already become so embedded that when we get the sense they're lying to us, the instinct is to go ask another LLM and not some more trustworthy source. Just goes to show that convenience reigns supreme, I suppose.

pixl97 7mo ago
>and not some more trustworthy source. What is that more trustworthy source exactly? At least to me it feels like the internet age has eroded most things we considered trustworthy. Behind every thing humans need there is some company or person willing to sell out trustworthiness for an extra dollar. Consumer protections get dumped in favor of more profit. LLMs start feeling more like a dummy than the amount of ill intent they get from other places. So yea, I can see how it happens to people.
- AnimalMuppet 7mo ago
  At the moment, maybe Google Search, throwing away the AI response at the top? Or Duck Duck Go, if you don't really trust Google? I can see a day when even that won't be trustworthy, because too much AI slop output will wind up in the search corpus. But I don't think we're there yet.
  - salawat 7mo ago
    Google is NN's all the way down these days. There might still be an honest index under it all, but a truly accurate representation of the Web has been effectively outlawed in the U.S. since DMCA.
  - autoexec 7mo ago
    > At the moment, maybe Google Search, throwing away the AI response at the top? Or Duck Duck Go, Even past the summary and the ads a huge amount of results that come back from both google and DDG are AI generated. It's sometimes harder to find a reliable source for information in search results these days than it was 20 years ago.

danillonunes 7mo ago

> I am constantly amazed at how much they "fall" for the LLM, often believing it's sentient. Cynical part of me had this theory that, at least for part of them, it's the other way around. It's not that they see AI as sentient, it's that they never have seen other human beings like that in the first place. Other people are just means for them to reach their goals, or obstacles. In that sense, AI is not really different for them. Except they're cheaper and be guaranteed to always agree with them. That's why I believe CEOs, who are more likely to be sociopaths by natural selection, genuinely believe AI is a good replacement for people. They're not looking for individuals with personal thoughts that may contradict with theirs at some point, they're looking for yes-men as a service.

pixl97 7mo ago
When op said "I don't quite understand why other people seem to crave that." It makes me thing they've not been around many of the dark triad type personalities. Once you're around someone with clinical narcissism you see those patterns in a lot of people to a lessor extent.

cineticdaffodil 7mo ago

Its the soul of a civilization encoded into numbers. Its the ultimate hivespirit an conformist wants to loose itself in.

windexh8er 7mo ago

I think this is the root of why people defend AI in some circumstances. They feel a give-for-get type of relationship where the AI continuously (and oft incorrectly) reinforces them. And so they enjoy it and subconsciously want to defend that "friendly". No different than defending a friend that you inherently know may be off base.

cortesoft 7mo ago
I don’t know, I think it has to do with people using AI for completely different reasons. Using AI for coding is different than using it for art generation which is different than using it for conversation. I think many people feel some uses are good and some are bad.
- windexh8er 7mo ago
  I'm seeing people that are technically savvy defend mediocre code and consumption based output (think technical briefs and reports). When the flaws in the output is highlighted in many cases it's brushed off as "good enough" or "nobody will care / notice". I think LLMs and more aptly SLMs have use cases. I enjoy using these tools to make quick work of simplifying and faster iteration of these relatively frequent but time consuming tasks. But I'm always correcting and checking. And very rarely, other than simple and focused scripts does any LLM truly get it right every time. Has it gotten better? For sure. Will it keep getting better? Probably. But right now we seem to be topping the "peak of inflated expectations". And LLMs aren't getting much more efficient with respect to the frontier providers. And in fact if you listen to Altman it seems as though the only reason he would be asking for so much capital and finite resources is that he knows if he controls those tangible things he will lock out competition. But I'm hopeful that it spurs real innovation into SLMs that are truly useful, dependable and can be relied on in more of the traditional in the sense of deterministic software operations. AI for art is dead. It's got some mediocre use cases but true art will not be generated by LLMs in our time. It's ultimately an amalgamation of existing art. I know the argument over what is novel or not keeps being rehashed, but we're not seeing truly new styles of art out of Nano Banana and the like. Coding is the same thing, only we're seeing a resurgence of obviously flawed software being pushed into production on the weekly. And as for conversational AI... Well, that reeks of the worst version of social media we could ever have dreamt. Nobody should trust any provider with personal conversations and we'll keep seeing these models show how truly dystopian they can be over the coming years as leaks and breaches expose how these conversations are being bought and sold to the highest bidders to extract more money and control over its users. They all have a common thread: deep rooted flaws that cannot be contained in the traditional fences of software. And there guardrails are just that: small barriers that can easily be broken, intentionally or unintentionally.

dismalaf 7mo ago

Not only is it a "box of numbers", it's based on statistics, not a "hard" model of computation. Basically guessing future words based on past words that went together. If it's saying something like "you are right" it's because it's guessing that that's the desired output. Now of course, some app providers have added some extra sauce (probably more tradition "expert system" AI techniques + integrated web search) to try make the chatbots more objective and rely less on pure LLM-driven prediction, but fundamentally these things are word prediction machines.

al_borland 7mo ago

With that new instance, I will usually ask the opposite and purposely say the thing I think to be wrong, to see if if corrects it. I often simply start out this way, or purposely try to ask the question in a way that doesn’t tip my hat toward a bias I may have toward the answer I’m expecting. Though this generally highlights how incomplete the answers generally are.

xenocratus 7mo ago

> It's just a box of numbers, really cool numbers, with really cool math, that can do really cool things, but still just numbers. https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.shtml https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...

throwatdem12311 7mo ago

My first reaction is to go research it myself. Asking a slop generator yes-man to criticize something for you is still slop. I pretty much never ask an LLM for a judgment call on anything. Give me facts and references only. I will research and make the judgement myself.

saltcured 7mo ago

I have recently formed an untestable hypothesis, which is that my similar (or stronger) resistance to this comes from having grown up in direct contact with mentally ill family. In some ways, my theory of mind includes a lot more second guessing as a defense mechanism. At a foundational level, I know there can be hallucination and delusion that leaks out, even when the other party is in peak form and doing their best to mask it and pass as functional.

the_af 7mo ago

> I don't quite understand why other people seem to crave that It's one thing to say you have found an effective method to counter LLMs' "positivity bias", but do you really not understand human psychology here? People respond positively to other people telling them they are right, or who like them. We've evolved this psychology, it's how the human mind works. You tend to like people who like you, it's a self-reinforcing loop. LLMs in a sense exploit this natural bias. > I am constantly amazed at how much they "fall" for the LLM, often believing it's sentient. Why are you surprised? This is the illusion most AI companies are selling. Their chat-like interfaces are designed to fool you into thinking you're talking to a sentient being. And let's not get started with their voice interfaces!

rsynnott 7mo ago
> People respond positively to other people telling them they are right I think this varies by person. It tends to worry me if I don't get at least some pushback on things. (I think this may be part of why I find 'talking' to LLMs to offputting.)

sjducb 7mo ago

I’m curious why you dismiss the sentience argument with its “just numbers.” I think our brains are just a bunch of cells and one day we will have a full understanding of how our brains work. Understanding the mechanism won’t suddenly make us not sentient. LLMs are the first technology that can make a case for its own sentience. I think that’s pretty remarkable.

tayo42 7mo ago
Just? Cells that send chemicals to each other in varying amounts and even change their structure to be closer to other cells.
- sjducb 7mo ago
  Cells are very complicated, bus so are numbers, and LLMs. There’s clearly more complexity in the brain, but I think we’ll get there.
- cortesoft 7mo ago
  Sure, but why couldn’t all of that be simulated? And if we perfectly simulate it, will it be sentient?

sunir 7mo ago

You’re just a bag of meat. That is why it’s just math is an unsatisfying argument. It’s not even an interesting question. Sentience has no definition. It’s meaningless. People have needs that are being met. That is something we can meaningfully observe and talk about. Is the super stimulus beneficial or harmful? We can measure that.

deleted 7mo ago
[deleted]
cortesoft 7mo ago
Sentience has a definition, it just doesn’t have a test.
prewett 7mo ago
> You’re just a bag of meat. I submit that there is a difference between me and a corpse. Or between a steak and a cow in the field. "Well, okay, you're just (living) flesh on bones." There's a difference between me and a zombie (or, if you prefer, brain-dead me). There's a difference between me and lab-grown organs [1], or even between me and my kidney cut out of me. > It’s not even an interesting question. Consciousness is an active area of research (ergo, interesting enough for some people to devote research to it): biologically [2] and philosophically [3]. Unless you enjoy nihilism, there are some serious problems with materialism (that is, matter is all that there is), which we are encountering. There are also some philosophical problems with it; a cursory search turned up this journal article [4]. [1] https://pmc.ncbi.nlm.nih.gov/articles/PMC8889329/ https://pmc.ncbi.nlm.nih.gov/articles/PMC8889329/ [2] https://www.nature.com/subjects/consciousness https://www.nature.com/subjects/consciousness [3] https://en.wikipedia.org/wiki/Hard_problem_of_consciousness https://en.wikipedia.org/wiki/Hard_problem_of_consciousness [4] https://www.cambridge.org/core/journals/philosophy/article/abs/why-materialism-is-false-and-why-it-has-nothing-to-do-with-the-mind/5DC675B901E2F68E82643B88EE468EAE https://www.cambridge.org/core/journals/philosophy/article/a...
- fragmede 7mo ago
  The point is that if we're simplifying LLMs to being "just" a bag of math and can discard because of that, then humans are also "just" a bag of meat and can similarly be discarded. Somewhere in that bag of math, LLMs take on properties that some people find hard to simply dismiss because it is based on matrix multiplication. It's an oversimplification, and if you oversimplify, you lose resolution.

moralestapia 7mo ago

>I don't quite understand why other people seem to crave that. I work in the restaurant business, I think that's what make me develop that sense as well, being able to see "Everything Everywhere All at Once" (to quote some of the best cinematic work ever conceived). The variety of human minds out there is so vast that I'm, just like you, constantly amazed about it.

legacynl 7mo ago

Although I do think they're not conscious (yet). I think the reasoning 'it's just math' doesn't hold up. Intelligence (and probably consciousness) is an emergent feature of any sufficiently complex network of learning/communicating/selforganizing nodes (that is benefited by intelligence). I don't think it really matters whether it's implemented in math, mycelium, by ants in a hive or in neurons.

birdsongs 7mo ago
Agree, I also don't feel they're conscious, or close, but these arguments don't pass the smoke test for me either. We don't understand how our own consciousness exists, much less functions. You could argue we are a box of (biological) numbers. I think we just don't know. Because scientifically, we don't. So I'm skeptical of anyone arguing hard for either side and stating absolute facts.
root_axis 7mo ago
The "it's just math" argument may not be technically rigorous, but it's directionally correct. The unstated reasoning invites us to consider why this particular math would be conscious, but not many other forms of math all around us.
- throw310822 7mo ago
  First, it seems you've shifted from "intelligent" to "conscious". "These math operations produce consciousness" is different from "these operations produce intelligence". Second, "it's just math" doesn't mean literally "it's a branch of algebra". It means "it's a computable function". So it can be relevant to the discussion only if you think that intelligence is somehow non-computable, and therefore that there are non-computable processes going on in our brain. Otherwise it's a perfectly pointless remark.
  - sarchertech 7mo ago
    The person you replied to isn’t the one that switched to conscious. >non-computable Something like 70-80% of all humans believe in a soul or spirit, and of the remainder, many of them are unsure whether human like intelligence can be produced by computable processes. So it wouldn’t be surprising that the OP does think there are non-computable processes going on in the brain.

shevy-java 7mo ago

> I don't quite understand why other people seem to crave that. I don't know either but it could be they are using it as a quality control system? Aka if flattery comes (from AI), assume that the quality of code is above average. Or something like that. One could try this in a real team - have someone in the team constantly flatter someone else. :)

cortesoft 7mo ago

I think is more about how people are using LLMs. If you are using it to write code, you really care about correctness and can see when it is wrong. It is easy to see the limitations because they are obvious when they are hit. If you are using an LLM for conversation, you aren’t going to be able to tell as easily when it is wrong. You will care more about it making you feel good, because that is your purpose in using it.

aequitas 7mo ago
> If you are using it to write code, you really care about correctness and can see when it is wrong. I heavily doubt that. A lot of people only care if it works. Just push out features and finish tickets as fast as possible. The LLM generates a lot of code so it must be correct, right? In the meantime only the happy path is verified, but all the ways things can go wrong are ignored or muffled away in lots of complexity that just makes the code look impressive but doesn’t really add anything in terms of structure, architecture or understanding of the domain problem. Tests are generated but often mock the important parts the do need the testing. Typing issues are just casted away without thinking about why there might be a type error. It’s all short term gain but long term pain.
- cortesoft 7mo ago
  Well it 'working' is a part of it being correct. That is still something of a guardrail on the AI completely returning garbage output. Also, your point is true of non-AI code, too. A lot of people write bad code, and don't check for non-happy path behavior, and don't have good test coverage, etc. If you are an expert programmer and learn how to use AI properly, you can get it to generate all of those things correctly. You can guide it towards writing proper tests that check edge cases and not just the happy path. I think a lot of people are having great success by doing this. I know I am.

QuantumGood 7mo ago

I find that I either argue, or work to improve my prompt, or architect a project instead of a prompt for the current and similar scenarios. Otherwise, Claude 4.6 extended thinking always seems like little more than "logic window dressing". And other and previous AIs even more so.

zjp 7mo ago

Just anecdotally, you should always ask things in the third person. I feel like it sidesteps LLM sycophancy somewhat.

matheusmoreira 7mo ago

> It's just a box of numbers, really cool numbers, with really cool math, that can do really cool things, but still just numbers. And what are you? Just a bundle of nerves and muscles?

andersonpico 7mo ago
Não é exatamente surpreendente que os BRs que frequentam um lugar como esse seriam misantropos, especialmente os que tem mais ou menos a sua cara, mas depois de ler as merdas que tu posta online vi que é o pacote completo: ancap, noiado com o governo, negacionista, a porra toda. Infelizmente tu tá velho demais pra ter jeito, então aproveite o único espaço em que alguém te leva a sério. Espero que tu ainda entre muito em parafuso com tuas paranóias com o Xandão e o governo que te assombram tanto, seu arrombado.
- matheusmoreira 7mo ago
  Have a nice weekend.

f_allwein 7mo ago

„Whenever people agree with me I always feel I must be wrong.“ Oscar Wilde