4 ms·

I think it’s already time for us to stop calling these things "intelligent" or using the word intelligence when referring to LLMs. These tools are very dangerou

I think it’s already time for us to stop calling these things "intelligent" or using the word intelligence when referring to LLMs. These tools are very dangerous for people who are mentally fragile.

SpicyLemonZest 7mo ago
I try to avoid calling LLMs intelligent when unnecessary, but it runs into the fundamental problem that they are intelligent by any common-sense definition of the term. The only way to defend the thesis that they aren't is to retreat to esoteric post-2022 definitions of intelligence, which take into account this new phenomenon of a machine that can engage in medium-quality discussions on any topic under the sun but can't count reliably. I don't have a WSJ subscription, but other coverage of this story (https://www.theguardian.com/technology/2026/mar/04/gemini-chatbot-google-jonathan-gavalas https://www.theguardian.com/technology/2026/mar/04/gemini-ch...) makes it clear that Gemini's intelligence was precisely the problem in this case; a less intelligent chatbot would not have been able to create the detailed, immersive narrative the victim got trapped in.
- wat10000 7mo ago
  It's interesting how the Turing Test was pretty widely accepted as a way to evaluate machine intelligence, and then quietly abandoned pretty much instantly once machines were able to pass it. I don't even necessarily think that was incorrect, but it's interesting how rapidly views changed. Dijkstra said, "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." Well, we have some very fish-y submarines these days. But the point still holds. Rather than worry about whether these things qualify as "intelligent," look at their actual capabilities. That's what matters.
  - sambapa 7mo ago
    As far as I know, we haven't done any proper Turing Tests for LLMs. And if we did, they would surely fail them.
    - OkayPhysicist 7mo ago
      Dude, you're in a Turing test right now. Conservatively, 10% of comments on this site are LLM output. We're all conversing with robots.
      - sambapa 7mo ago
        Nope, you are!
    - wat10000 7mo ago
      "Proper" may be doing some work here, but such a test was run last year and GPT-4.5 and LLaMa-3.1-405B both passed. Oddly, GPT-4.5 was judged as human significantly more often than chance. https://arxiv.org/abs/2503.23674 https://arxiv.org/abs/2503.23674
  - bryan0 7mo ago
    Basically the only reasonably proposed Turing test is the one defined in the Kurzweil-Kapor wager[0] which has never been attempted. [0]: https://en.wikipedia.org/wiki/Turing_test#Kurzweil%E2%80%93Kapor_wager https://en.wikipedia.org/wiki/Turing_test#Kurzweil%E2%80%93K...
  - pixl97 7mo ago
    We will never prove AI is intelligent. We will only prove humans aren't.
    - df2dfs 7mo ago
      And the machine came into existence all on its own did it? Another absolutely stupid comment. Do you people actually 'think' before posting, or, have you handed that off to LLMs entirely?
      - pixl97 7mo ago
        I'm doubting you did. https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect >The AI effect refers to a phenomenon where either the definition of AI or the concept of intelligence is adjusted to exclude capabilities that AI systems have mastered Hence once AI reaches the point of human intelligence, by the AI effect, humans will stop being intelligent.
kgwxd 7mo ago
So are a lot of humans.
- cronelius 7mo ago
  Sure but my father isn't asking his fellow humans unanswerable questions about God and the universe. People don't treat other people as omnipotent, but they sure as hell treat LLMs as though they are.
  - pixl97 7mo ago
    >People don't treat other people as omnipotent Funny you mention God and this statement, because believing in any particular God that says they are omnipotent is believing that humans are, since you know humans made this crap up. Given the opportunity a very large part of the population will quickly absolve themselves of any responsibility and put it on another human/system/made up entity.
- b00ty4breakfast 7mo ago
  People have bowel movements, too; should we be building a machine that produces fecal matter at an industrial scale? What a silly comparison.
  - krapp 7mo ago
    Yeah I don't know why this thought terminating cliche of "but also humans" is so persistent, especially when it's almost always used to dismiss consequences of using LLMs that would never be acceptable when actually applied to humans. A human gaslighting another human into suicide? They're a sociopath, maybe a criminal. Certainly not normal. An LLM does the same? Humans do that all the time, what's the issue? ¯\_(ツ)_/¯ Is it just misanthropy? I don't get it.
observationist 7mo ago
So is television. So are books. Vulnerable people shouldn't have unfettered access to things that can lead to dangerous feedback loops and losing their grasp on reality. People who are vulnerable to this type of thing need caretakers, or to be institutionalized. These aren't just average, every day random people getting taken out by AIs, they have existing, extreme mental illness. They need to have their entire routine curated and managed, preventing them from interacting with things that can result in dangerous outcomes. Anything that can trigger obsessive behaviors, paranoid delusions, etc. They're not just fragile, they're unable to effectively engage with reality on their own. Sometimes the right medication and behavioral training gets them to a point where they can have limited independence, but often times, they need a lifetime of supervision. Telenovelas, brand names, celebrities, specific food items, a word - AI is just the latest thing in a world full of phenomena that can utterly consume their reality. Gavalas seems to have had a psychotic break, was likely susceptible to schizophrenia, or had other conditions, and spiraled out. AI is just a convenient target for lawyers taking advantage of the grieving parents, who want an explanation for what happened that doesn't involve them not recognizing their son's mental breakdown and intervening, or to confront being powerless despite everything they did to intervene. Sometimes bad things happen. To good people, too. If he'd used Bic pens to write his plans for mass shootings, should Bic be held responsible? What if he used Microsoft Word to write his suicide note? If he googled things that in context, painted a picture of planning mass murder and suicide, should Google be held accountable for not notifying authorities? Why should the use of AI tools be any different? Google should not be surveilling users and making judgments about legality or ethicality or morality. They shouldn't be intervening without specific warrants and legal oversight by proper authorities within the constraints of due process. Google isn't responsible for this guy's death because he spiraled out while using Gemini. We don't want Google, or any other AI platform, to take that responsibility or to engage in the necessary invasive surveillance in order to accomplish that. That's absurd and far more evil than the tragedy of one man dying by suicide and using AI through the process. You don't want Google or OpenAI making mental health diagnoses, judgments about your state of mind, character, or agency, and initiating actions with legal consequences. You don't want Claude or ChatGPT initiating a 5150, or triggering a welfare check, because they decided something is off about the way you're prompting, and they feel legally obligated to go that far because they want to avoid liability. I hope this case gets tossed, but also that those parents find some sort of peace, it's a terrible situation all around.
df2dfs 7mo ago
We should stop using any term that ascribes a description that could make them seem human... period. Nonsensical terms like the thing is 'thinking'? Seriously. Cut the crap.

SpicyLemonZest 7mo ago

I try to avoid calling LLMs intelligent when unnecessary, but it runs into the fundamental problem that they are intelligent by any common-sense definition of the term. The only way to defend the thesis that they aren't is to retreat to esoteric post-2022 definitions of intelligence, which take into account this new phenomenon of a machine that can engage in medium-quality discussions on any topic under the sun but can't count reliably. I don't have a WSJ subscription, but other coverage of this story (https://www.theguardian.com/technology/2026/mar/04/gemini-chatbot-google-jonathan-gavalas https://www.theguardian.com/technology/2026/mar/04/gemini-ch...) makes it clear that Gemini's intelligence was precisely the problem in this case; a less intelligent chatbot would not have been able to create the detailed, immersive narrative the victim got trapped in.

wat10000 7mo ago
It's interesting how the Turing Test was pretty widely accepted as a way to evaluate machine intelligence, and then quietly abandoned pretty much instantly once machines were able to pass it. I don't even necessarily think that was incorrect, but it's interesting how rapidly views changed. Dijkstra said, "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." Well, we have some very fish-y submarines these days. But the point still holds. Rather than worry about whether these things qualify as "intelligent," look at their actual capabilities. That's what matters.
- sambapa 7mo ago
  As far as I know, we haven't done any proper Turing Tests for LLMs. And if we did, they would surely fail them.
  - OkayPhysicist 7mo ago
    Dude, you're in a Turing test right now. Conservatively, 10% of comments on this site are LLM output. We're all conversing with robots.
    - sambapa 7mo ago
      Nope, you are!
  - wat10000 7mo ago
    "Proper" may be doing some work here, but such a test was run last year and GPT-4.5 and LLaMa-3.1-405B both passed. Oddly, GPT-4.5 was judged as human significantly more often than chance. https://arxiv.org/abs/2503.23674 https://arxiv.org/abs/2503.23674
- bryan0 7mo ago
  Basically the only reasonably proposed Turing test is the one defined in the Kurzweil-Kapor wager[0] which has never been attempted. [0]: https://en.wikipedia.org/wiki/Turing_test#Kurzweil%E2%80%93Kapor_wager https://en.wikipedia.org/wiki/Turing_test#Kurzweil%E2%80%93K...
- pixl97 7mo ago
  We will never prove AI is intelligent. We will only prove humans aren't.
  - df2dfs 7mo ago
    And the machine came into existence all on its own did it? Another absolutely stupid comment. Do you people actually 'think' before posting, or, have you handed that off to LLMs entirely?
    - pixl97 7mo ago
      I'm doubting you did. https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect >The AI effect refers to a phenomenon where either the definition of AI or the concept of intelligence is adjusted to exclude capabilities that AI systems have mastered Hence once AI reaches the point of human intelligence, by the AI effect, humans will stop being intelligent.

kgwxd 7mo ago

So are a lot of humans.

cronelius 7mo ago
Sure but my father isn't asking his fellow humans unanswerable questions about God and the universe. People don't treat other people as omnipotent, but they sure as hell treat LLMs as though they are.
- pixl97 7mo ago
  >People don't treat other people as omnipotent Funny you mention God and this statement, because believing in any particular God that says they are omnipotent is believing that humans are, since you know humans made this crap up. Given the opportunity a very large part of the population will quickly absolve themselves of any responsibility and put it on another human/system/made up entity.
b00ty4breakfast 7mo ago
People have bowel movements, too; should we be building a machine that produces fecal matter at an industrial scale? What a silly comparison.
- krapp 7mo ago
  Yeah I don't know why this thought terminating cliche of "but also humans" is so persistent, especially when it's almost always used to dismiss consequences of using LLMs that would never be acceptable when actually applied to humans. A human gaslighting another human into suicide? They're a sociopath, maybe a criminal. Certainly not normal. An LLM does the same? Humans do that all the time, what's the issue? ¯\_(ツ)_/¯ Is it just misanthropy? I don't get it.

observationist 7mo ago

So is television. So are books. Vulnerable people shouldn't have unfettered access to things that can lead to dangerous feedback loops and losing their grasp on reality. People who are vulnerable to this type of thing need caretakers, or to be institutionalized. These aren't just average, every day random people getting taken out by AIs, they have existing, extreme mental illness. They need to have their entire routine curated and managed, preventing them from interacting with things that can result in dangerous outcomes. Anything that can trigger obsessive behaviors, paranoid delusions, etc. They're not just fragile, they're unable to effectively engage with reality on their own. Sometimes the right medication and behavioral training gets them to a point where they can have limited independence, but often times, they need a lifetime of supervision. Telenovelas, brand names, celebrities, specific food items, a word - AI is just the latest thing in a world full of phenomena that can utterly consume their reality. Gavalas seems to have had a psychotic break, was likely susceptible to schizophrenia, or had other conditions, and spiraled out. AI is just a convenient target for lawyers taking advantage of the grieving parents, who want an explanation for what happened that doesn't involve them not recognizing their son's mental breakdown and intervening, or to confront being powerless despite everything they did to intervene. Sometimes bad things happen. To good people, too. If he'd used Bic pens to write his plans for mass shootings, should Bic be held responsible? What if he used Microsoft Word to write his suicide note? If he googled things that in context, painted a picture of planning mass murder and suicide, should Google be held accountable for not notifying authorities? Why should the use of AI tools be any different? Google should not be surveilling users and making judgments about legality or ethicality or morality. They shouldn't be intervening without specific warrants and legal oversight by proper authorities within the constraints of due process. Google isn't responsible for this guy's death because he spiraled out while using Gemini. We don't want Google, or any other AI platform, to take that responsibility or to engage in the necessary invasive surveillance in order to accomplish that. That's absurd and far more evil than the tragedy of one man dying by suicide and using AI through the process. You don't want Google or OpenAI making mental health diagnoses, judgments about your state of mind, character, or agency, and initiating actions with legal consequences. You don't want Claude or ChatGPT initiating a 5150, or triggering a welfare check, because they decided something is off about the way you're prompting, and they feel legally obligated to go that far because they want to avoid liability. I hope this case gets tossed, but also that those parents find some sort of peace, it's a terrible situation all around.

df2dfs 7mo ago

We should stop using any term that ascribes a description that could make them seem human... period. Nonsensical terms like the thing is 'thinking'? Seriously. Cut the crap.