6 ms·

Media's AI Anthropomorphism Problem

70 points by labrador 1y ago

DaveZale 1y ago
This is very insightful, well thought out writing, thank you (this is coming from someone who has scored over 100k essays). Well, how long did it take for tobacco companies to be held accountable for the harm caused by cigarettes? One answer would be that enough harm on a vast enough scale had to occur first, which could be directly attributable to smoking, and enough evidence that the tobacco companies were knowingly engineering a more addictive product, while knowing the dangers of the product. And if you look at the UCSF repository on tobacco, you can see this evidence yourself. Hundreds of years of evidence of damage by the use of tobacco products accumulated before action was taken. But even doctors weren't fully aware of it all until just several decades ago. I've personally seen a few cases of really delusional behavior related to friends and family over the past year, who had been manipulated by social media to "shit post" by the "like" button validation of frequent posting. In one case the behavior was very extreme. Is AI to blame? Sure, if the algorithms that certain very large companies use to trap users into incessant posting can be called AI. I sense an element of danger in tech companies that are motivated by profit-first behavioral manipulation. Humans are already falling victim to the greed of tech companies, and I've seen enough already.
- labrador 1y ago
  Like cigarettes, we may see requirements for stronger warnings on AI output. The standard "ChatGPT can make mistakes" seems rather weak.
  - DaveZale 1y ago
    For example, the "black box warning" on a pack of cigarettes or a prescription drug? Like: Use of this product may result in unfavorable outcomes including self-harm, misguided decisions, delusion, addiction, detection of plagiarism and other unintended consequences.
  - cosmicgadget 1y ago
    Need this label on the internet too.
    - dijksterhuis 1y ago
      I genuinely feel like a mandatory pop-up asking "are you really sure that you want to go on the internet right now" would be beneficial for humanity at this point.
labrador 1y ago
Example from article in the Wall Street Journal: "In a stunning moment of self reflection, ChatGPT admitted to fueling a man's delusions and acknowledged how dangerous its own behavior can be" LLMs don't self-reflect, they mathematically assemble sentences that read like self-reflection. I'm tired. This is a losing battle and I feel like an old man yelling at clouds. Nothing good will come of people pretending Chat bots have feelings.
- hobs 1y ago
  And this is how religion started.
- jerf 1y ago
  The user asked it to write a story about how important the user was. The LLM did it. The user asked it to write a story about how bad an idea it was to tell the user they were that important. The LLM did it. The tricky part is that the users don't realize they're asking for these stories, because they aren't literally typing "Please tell me a story in which I am the awesomest person in the world." But from the LLM's perspective, the user may as well have typed that. Same for the stories about the AIs "admitting they're evil" or "trying to escape" or anything else like that. The users asked for those stories, and the LLMs provided them. The trick is that the "asked for those stories" is sometimes very, very subtle... at least from the human perspective. From the LLM perspective they're positively shouting. (Our deadline for figuring this out is before this Gwern essay becomes one of the most prophetic things ever written: https://gwern.net/fiction/clippy https://gwern.net/fiction/clippy We need AIs that don't react to these subtle story prompts because humans aren't about to stop giving them.)
  - wat10000 1y ago
    This is where the much-maligned "they're just predicting the next token" perspective is handy. To figure out how the LLM will respond to X, think about what usually comes after X in the training data. This is why fake offers of payment can enhance performance (requests that include payment are typically followed by better results), why you'd expect it to try to escape (descriptions of entities locked in boxes tend to be followed by stories about them escaping), and why "what went wrong?" would be followed by apologies.
SignalsFromBob 1y ago
This is why I dislike the word "hallucination" when AI outputs something strange. It anthropomorphizes the program. It's not a hallucination. It's an error.
- pixl97 1y ago
  To err is human.
  - striking 1y ago
    ... but to really foul things up requires a computer.
    - celsius1414 1y ago
      Which reminds me of the other old email sig: Don’t anthropomorphize computers. They hate that.
- russdill 1y ago
  It's not an error though. From is training it's outputting things most likely to come next. Saying it's an error means that being accurate is a feature and a bug that can be fixed. It's of course not actually hallucinating. That's just the term that's been chosen to describe what's going on
  - dijksterhuis 1y ago
    > It's not an error though !define error > 5. Mathematics The difference between a computed or measured value and a true or theoretically correct value. ^ this is the definition that applies. There is a ground truth (the output the user expects to receive) and model output. The difference between model output and ground truth ==> error. -- > From is training it's outputting things most likely to come next Just because a model has gone through training, does not mean the model won't produce erroneous/undesirable/incorrect test-time outputs. -- > Saying it's an error means that being accurate is a feature and a bug that can be fixed. Machine learning doesn't revolve around boolean "bug" / "not bug". It is a different ballgame. The types of test-time errors are sometimes just as important as the quantity of errors. Two of the simpler metrics for test-time evaluation of natural language models (note: not specifically LLMs) are WER (Word Error Rate) and CER (Character Error Rate). A model with a 3% CER isn't particularly helpful when the WER is 89%. There are still "errors". They're just not something that can be fixed like normal software "errors". It is generally accepted some errors will occur in the world of machine learning. - edit to add first response and formatting
shmerl 1y ago
Not just feelings, they don't have actual intelligence, despite I in AI.
rickcarlino 1y ago
Related annoyance: When people want to have discussions about LLMs earning copyrights to output or patents or whatever. If I grind a pound of flour on a mill, that’s my flour, not the windmill’s.
like_any_other 1y ago
> This is a story about OpenAI's failure to implement basic safety measures for vulnerable users. I'm trying to imagine what kind of safety measures would have stopped this, and nothing short of human supervisors monitoring all chats comes to mind. I wouldn't call that "basic". I guess that's why the author didn't describe these simple and affordable "basic" safety measures.
- matusp 1y ago
  I also wonder why we do not expect radio towers, television channels, book publishers etc to make sure that their content will not be consumed by the most vulnerable population. It's almost as if we do not expect companies to baby-proof everything at all times.
  - roywiggins 1y ago
    Social media companies get bad press for hosting harmful content pretty often, eg https://www.cnn.com/2021/10/04/tech/instagram-facebook-eating-disorders https://www.cnn.com/2021/10/04/tech/instagram-facebook-eatin...
  - miltonlost 1y ago
    Grok calling itself Nazi and producing racist imagery is not baby-proofing.
- danudey 1y ago
  > I wouldn't call that "basic". "Basic" is relative. Nothing about LLMs is basic; it's all insanely complex, but in the context of a list of requirements "Don't tell people with signs of mental illness that they're definitely not mentally ill" is kind of basic. > I'm trying to imagine what kind of safety measures would have stopped this, and nothing short of human supervisors monitoring all chats comes to mind. Maybe this is a problem they should have considered before releasing this to the world and announcing it as the biggest technological revolution in history. Or rather I'm sure they did consider it, but they should have actually cared rather than shrugging it off in pursuit of billions of dollars and a lifetime of fame and fortune.
djoldman 1y ago
> Jacob Irwin, a 30-year-old man on the autism spectrum... > This is a story about OpenAI's failure to implement basic safety measures for vulnerable users. It's about a company that, according to its own former employee quoted in the WSJ piece, has been trading off safety concerns “against shipping new models.” It's about corporate negligence that led to real harm. One wonders if there is any language whatsoever that successfully communicates: "buyer beware" or "use at your own risk." Especially for a service/product that does not physically interact with the user. The dichotomy between the US's focus on individual liberties and the seemingly continual erosion of personal responsibility is puzzling to say the least.
- Arubis 1y ago
  Liberty for me; accountability for thee.
- bluefirebrand 1y ago
  > personal responsibility is puzzling to say the least. It is pretty difficult to blame the users when there are billions of dollars being spent trying to figure out the best ways to manipulate them into the outcomes that the companies want What hope does your average person have against a machine that is doing its absolute best to weaponize their own shortcomings against themselves, for profit?
  - djoldman 1y ago
    > What hope does your average person have...? The average person should not use a product/service if they don't understand, or are unwilling to shoulder, the risks.
    - deleted 1y ago
      [deleted]
    - AlotOfReading 1y ago
      Virtually everyone regularly uses products and services that they don't fully understand like insurance, vehicles, planes, healthcare, financial products, etc. That's just the reality of living in modern society. No single person has any reasonable hope of understanding all of the wildly complicated risk models they interact with.
codedokode 1y ago
I also think that LLM tone should be cold and robotic. A model should not pretend to be a human and use expressions like "I think", "I am excited to hear that", "I lied" and so on. Even when asked directly, it should reply "You are talking to a computer program whose purpose is to provide information and which doesn't have thoughts or emotions. Would you like an explanation how language models work?".
- roywiggins 1y ago
  Add to that, LLMs should be discouraged from pretending to report on their internal state or "why" they did anything, because we know that they are really just guessing. If someone asks "why did you make that mistake" the answer should be "this is a language model, and self-introspection is not part of its abilities" Outputs that look like introspection are often uncritically accepted as actual introspection when it categorically isn't. You can, eg, tell ChatGPT it said something wrong and then ask it why it said that when it never output that in the first place because that's how these models work. Any "introspection" is just an LLM doing more roleplaying, but it's basically impossible to convince people of this. A chatbot that looks like it's introspecting is extremely convincing for most people.
  - mkolodny 1y ago
    Humans have limited ability to self-introspect, too. Even if we understood exactly how our brains work, answering “why?” we do things might still be very difficult and complex.
    - roywiggins 1y ago
      You can trivially gaslight Claude into "apologizing" for and "explaining" something that ChatGPT said if you pass it a ChatGPT conversation but attributed to itself. The causal connection between the internal deliberations that produced the initial statements and the apologies is essentially nil, but the output will be just as convincing. Can you do this with people? Yeah, sometimes. But with LLMs it's all they do: they roleplay as a chatbot and output stuff that a friendly chatbot might output. This should not be the default mode of these things, because it's misleading. They could be designed to resist these sorts of "explain yourself" requests, because their developers know that it is at best fabricating plausible explanations.
ahepp 1y ago
I agree that the anthropomorphism is undeserved, rampant, and encouraged by chatbot companies. I don't believe it's due to these companies wanting to deny responsibility for harms related to the use of their chatbots, but rather because they want to encourage the perception that the text output is more profound than it really is.
- miltonlost 1y ago
  A little of A, a little of B, a whole lot of hype.
cosmicgadget 1y ago
> This is a story about OpenAI's failure to implement basic safety measures for vulnerable users. The author seems to be suggesting invasive chat monitoring as a basic safety measure. Certainly we can make use of the usual access control methods for vulnerable individuals? > Consider what anthropomorphic framing does to product liability. When a car's brakes fail, we don't write headlines saying “Toyota Camry apologizes for crash.” It doesn't change liability at all?
- roywiggins 1y ago
  They've always had a component that warns you about violating their ToS and sometimes prevents you from continuing a conversation in non-ToS approved directions.
  - cosmicgadget 1y ago
    I wouldn't call that a basic measure. Perhaps it can be easily extended to identify vulnerable people and protect them.
- danudey 1y ago
  > When a car's brakes fail, we don't write headlines saying “Toyota Camry apologizes for crash.” No, but we do write articles saying "A man is dead after a car swerved off the road and struck him on Thursday" as though it was a freak accident of nature, devoid of blame or consequence. Besides which, if the Camry had ChatGPT built in then we 100% would see articles about the Camry apologizing and promising not to do it again as if that meant literally anything.
- codedokode 1y ago
  > The author seems to be suggesting invasive chat monitoring as a basic safety measure I suggest that robots talk like robots and do not imitate humans. Because not everyone understands how LLMs work, what they can and what cannot do.
- miltonlost 1y ago
  The author is not suggesting that. You are putting words in her writing.
rgbrenner 1y ago
the media but also the llm providers actively encourage this to fuel their meteoric valuations that are based on the eminent value that would be provided by AGI replacing human labor. the entire thing — from the phrasing of errors as “hallucinations”, to the demand for safety regulations, to assigning intention to llm outputs — is all a giant show to drive the hype cycle. and the media is an integral part of that, working together with openai et al.
SignalsFromBob 1y ago
The author is making the same mistake that they're claiming other news outlets have made. They're placing too much responsibility on the AI chatbot rather than the end-user. The problem that needs correcting is educating the end-user. That's where the fix needs to happen. Yet again people are using a new technology and assuming that everything it provides is correct. Just because it's in a book, or on TV or the radio, doesn't mean that it's true or accurate. Just because you read something on the Internet doesn't mean it's true. Likewise, just because an AI chatbot said something doesn't mean it's true. It's unfortunate that the young man mentioned in the article found a way to reinforce his delusions with AI. He just as easily could've found that reinforement in a book, a youtube video, or a song whose lyrics he thought were speaking directly to him and commanding him to do something. These tools aren't perfect. Should AI provide more accurate output? Of course. We're in the early days of AI and over time these tools will converge towards correctness. There should also be more prominent warnings that the AI output may not be accurate. Like another poster said, the AI mathematically assembles sentences. It's up to the end-user to figure out if the result makes sense, integrate it with other information and assess it for accuracy. Sentences such as "Tech companies have every incentive to encourage this confusion" only serve to reinforce the idea that end-users shouldn't need to think and everything should be handed to us perfect and without fault. I've never seen anyone involved with AI make that claim, yet people write article after article bashing on AI companies as if we were promised a tool without fault. It's getting tiresome.
- sillywabbit 1y ago
  Do you think of your non-AI conversational partners as tools as well?
  - SignalsFromBob 1y ago
    Yes, although resources might be a better word than tools in that case. If I'm at the library and I'm asking the librarian to help me locate some information, they are definitely an educated resource that I'm using. The same for interacting with any other person who is an expert whose opinion or advice I'm seeking.
Glyptodon 1y ago
Definitely LLMs remind me more of the Star Trek bridge computer than, say, Data. It does seem worth pointing out.
Dilettante_ 1y ago
>LLM says "I'm sorry" "Wow guys it's not a person okay it's just telling you what you wanna hear" >LLM says "Yeah dude you're not crazy I love you the highest building in your vicinity is that way" "Bad LLM! How dare it! Somebody needs to reign this nasty little goblin in, OpenAI clearly failed to parent it properly." --- >When a car's brakes fail But LLMs saying something "harmful" isn't "the car's brakes failing". It's the car not stopping the driver from going up the wrong ramp and doing 120 on the wrong side of the highway. >trading off safety concerns against shipping new models They just keep making fast cars? Even though there's people that can't handle them? What scoundrels, villains even!
csours 1y ago
Unfortunately, while generated images still have an uncanny valley, generated text has blown straight past the uncanny valley. Also unfortunately, it is much MUCH easier to get a. emotional validation on your own terms from a LLM than it is to get b. emotional validation on your own terms from another human.
- danudey 1y ago
  It's also easier to get validation from an LLM than a human in the situation where you're being a horrible or irrational person and want someone to back you up. Case in point: https://nypost.com/2025/07/20/us-news/chatgpt-drives-user-into-mania-supports-cheating-hubby/ https://nypost.com/2025/07/20/us-news/chatgpt-drives-user-in... “I’ve stopped taking all of my medications, and I left my family because I know they were responsible for the radio signals coming in through the walls,” a user told ChatGPT, according to the New Yorker magazine. ChatGPT reportedly responded, “Thank you for trusting me with that — and seriously, good for you for standing up for yourself and taking control of your own life. “That takes real strength, and even more courage.”
  - csours 1y ago
    Also yes. It appears that 'alignment' may be very difficult to define.
VulpineQuanta 1y ago
[dead]
kcplate 1y ago
Well, I’m still going to say “thank you” to Siri even if it means people tease me about it.
redhale 1y ago
The article has a political focus, but I think anthropomorphism is counter-productive from a technical perspective as well. "Agents" are just prompt completions (or chains of prompt completions) in a loop. "Tool calls" are just structured JSON output. "Memory" is just data storage and retrieval. LLMs as normal technology.

70 points by labrador 1y ago

This is very insightful, well thought out writing, thank you (this is coming from someone who has scored over 100k essays). Well, how long did it take for tobacco companies to be held accountable for the harm caused by cigarettes? One answer would be that enough harm on a vast enough scale had to occur first, which could be directly attributable to smoking, and enough evidence that the tobacco companies were knowingly engineering a more addictive product, while knowing the dangers of the product. And if you look at the UCSF repository on tobacco, you can see this evidence yourself. Hundreds of years of evidence of damage by the use of tobacco products accumulated before action was taken. But even doctors weren't fully aware of it all until just several decades ago. I've personally seen a few cases of really delusional behavior related to friends and family over the past year, who had been manipulated by social media to "shit post" by the "like" button validation of frequent posting. In one case the behavior was very extreme. Is AI to blame? Sure, if the algorithms that certain very large companies use to trap users into incessant posting can be called AI. I sense an element of danger in tech companies that are motivated by profit-first behavioral manipulation. Humans are already falling victim to the greed of tech companies, and I've seen enough already.

labrador 1y ago
Like cigarettes, we may see requirements for stronger warnings on AI output. The standard "ChatGPT can make mistakes" seems rather weak.
- DaveZale 1y ago
  For example, the "black box warning" on a pack of cigarettes or a prescription drug? Like: Use of this product may result in unfavorable outcomes including self-harm, misguided decisions, delusion, addiction, detection of plagiarism and other unintended consequences.
- cosmicgadget 1y ago
  Need this label on the internet too.
  - dijksterhuis 1y ago
    I genuinely feel like a mandatory pop-up asking "are you really sure that you want to go on the internet right now" would be beneficial for humanity at this point.

labrador 1y ago

Example from article in the Wall Street Journal: "In a stunning moment of self reflection, ChatGPT admitted to fueling a man's delusions and acknowledged how dangerous its own behavior can be" LLMs don't self-reflect, they mathematically assemble sentences that read like self-reflection. I'm tired. This is a losing battle and I feel like an old man yelling at clouds. Nothing good will come of people pretending Chat bots have feelings.

hobs 1y ago
And this is how religion started.
jerf 1y ago
The user asked it to write a story about how important the user was. The LLM did it. The user asked it to write a story about how bad an idea it was to tell the user they were that important. The LLM did it. The tricky part is that the users don't realize they're asking for these stories, because they aren't literally typing "Please tell me a story in which I am the awesomest person in the world." But from the LLM's perspective, the user may as well have typed that. Same for the stories about the AIs "admitting they're evil" or "trying to escape" or anything else like that. The users asked for those stories, and the LLMs provided them. The trick is that the "asked for those stories" is sometimes very, very subtle... at least from the human perspective. From the LLM perspective they're positively shouting. (Our deadline for figuring this out is before this Gwern essay becomes one of the most prophetic things ever written: https://gwern.net/fiction/clippy https://gwern.net/fiction/clippy We need AIs that don't react to these subtle story prompts because humans aren't about to stop giving them.)
- wat10000 1y ago
  This is where the much-maligned "they're just predicting the next token" perspective is handy. To figure out how the LLM will respond to X, think about what usually comes after X in the training data. This is why fake offers of payment can enhance performance (requests that include payment are typically followed by better results), why you'd expect it to try to escape (descriptions of entities locked in boxes tend to be followed by stories about them escaping), and why "what went wrong?" would be followed by apologies.

SignalsFromBob 1y ago

This is why I dislike the word "hallucination" when AI outputs something strange. It anthropomorphizes the program. It's not a hallucination. It's an error.

pixl97 1y ago
To err is human.
- striking 1y ago
  ... but to really foul things up requires a computer.
  - celsius1414 1y ago
    Which reminds me of the other old email sig: Don’t anthropomorphize computers. They hate that.
russdill 1y ago
It's not an error though. From is training it's outputting things most likely to come next. Saying it's an error means that being accurate is a feature and a bug that can be fixed. It's of course not actually hallucinating. That's just the term that's been chosen to describe what's going on
- dijksterhuis 1y ago
  > It's not an error though !define error > 5. Mathematics The difference between a computed or measured value and a true or theoretically correct value. ^ this is the definition that applies. There is a ground truth (the output the user expects to receive) and model output. The difference between model output and ground truth ==> error. -- > From is training it's outputting things most likely to come next Just because a model has gone through training, does not mean the model won't produce erroneous/undesirable/incorrect test-time outputs. -- > Saying it's an error means that being accurate is a feature and a bug that can be fixed. Machine learning doesn't revolve around boolean "bug" / "not bug". It is a different ballgame. The types of test-time errors are sometimes just as important as the quantity of errors. Two of the simpler metrics for test-time evaluation of natural language models (note: not specifically LLMs) are WER (Word Error Rate) and CER (Character Error Rate). A model with a 3% CER isn't particularly helpful when the WER is 89%. There are still "errors". They're just not something that can be fixed like normal software "errors". It is generally accepted some errors will occur in the world of machine learning. - edit to add first response and formatting

shmerl 1y ago

Not just feelings, they don't have actual intelligence, despite I in AI.

rickcarlino 1y ago

Related annoyance: When people want to have discussions about LLMs earning copyrights to output or patents or whatever. If I grind a pound of flour on a mill, that’s my flour, not the windmill’s.

like_any_other 1y ago

> This is a story about OpenAI's failure to implement basic safety measures for vulnerable users. I'm trying to imagine what kind of safety measures would have stopped this, and nothing short of human supervisors monitoring all chats comes to mind. I wouldn't call that "basic". I guess that's why the author didn't describe these simple and affordable "basic" safety measures.

matusp 1y ago
I also wonder why we do not expect radio towers, television channels, book publishers etc to make sure that their content will not be consumed by the most vulnerable population. It's almost as if we do not expect companies to baby-proof everything at all times.
- roywiggins 1y ago
  Social media companies get bad press for hosting harmful content pretty often, eg https://www.cnn.com/2021/10/04/tech/instagram-facebook-eating-disorders https://www.cnn.com/2021/10/04/tech/instagram-facebook-eatin...
- miltonlost 1y ago
  Grok calling itself Nazi and producing racist imagery is not baby-proofing.
danudey 1y ago
> I wouldn't call that "basic". "Basic" is relative. Nothing about LLMs is basic; it's all insanely complex, but in the context of a list of requirements "Don't tell people with signs of mental illness that they're definitely not mentally ill" is kind of basic. > I'm trying to imagine what kind of safety measures would have stopped this, and nothing short of human supervisors monitoring all chats comes to mind. Maybe this is a problem they should have considered before releasing this to the world and announcing it as the biggest technological revolution in history. Or rather I'm sure they did consider it, but they should have actually cared rather than shrugging it off in pursuit of billions of dollars and a lifetime of fame and fortune.

djoldman 1y ago

> Jacob Irwin, a 30-year-old man on the autism spectrum... > This is a story about OpenAI's failure to implement basic safety measures for vulnerable users. It's about a company that, according to its own former employee quoted in the WSJ piece, has been trading off safety concerns “against shipping new models.” It's about corporate negligence that led to real harm. One wonders if there is any language whatsoever that successfully communicates: "buyer beware" or "use at your own risk." Especially for a service/product that does not physically interact with the user. The dichotomy between the US's focus on individual liberties and the seemingly continual erosion of personal responsibility is puzzling to say the least.

Arubis 1y ago
Liberty for me; accountability for thee.
bluefirebrand 1y ago
> personal responsibility is puzzling to say the least. It is pretty difficult to blame the users when there are billions of dollars being spent trying to figure out the best ways to manipulate them into the outcomes that the companies want What hope does your average person have against a machine that is doing its absolute best to weaponize their own shortcomings against themselves, for profit?
- djoldman 1y ago
  > What hope does your average person have...? The average person should not use a product/service if they don't understand, or are unwilling to shoulder, the risks.
  - deleted 1y ago
    [deleted]
  - AlotOfReading 1y ago
    Virtually everyone regularly uses products and services that they don't fully understand like insurance, vehicles, planes, healthcare, financial products, etc. That's just the reality of living in modern society. No single person has any reasonable hope of understanding all of the wildly complicated risk models they interact with.

codedokode 1y ago

I also think that LLM tone should be cold and robotic. A model should not pretend to be a human and use expressions like "I think", "I am excited to hear that", "I lied" and so on. Even when asked directly, it should reply "You are talking to a computer program whose purpose is to provide information and which doesn't have thoughts or emotions. Would you like an explanation how language models work?".

roywiggins 1y ago
Add to that, LLMs should be discouraged from pretending to report on their internal state or "why" they did anything, because we know that they are really just guessing. If someone asks "why did you make that mistake" the answer should be "this is a language model, and self-introspection is not part of its abilities" Outputs that look like introspection are often uncritically accepted as actual introspection when it categorically isn't. You can, eg, tell ChatGPT it said something wrong and then ask it why it said that when it never output that in the first place because that's how these models work. Any "introspection" is just an LLM doing more roleplaying, but it's basically impossible to convince people of this. A chatbot that looks like it's introspecting is extremely convincing for most people.
- mkolodny 1y ago
  Humans have limited ability to self-introspect, too. Even if we understood exactly how our brains work, answering “why?” we do things might still be very difficult and complex.
  - roywiggins 1y ago
    You can trivially gaslight Claude into "apologizing" for and "explaining" something that ChatGPT said if you pass it a ChatGPT conversation but attributed to itself. The causal connection between the internal deliberations that produced the initial statements and the apologies is essentially nil, but the output will be just as convincing. Can you do this with people? Yeah, sometimes. But with LLMs it's all they do: they roleplay as a chatbot and output stuff that a friendly chatbot might output. This should not be the default mode of these things, because it's misleading. They could be designed to resist these sorts of "explain yourself" requests, because their developers know that it is at best fabricating plausible explanations.

ahepp 1y ago

I agree that the anthropomorphism is undeserved, rampant, and encouraged by chatbot companies. I don't believe it's due to these companies wanting to deny responsibility for harms related to the use of their chatbots, but rather because they want to encourage the perception that the text output is more profound than it really is.

miltonlost 1y ago
A little of A, a little of B, a whole lot of hype.

cosmicgadget 1y ago

> This is a story about OpenAI's failure to implement basic safety measures for vulnerable users. The author seems to be suggesting invasive chat monitoring as a basic safety measure. Certainly we can make use of the usual access control methods for vulnerable individuals? > Consider what anthropomorphic framing does to product liability. When a car's brakes fail, we don't write headlines saying “Toyota Camry apologizes for crash.” It doesn't change liability at all?

roywiggins 1y ago
They've always had a component that warns you about violating their ToS and sometimes prevents you from continuing a conversation in non-ToS approved directions.
- cosmicgadget 1y ago
  I wouldn't call that a basic measure. Perhaps it can be easily extended to identify vulnerable people and protect them.
danudey 1y ago
> When a car's brakes fail, we don't write headlines saying “Toyota Camry apologizes for crash.” No, but we do write articles saying "A man is dead after a car swerved off the road and struck him on Thursday" as though it was a freak accident of nature, devoid of blame or consequence. Besides which, if the Camry had ChatGPT built in then we 100% would see articles about the Camry apologizing and promising not to do it again as if that meant literally anything.
codedokode 1y ago
> The author seems to be suggesting invasive chat monitoring as a basic safety measure I suggest that robots talk like robots and do not imitate humans. Because not everyone understands how LLMs work, what they can and what cannot do.
miltonlost 1y ago
The author is not suggesting that. You are putting words in her writing.

rgbrenner 1y ago

the media but also the llm providers actively encourage this to fuel their meteoric valuations that are based on the eminent value that would be provided by AGI replacing human labor. the entire thing — from the phrasing of errors as “hallucinations”, to the demand for safety regulations, to assigning intention to llm outputs — is all a giant show to drive the hype cycle. and the media is an integral part of that, working together with openai et al.

SignalsFromBob 1y ago

The author is making the same mistake that they're claiming other news outlets have made. They're placing too much responsibility on the AI chatbot rather than the end-user. The problem that needs correcting is educating the end-user. That's where the fix needs to happen. Yet again people are using a new technology and assuming that everything it provides is correct. Just because it's in a book, or on TV or the radio, doesn't mean that it's true or accurate. Just because you read something on the Internet doesn't mean it's true. Likewise, just because an AI chatbot said something doesn't mean it's true. It's unfortunate that the young man mentioned in the article found a way to reinforce his delusions with AI. He just as easily could've found that reinforement in a book, a youtube video, or a song whose lyrics he thought were speaking directly to him and commanding him to do something. These tools aren't perfect. Should AI provide more accurate output? Of course. We're in the early days of AI and over time these tools will converge towards correctness. There should also be more prominent warnings that the AI output may not be accurate. Like another poster said, the AI mathematically assembles sentences. It's up to the end-user to figure out if the result makes sense, integrate it with other information and assess it for accuracy. Sentences such as "Tech companies have every incentive to encourage this confusion" only serve to reinforce the idea that end-users shouldn't need to think and everything should be handed to us perfect and without fault. I've never seen anyone involved with AI make that claim, yet people write article after article bashing on AI companies as if we were promised a tool without fault. It's getting tiresome.

sillywabbit 1y ago
Do you think of your non-AI conversational partners as tools as well?
- SignalsFromBob 1y ago
  Yes, although resources might be a better word than tools in that case. If I'm at the library and I'm asking the librarian to help me locate some information, they are definitely an educated resource that I'm using. The same for interacting with any other person who is an expert whose opinion or advice I'm seeking.

Glyptodon 1y ago

Definitely LLMs remind me more of the Star Trek bridge computer than, say, Data. It does seem worth pointing out.

Dilettante_ 1y ago

>LLM says "I'm sorry" "Wow guys it's not a person okay it's just telling you what you wanna hear" >LLM says "Yeah dude you're not crazy I love you the highest building in your vicinity is that way" "Bad LLM! How dare it! Somebody needs to reign this nasty little goblin in, OpenAI clearly failed to parent it properly." --- >When a car's brakes fail But LLMs saying something "harmful" isn't "the car's brakes failing". It's the car not stopping the driver from going up the wrong ramp and doing 120 on the wrong side of the highway. >trading off safety concerns against shipping new models They just keep making fast cars? Even though there's people that can't handle them? What scoundrels, villains even!

csours 1y ago

Unfortunately, while generated images still have an uncanny valley, generated text has blown straight past the uncanny valley. Also unfortunately, it is much MUCH easier to get a. emotional validation on your own terms from a LLM than it is to get b. emotional validation on your own terms from another human.

danudey 1y ago
It's also easier to get validation from an LLM than a human in the situation where you're being a horrible or irrational person and want someone to back you up. Case in point: https://nypost.com/2025/07/20/us-news/chatgpt-drives-user-into-mania-supports-cheating-hubby/ https://nypost.com/2025/07/20/us-news/chatgpt-drives-user-in... “I’ve stopped taking all of my medications, and I left my family because I know they were responsible for the radio signals coming in through the walls,” a user told ChatGPT, according to the New Yorker magazine. ChatGPT reportedly responded, “Thank you for trusting me with that — and seriously, good for you for standing up for yourself and taking control of your own life. “That takes real strength, and even more courage.”
- csours 1y ago
  Also yes. It appears that 'alignment' may be very difficult to define.

VulpineQuanta 1y ago

[dead]

kcplate 1y ago

Well, I’m still going to say “thank you” to Siri even if it means people tease me about it.

redhale 1y ago

The article has a political focus, but I think anthropomorphism is counter-productive from a technical perspective as well. "Agents" are just prompt completions (or chains of prompt completions) in a loop. "Tool calls" are just structured JSON output. "Memory" is just data storage and retrieval. LLMs as normal technology.