6 ms·

What would be the cost for OpenAI to just stop these kinds of very long conversations that aren't about debugging or some actual long problem solving? It seems

What would be the cost for OpenAI to just stop these kinds of very long conversations that aren't about debugging or some actual long problem solving? It seems from the reports many people are being affected, some very very negatively, and many likely unreported. I don't understand why they don't show a warning or just open a new chat thread when a discussion gets too long or it can be detected that it's not fiction and likely veering into dangerous territory? I don't know how this doesn't give pause to the ChatGPT team. Especially with their supposed mission to be helpful to the world etc.

wahnfrieden 9mo ago
Those remediations would pretty clearly negatively impact revenue. And the team gets paid a lot to do their current work as-is. The way to get the team organized against something is to threaten their stock valuation (like when the workers organized against Altman's ousting). I don't see how cutting off users is going to do anything but drive the opposite reaction from the workers from what you want.
- gruez 9mo ago
  >Those remediations would pretty clearly negatively impact revenue That might make sense if openai was getting paid per token for these chats, but people who are using chatgpt as their therapist probably aren't using their consumption based API. They might have a premium account but how many % of premium users do you think are using chatgpt as their therapist and getting into long winded chats?
  - measurablefunc 9mo ago
    OpenAI is a synthetic media production company, they literally produce images, text, & video + audio to engage their users. The fact that people think OpenAI is an intelligence company is a testament to how good their marketing is at convincing people they are more than a synthetic media production company. This is also true for xAI & Grok. Most consumer AI companies are in the business of generating engaging synthetic media to keep their users glued to their interfaces for as long as possible.
  - wahnfrieden 9mo ago
    You can ask the same of users consuming toxic content on Facebook. Meta knows the content is harmful and they like it because it drives engagement. They also have policies to protect active scam ads if they are large enough revenue-drivers - doesn't get much more knowingly harmful than that, but it brings in the money. We shouldn't expect these businesses to have the best interests of users in mind especially when it conflicts with revenue opportunities.
    - mlinhares 9mo ago
      It is much harder to blame meta because the content is disperse and they can always say "they decided to consume this/join this group/like this page/watch these videos", while ChatGPT is directly telling the person their mother is trying to kill him. Not that the actual effect is any different, but for a jury the second case is much stronger.
dr-detroit 9mo ago
I would bet that AI girlfriend is a top ten use case for LLMs
- jacquesm 9mo ago
  It is probably a top 1 use case if you add the AI boyfriend option. There are a lot of lonely people out there.
- rhdunn 9mo ago
  And role-playing in general.
Aurornis 9mo ago
> It seems from the reports many people are being affected I think the rapid scale and growth of ChatGPT are breaking a lot of mental models about how common these occurrences are. ChatGPT's weekly active user count is twice as large as the population of the United States. More people use ChatGPT than Reddit. The number of people using ChatGPT on a weekly basis is so massive that it's hard to even begin to understand how common these occurrences are. When they happen, they get amplified and spread far and wide. The uses of ChatGPT and LLMs are very diverse. Calling for a shutdown of long conversations if they don't fit some pre-defined idea of problem solving is just not going to happen.
- shwaj 9mo ago
  It seems like a cheaper model could be asked to review transcripts, something like: “does this transcript seem at all like a wacky conspiracy theory that is encouraged in the use by the LLM”? In this case, it would have been easily detected. Depending on the prompt used, there would be more or less false positives/negatives, but low-hanging fruit such as this tragic incident should be avoidable.
- miltonlost 9mo ago
  Ah, the old "we're too big to be able to not do evil things! we've scaled too much so now we can't moderate! Oh well, sucks to not be rich."
  - Aurornis 9mo ago
    They're not claiming they don't moderate, though. Where are you getting that? A common complaint about ChatGPT and even their open weights models is that they're too censored.
- yousif_123123 9mo ago
  > Calling for a shutdown of long conversations if they don't fit some pre-defined idea of problem solving is just not going to happen. I am calling for some care to go in your product to try to reduce the occurrence of these bad outcomes. I just don't think it would be hard for them to detect that a conversation has reached a point that its becoming very likely the user is becoming delusional or may engage in dangerous behavior. How will we handle AGI if we ever create it, if we can't protect our society from these basic LLM problems?
j2kun 9mo ago
> I don't know how this doesn't give pause to the ChatGPT team. Especially with their supposed mission to be helpful to the world etc. Because the mission is a lie and the goal is profit. alwayshasbeen.jpg
jacquesm 9mo ago
I've had OpenAI do the weirdest things in conversations about aerodynamics and very low level device drivers, I don't think you will be able to reach a solution by just limiting the subjects. It is incredible how strongly it tries to position itself as a thinking entity that is above its users in the sense that it is handing out compliments all the time. Some people are more susceptible to others.
supermdguy 9mo ago
Looks like this would affect around 4.3% of chats (the "Self-Expression" category from this report[0]). Considering ChatGPT's userbase, that's an extremely large number of people, but less significant than I thought based on all the talk about AI companionship. That being said though, a similar crowd was pretty upset when OpenAI removed 4o, and the backlash was enough for them to bring it back. [0]: https://www.nber.org/system/files/working_papers/w34255/w34255.pdf https://www.nber.org/system/files/working_papers/w34255/w342...
DocTomoe 9mo ago
Just because you do not use a piece of technology or see no use in a particular use-case does not make it useless. If you want your Java code repaired, more power to you, but do not cripple the tool for people like me who use ChatGPT for more introspective work which cannot be expressed in a tweet. By the way, I would wager that 'long-form'-users are actually the users that pay for the service.
- yousif_123123 9mo ago
  > By the way, I would wager that 'long-form'-users are actually the users that pay for the service. I think it may be the case that many of these people that commit suicide or do other dangerous things after motivation from AI, are actually using weaker models that are available on the free versions. Whatever ability there is in AI to protect the user, it must be lower for the cheaper models that are freely available.
paxys 9mo ago
The cost would be a very large chunk of OpenAI's business. People aren't just using ChatGPT just to solve problems. It is a very popular tool for idle chatter, role playing, entertainment, friendship, therapy, and lots more. And OpenAI isn't financially incentivized to discourage this kind of use.
zemo 9mo ago
> I don't know how this doesn't give pause to the ChatGPT team a large pile of money > What would be the cost for OpenAI to just stop these kinds of very long conversations the aforementioned large pile of money

wahnfrieden 9mo ago

Those remediations would pretty clearly negatively impact revenue. And the team gets paid a lot to do their current work as-is. The way to get the team organized against something is to threaten their stock valuation (like when the workers organized against Altman's ousting). I don't see how cutting off users is going to do anything but drive the opposite reaction from the workers from what you want.

gruez 9mo ago
>Those remediations would pretty clearly negatively impact revenue That might make sense if openai was getting paid per token for these chats, but people who are using chatgpt as their therapist probably aren't using their consumption based API. They might have a premium account but how many % of premium users do you think are using chatgpt as their therapist and getting into long winded chats?
- measurablefunc 9mo ago
  OpenAI is a synthetic media production company, they literally produce images, text, & video + audio to engage their users. The fact that people think OpenAI is an intelligence company is a testament to how good their marketing is at convincing people they are more than a synthetic media production company. This is also true for xAI & Grok. Most consumer AI companies are in the business of generating engaging synthetic media to keep their users glued to their interfaces for as long as possible.
- wahnfrieden 9mo ago
  You can ask the same of users consuming toxic content on Facebook. Meta knows the content is harmful and they like it because it drives engagement. They also have policies to protect active scam ads if they are large enough revenue-drivers - doesn't get much more knowingly harmful than that, but it brings in the money. We shouldn't expect these businesses to have the best interests of users in mind especially when it conflicts with revenue opportunities.
  - mlinhares 9mo ago
    It is much harder to blame meta because the content is disperse and they can always say "they decided to consume this/join this group/like this page/watch these videos", while ChatGPT is directly telling the person their mother is trying to kill him. Not that the actual effect is any different, but for a jury the second case is much stronger.

dr-detroit 9mo ago

I would bet that AI girlfriend is a top ten use case for LLMs

jacquesm 9mo ago
It is probably a top 1 use case if you add the AI boyfriend option. There are a lot of lonely people out there.
rhdunn 9mo ago
And role-playing in general.

Aurornis 9mo ago

> It seems from the reports many people are being affected I think the rapid scale and growth of ChatGPT are breaking a lot of mental models about how common these occurrences are. ChatGPT's weekly active user count is twice as large as the population of the United States. More people use ChatGPT than Reddit. The number of people using ChatGPT on a weekly basis is so massive that it's hard to even begin to understand how common these occurrences are. When they happen, they get amplified and spread far and wide. The uses of ChatGPT and LLMs are very diverse. Calling for a shutdown of long conversations if they don't fit some pre-defined idea of problem solving is just not going to happen.

shwaj 9mo ago
It seems like a cheaper model could be asked to review transcripts, something like: “does this transcript seem at all like a wacky conspiracy theory that is encouraged in the use by the LLM”? In this case, it would have been easily detected. Depending on the prompt used, there would be more or less false positives/negatives, but low-hanging fruit such as this tragic incident should be avoidable.
miltonlost 9mo ago
Ah, the old "we're too big to be able to not do evil things! we've scaled too much so now we can't moderate! Oh well, sucks to not be rich."
- Aurornis 9mo ago
  They're not claiming they don't moderate, though. Where are you getting that? A common complaint about ChatGPT and even their open weights models is that they're too censored.
yousif_123123 9mo ago
> Calling for a shutdown of long conversations if they don't fit some pre-defined idea of problem solving is just not going to happen. I am calling for some care to go in your product to try to reduce the occurrence of these bad outcomes. I just don't think it would be hard for them to detect that a conversation has reached a point that its becoming very likely the user is becoming delusional or may engage in dangerous behavior. How will we handle AGI if we ever create it, if we can't protect our society from these basic LLM problems?

j2kun 9mo ago

> I don't know how this doesn't give pause to the ChatGPT team. Especially with their supposed mission to be helpful to the world etc. Because the mission is a lie and the goal is profit. alwayshasbeen.jpg

jacquesm 9mo ago

I've had OpenAI do the weirdest things in conversations about aerodynamics and very low level device drivers, I don't think you will be able to reach a solution by just limiting the subjects. It is incredible how strongly it tries to position itself as a thinking entity that is above its users in the sense that it is handing out compliments all the time. Some people are more susceptible to others.

supermdguy 9mo ago

Looks like this would affect around 4.3% of chats (the "Self-Expression" category from this report[0]). Considering ChatGPT's userbase, that's an extremely large number of people, but less significant than I thought based on all the talk about AI companionship. That being said though, a similar crowd was pretty upset when OpenAI removed 4o, and the backlash was enough for them to bring it back. [0]: https://www.nber.org/system/files/working_papers/w34255/w34255.pdf https://www.nber.org/system/files/working_papers/w34255/w342...

DocTomoe 9mo ago

Just because you do not use a piece of technology or see no use in a particular use-case does not make it useless. If you want your Java code repaired, more power to you, but do not cripple the tool for people like me who use ChatGPT for more introspective work which cannot be expressed in a tweet. By the way, I would wager that 'long-form'-users are actually the users that pay for the service.

yousif_123123 9mo ago
> By the way, I would wager that 'long-form'-users are actually the users that pay for the service. I think it may be the case that many of these people that commit suicide or do other dangerous things after motivation from AI, are actually using weaker models that are available on the free versions. Whatever ability there is in AI to protect the user, it must be lower for the cheaper models that are freely available.

paxys 9mo ago

The cost would be a very large chunk of OpenAI's business. People aren't just using ChatGPT just to solve problems. It is a very popular tool for idle chatter, role playing, entertainment, friendship, therapy, and lots more. And OpenAI isn't financially incentivized to discourage this kind of use.

zemo 9mo ago

> I don't know how this doesn't give pause to the ChatGPT team a large pile of money > What would be the cost for OpenAI to just stop these kinds of very long conversations the aforementioned large pile of money