4 ms·

Of all generative AI blunders, and it has plenty, this one is perhaps one of the least harmful ones. I mean, I can understand that someone might be distressed b

Of all generative AI blunders, and it has plenty, this one is perhaps one of the least harmful ones. I mean, I can understand that someone might be distressed by reading it, but at the same time, once you understand it is just outputting text from training data, you can dismiss it as a bullshit response, probably tied to a bad prompt. Much worse than that, and what makes Generative AI very useless to me, is its propensity to give out wrong answers that sound right or reasonable, especially on topics where I have low familiarity with. It's a massive waste of time, that mostly negates any benefits of using Generative AI in the first place. I don't see it ever getting better than that, too. If the training data is bad, the output will be bad, and it reached a point where I think it consumed all good training data it could. From now on it will be larger models of "garbage in, garbage out".

throwaway71271 2y ago
https://edition.cnn.com/2024/10/30/tech/teen-suicide-character-ai-lawsuit/index.html https://edition.cnn.com/2024/10/30/tech/teen-suicide-charact... there are more extreme cases
- techjamie 2y ago
  https://nypost.com/2024/10/23/us-news/florida-boy-14-killed-himself-after-falling-in-love-with-game-of-thrones-a-i-chatbot-lawsuit/ https://nypost.com/2024/10/23/us-news/florida-boy-14-killed-... This article has screenshots from the conversation. While the outcome is definitely more extreme, the actual conversation between him and the bot when it came to the message that encouraged him to go through with it is a little more questionable. He didn't ask it if he should kill himself directly, he told it he was going to "come home" to it, and it told him that he should. But the bot being a roleplay bot could easily respond as if that's just a part of the roleplay, and his character was literally going home. It isn't responding with a prompt indicating that it's responses might effect a real person that way. I would say what could really make it damning depends on the rest of the conversation, which will likely come out in court, and seeing if suicidal tendencies are present within its context window.
  - Ferret7446 2y ago
    The other problem is that this is for roleplaying. What if you wanted to roleplay a suicidal couple situation? To what extent should websites be legally obligated to verify the mental condition of its visitors?
svantana 2y ago
The raw language models will always have strange edge cases, for sure. But chat services are systems, and they almost certainly have additional models to detect strange or harmful answers, which can trigger the "As a chatbot" type responses. These systems will get more resilient and accurate over time, and big tech co:s tend to err on the side of caution.
- surgical_fire 2y ago
  "will get more resilient and accurate over time" is doing a lot of heavy lifting there. I don't think it will, because it depends on the training data. The largest models available already consumed the quality data available. Now they grow by ingesting lower quality data - possibly AI generated low quality data. A generative AI human centipede scenario. And I was not talking about edge cases. In plenty of interactions with gen AI, I have seen way too many confident answers that sounded reasonable, but were broken in ways that it require me more time to find out the problems than if I just looked for the answers myself. Those are not edge cases, those are just natural consequences of a system that just predicts the most likely next token. > big tech co:s tend to err on the side of caution. Good joke, I needed a laugh in this gray Sunday morning. Big tech CEOs err on the side of a bigger quarterly profit. That is all.
  - svantana 2y ago
    The training data in this case is feedback from users - reported responses. It's only logical that as that dataset grows and the developers have worked on it for longer, the 'toxic' answers will become more rare. And of course, 'caution' in this case refers to avoiding bad PR, nothing else.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]

throwaway71271 2y ago

https://edition.cnn.com/2024/10/30/tech/teen-suicide-character-ai-lawsuit/index.html https://edition.cnn.com/2024/10/30/tech/teen-suicide-charact... there are more extreme cases

techjamie 2y ago
https://nypost.com/2024/10/23/us-news/florida-boy-14-killed-himself-after-falling-in-love-with-game-of-thrones-a-i-chatbot-lawsuit/ https://nypost.com/2024/10/23/us-news/florida-boy-14-killed-... This article has screenshots from the conversation. While the outcome is definitely more extreme, the actual conversation between him and the bot when it came to the message that encouraged him to go through with it is a little more questionable. He didn't ask it if he should kill himself directly, he told it he was going to "come home" to it, and it told him that he should. But the bot being a roleplay bot could easily respond as if that's just a part of the roleplay, and his character was literally going home. It isn't responding with a prompt indicating that it's responses might effect a real person that way. I would say what could really make it damning depends on the rest of the conversation, which will likely come out in court, and seeing if suicidal tendencies are present within its context window.
- Ferret7446 2y ago
  The other problem is that this is for roleplaying. What if you wanted to roleplay a suicidal couple situation? To what extent should websites be legally obligated to verify the mental condition of its visitors?

svantana 2y ago

The raw language models will always have strange edge cases, for sure. But chat services are systems, and they almost certainly have additional models to detect strange or harmful answers, which can trigger the "As a chatbot" type responses. These systems will get more resilient and accurate over time, and big tech co:s tend to err on the side of caution.

surgical_fire 2y ago
"will get more resilient and accurate over time" is doing a lot of heavy lifting there. I don't think it will, because it depends on the training data. The largest models available already consumed the quality data available. Now they grow by ingesting lower quality data - possibly AI generated low quality data. A generative AI human centipede scenario. And I was not talking about edge cases. In plenty of interactions with gen AI, I have seen way too many confident answers that sounded reasonable, but were broken in ways that it require me more time to find out the problems than if I just looked for the answers myself. Those are not edge cases, those are just natural consequences of a system that just predicts the most likely next token. > big tech co:s tend to err on the side of caution. Good joke, I needed a laugh in this gray Sunday morning. Big tech CEOs err on the side of a bigger quarterly profit. That is all.
- svantana 2y ago
  The training data in this case is feedback from users - reported responses. It's only logical that as that dataset grows and the developers have worked on it for longer, the 'toxic' answers will become more rare. And of course, 'caution' in this case refers to avoiding bad PR, nothing else.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]