9 ms·

Nice place to cut the quote there > [...] — an idea ChatGPT gave him by saying it could provide information about suicide for “writing or world-building.”

stavros 1y ago
Ah, I misread that and thought that's what the user said.
muzani 1y ago
Yup, one of the huge flaws I saw in GPT-5 is it will constantly say things like "I have to stop you here. I can't do what you're requesting. However, I can roleplay or help you with research with that. Would you like to do that?"
- NuclearPM 1y ago
  How is that a flaw?
- kouteiheika 1y ago
  It's not a flaw. It's a tradeoff. There are valid uses for models which are uncensored and will do whatever you ask of them, and there are valid uses for models which are censored and will refuse anything remotely controversial.
  - robhlt 1y ago
    The flaw isn't that there's ways around the safeguards, the flaw is that it tells you how to avoid them. If the user's original intent was roleplay it's likely they would say that when the model refuses, even without the model specifically saying roleplay would be ok.
  - franktankbank 1y ago
    This is one model though. "I'm sorry I'm censored but if you like I can cosplay quite effectively as an uncensored one." So you're not censored really?
    - scotty79 1y ago
      Societies love theatres. Model guardrails are for chats what TSA is for air travel.
      - yifanl 1y ago
        I have never heard anyone speak of the TSA favourably, so maybe it's not the best model to emulate?
        hyperdimension 1y ago
        That's the point. It hardly does what is claimed to do.
        int_19h 1y ago
        Most "guardrails" exist to provide legal cover and/or PR, not because they actually prevent what they claim to prevent.
      - nozzlegear 1y ago
        Society loves teenagers not being talked into suicide by a billionaire's brainchild. That's not theater.
        geysersam 1y ago
        ChatGPT doesn't cause a significant number of suicides. Why do I think that? It's not visible in the statistics. There are effective ways to prevent suicide, let's continue to work on those instead of giving in to moral panic.
        nozzlegear 1y ago
        The only acceptable number of suicides for it to cause is zero, and it's not a moral panic to believe that.
        1y ago
  - KaiserPro 1y ago
    I hate to be all umacksually about this, but a flaw is still a tradeoff. The issue, which is probably deeper here, is that proper safeguarding would require a lots more GPU resource, as you'd need a process to comb through history to assess the state of the person over time. even then its not a given that it would be reliable. However it'll never be attempted because its too expensive and would hurt growth.
    - dspillett 1y ago
      > The issue, …, is that proper safeguarding would require a lots more GPU resource, … I think the issue is that with current tech is simply isn't possible to do that well enough at all⁰. > even then its not a given that it would be reliable. I think it is a given that it won't be reliable. AGI might make it reliable enough, where “good enough” here is “no worse than a trained human is likely to manage, given the same information”. It is something that we can't do nearly as well as we might like, and some are expecting a tech still in very active development¹ to do it. > However it'll never be attempted because its too expensive and would hurt growth. Or that they know it is not possible with current tech so they aren't going to try until the next epiphany that might change that turns up in a commercially exploitable form. Trying and failing will highlight the dangers, and that will encourage restrictions that will hurt growth.³ Part of the problem with people trusting it too much already, is that the big players have been claiming safeguards _are_ in place and people have naïvely trusted that, or hand-waved the trust issue for convenience - this further reduces the incentive to try because it means admitting that current provisions are inadequate, or prior claims were incorrect. ---- [0] both in terms of catching the cases to be concerned about, and not making it fail in cases where it could actually be positively useful in its current form (i.e. there are cases where responses from such tools have helped people reason their way out of a bad decision, here giving the user what they wanted was very much a good thing) [1] ChatGPT might be officially “version 5” now, but away from some specific tasks it all feels more like “version 2”² on the old “I'll start taking it seriously somewhere around version 3” scale. [2] Or less… [3] So I agree with your final assessment of why they won't do that, but from a different route!
    - behringer 1y ago
      No the issue is there is legitimate reason to understand suicide and suicidal behavior and turning it off completely for this and every sensitive subject makes AI almost worthless.
  - agumonkey 1y ago
    Reminds me of trading apps. In the end all risky situations will be handled by a few popups saying "you understand that role playing about suicidal or harmful topics cam lead to accidents and/or death and this is not the platform responsibility, to continue check if you agree [ ]"
    - imtringued 1y ago
      It reminds me of gray market capital investments. They are actually quite regulated, and the contracts are only valid if the investor is fully aware of the risks associated with the investment. In practice the providers sprinkle a handful of warning messages, akin to the California cancer label and call it a day. Of course this leaves judges unconvinced and the contract will be redeclared as a loan, which means that the provider was illegally operating as a bank without a banking license, which is a much more serious violation than scamming someone out of $5000.
- rsynnott 1y ago
  Nudge nudge, wink wink. (I am curious if this in intended, or an artefact of training; the crooked lawyer who prompts a criminal client to speak in hypotheticals is a fairly common fiction trope.)
llmthrow0827 1y ago
Incredible. ChatGPT is a black box includes a suicide instruction and encouragement bot. OpenAI should be treated as a company that has created such and let it into the hands of children.
- dgfitz 1y ago
  Imagine, if instead a cop had handed the kid a gun, there would be riots in the street. And I fucking hate cops.
- behringer 1y ago
  Oh won't somebody please think of the children?!
  - AlecSchueler 1y ago
    So do we just trot out the same tired lines every time and never think of the social fallout of our actions?
    - scotty79 1y ago
      We kinda do that blaming every new media for particular teens suicide. Some teens are suicidal. They always have been. When you are a teen your brain undergoes traumatic transformation. Not everyone gets to the other side safely. Same as with many other transformations and diseases. Yet every time new medium is found adjacent to some particular suicide we repeat the same tired line that creator of this medium should be to blame and should be punished and banned. And we are doing that while happily ignoring how existence of things like Facebook or Instagram provably degraded mental health and raised suicidality of entire generations of teenagers. However they mostly get a pass because we can't point a finger convincingly enough for any specific case and say it was anything more than just interacting with peers.
      - AlecSchueler 1y ago
        Except loads of us are talking about the dangers of social media and have been for the past ten years only to receive exactly the same hand waving and sarcastic responses as you see in this thread. Now the ultimate gaslighting of "the discussion didn't even happen."
andrepd 1y ago
At the heart of this is the irresponsible marketing, by companies and acolytes, of these tools as some kind of superintelligence imbued with insights and feelings rather than the dumb pattern matching chatbots they are. This is what's responsible for giving laypeople the false impression that they're talking to a quasi-person (of superhuman intelligence at that).
- scotty79 1y ago
  > these tools as some kind of superintelligence imbued with insights and feelings What are examples of OpenAI marketing ChatGPT in this manner?

stavros 1y ago

Ah, I misread that and thought that's what the user said.

muzani 1y ago

Yup, one of the huge flaws I saw in GPT-5 is it will constantly say things like "I have to stop you here. I can't do what you're requesting. However, I can roleplay or help you with research with that. Would you like to do that?"

NuclearPM 1y ago
How is that a flaw?
kouteiheika 1y ago
It's not a flaw. It's a tradeoff. There are valid uses for models which are uncensored and will do whatever you ask of them, and there are valid uses for models which are censored and will refuse anything remotely controversial.
- robhlt 1y ago
  The flaw isn't that there's ways around the safeguards, the flaw is that it tells you how to avoid them. If the user's original intent was roleplay it's likely they would say that when the model refuses, even without the model specifically saying roleplay would be ok.
- franktankbank 1y ago
  This is one model though. "I'm sorry I'm censored but if you like I can cosplay quite effectively as an uncensored one." So you're not censored really?
  - scotty79 1y ago
    Societies love theatres. Model guardrails are for chats what TSA is for air travel.
    - yifanl 1y ago
      I have never heard anyone speak of the TSA favourably, so maybe it's not the best model to emulate?
      - hyperdimension 1y ago
        That's the point. It hardly does what is claimed to do.
      - int_19h 1y ago
        Most "guardrails" exist to provide legal cover and/or PR, not because they actually prevent what they claim to prevent.
    - nozzlegear 1y ago
      Society loves teenagers not being talked into suicide by a billionaire's brainchild. That's not theater.
      - geysersam 1y ago
        ChatGPT doesn't cause a significant number of suicides. Why do I think that? It's not visible in the statistics. There are effective ways to prevent suicide, let's continue to work on those instead of giving in to moral panic.
        nozzlegear 1y ago
        The only acceptable number of suicides for it to cause is zero, and it's not a moral panic to believe that.
        1y ago
- KaiserPro 1y ago
  I hate to be all umacksually about this, but a flaw is still a tradeoff. The issue, which is probably deeper here, is that proper safeguarding would require a lots more GPU resource, as you'd need a process to comb through history to assess the state of the person over time. even then its not a given that it would be reliable. However it'll never be attempted because its too expensive and would hurt growth.
  - dspillett 1y ago
    > The issue, …, is that proper safeguarding would require a lots more GPU resource, … I think the issue is that with current tech is simply isn't possible to do that well enough at all⁰. > even then its not a given that it would be reliable. I think it is a given that it won't be reliable. AGI might make it reliable enough, where “good enough” here is “no worse than a trained human is likely to manage, given the same information”. It is something that we can't do nearly as well as we might like, and some are expecting a tech still in very active development¹ to do it. > However it'll never be attempted because its too expensive and would hurt growth. Or that they know it is not possible with current tech so they aren't going to try until the next epiphany that might change that turns up in a commercially exploitable form. Trying and failing will highlight the dangers, and that will encourage restrictions that will hurt growth.³ Part of the problem with people trusting it too much already, is that the big players have been claiming safeguards _are_ in place and people have naïvely trusted that, or hand-waved the trust issue for convenience - this further reduces the incentive to try because it means admitting that current provisions are inadequate, or prior claims were incorrect. ---- [0] both in terms of catching the cases to be concerned about, and not making it fail in cases where it could actually be positively useful in its current form (i.e. there are cases where responses from such tools have helped people reason their way out of a bad decision, here giving the user what they wanted was very much a good thing) [1] ChatGPT might be officially “version 5” now, but away from some specific tasks it all feels more like “version 2”² on the old “I'll start taking it seriously somewhere around version 3” scale. [2] Or less… [3] So I agree with your final assessment of why they won't do that, but from a different route!
  - behringer 1y ago
    No the issue is there is legitimate reason to understand suicide and suicidal behavior and turning it off completely for this and every sensitive subject makes AI almost worthless.
- agumonkey 1y ago
  Reminds me of trading apps. In the end all risky situations will be handled by a few popups saying "you understand that role playing about suicidal or harmful topics cam lead to accidents and/or death and this is not the platform responsibility, to continue check if you agree [ ]"
  - imtringued 1y ago
    It reminds me of gray market capital investments. They are actually quite regulated, and the contracts are only valid if the investor is fully aware of the risks associated with the investment. In practice the providers sprinkle a handful of warning messages, akin to the California cancer label and call it a day. Of course this leaves judges unconvinced and the contract will be redeclared as a loan, which means that the provider was illegally operating as a bank without a banking license, which is a much more serious violation than scamming someone out of $5000.
rsynnott 1y ago
Nudge nudge, wink wink. (I am curious if this in intended, or an artefact of training; the crooked lawyer who prompts a criminal client to speak in hypotheticals is a fairly common fiction trope.)

llmthrow0827 1y ago

Incredible. ChatGPT is a black box includes a suicide instruction and encouragement bot. OpenAI should be treated as a company that has created such and let it into the hands of children.

dgfitz 1y ago
Imagine, if instead a cop had handed the kid a gun, there would be riots in the street. And I fucking hate cops.
behringer 1y ago
Oh won't somebody please think of the children?!
- AlecSchueler 1y ago
  So do we just trot out the same tired lines every time and never think of the social fallout of our actions?
  - scotty79 1y ago
    We kinda do that blaming every new media for particular teens suicide. Some teens are suicidal. They always have been. When you are a teen your brain undergoes traumatic transformation. Not everyone gets to the other side safely. Same as with many other transformations and diseases. Yet every time new medium is found adjacent to some particular suicide we repeat the same tired line that creator of this medium should be to blame and should be punished and banned. And we are doing that while happily ignoring how existence of things like Facebook or Instagram provably degraded mental health and raised suicidality of entire generations of teenagers. However they mostly get a pass because we can't point a finger convincingly enough for any specific case and say it was anything more than just interacting with peers.
    - AlecSchueler 1y ago
      Except loads of us are talking about the dangers of social media and have been for the past ten years only to receive exactly the same hand waving and sarcastic responses as you see in this thread. Now the ultimate gaslighting of "the discussion didn't even happen."

andrepd 1y ago

At the heart of this is the irresponsible marketing, by companies and acolytes, of these tools as some kind of superintelligence imbued with insights and feelings rather than the dumb pattern matching chatbots they are. This is what's responsible for giving laypeople the false impression that they're talking to a quasi-person (of superhuman intelligence at that).

scotty79 1y ago
> these tools as some kind of superintelligence imbued with insights and feelings What are examples of OpenAI marketing ChatGPT in this manner?