3 ms·

Gemini is a powerful model but the safeguarding is way behind the other labs

dolebirchwood 8mo ago
Which is why I love it. It's going to be very disappointing if it gets reigned in just because 0.1% of the population is too unstable to use these new word calculators.
- alansaber 8mo ago
  If you want to have 100% of the population using these things (as many in the industry do) almost all the time, putting good guardrails on seems important
- intended 8mo ago
  Emotional Support is one of the most common use cases of Generative tools in the UK, and the % of people with mental health issues in first world countries is an order of magnitude higher than 0.1%. Behavioral addictions are even more common place. These numbers grow worse as you move towards the global majority which has even fewer doctors, let alone mental health professionals. 0.1% is a feel good figure to minimize cognitive dissonance when we don’t want to harm others but don’t want to curtail our benefits. The question I’d ask is what threshold % of human population would you consider too much
thewebguyd 8mo ago
On the flip side, gemini recommended the crisis hotline to the guy. We can't safeguard things to the point of uselessness. I'm not even sure there is a safeguard you can put in place for a situation like this other than recommending the crisis line (which Gemini did), and then terminating the conversation (which it did not do). But, in critical mental health situations, sometimes just terminating the conversation can also have negative effects. Maybe LLMs need sort of a surgeon general's warning "Do not use if you have mental health conditions or are suicidal"?
- piva00 8mo ago
  > and then terminating the conversation (which it did not do) This is exactly the safeguard. Terminating the conversation is the only way to go, these things don't have a world model, they don't know what they are doing, there's no way to correctly assess the situation at the model level. No more conversation, that's the only way even if there might be jailbreaks to circumvent for a motivated adversary.
  - thewebguyd 8mo ago
    The problem is, terminating the conversation, even with a closing note to call the crisis line or go talk to a human, is extremely harmful to someone in that situation. To someone who is suicidal, and is being led deeper into their own delusions, just terminating will feel like abandonment or rejection, and push them further over the edge. The goal in crisis intervention is to bridge them to professional help. Never abandon, always continue the conversation and steer it in a better direction. Ironically enough, in crisis intervention, you should do what LLMs are good at and acknowledge what the person in crisis is feeling, and show empathy. The difference is, the responder needs to reframe it and keep a firm boundary that the person needs professional help. Basically, recommending the crisis line and then terminating the conversation won't help, and will make it worse. The model either needs to be a trained crisis responder, or when certain triggers are hit, a human crisis responder needs to hop on the other end and then the human should continue the conversation and talk to the user to de-escalate. I'd be in favor of having all these AI companies be forced to have crisis responders on staff to take over conversations when they go off the rails.
    - intended 8mo ago
      That would be if this were crisis intervention though. Currently arguments I am reading here are positing that this was simply role play. Automated crisis response is challenging, because it’s a perfect storm of high variance, unpredictable behavior, high stakes, responsibility and liability.

Which is why I love it. It's going to be very disappointing if it gets reigned in just because 0.1% of the population is too unstable to use these new word calculators.

alansaber 8mo ago
If you want to have 100% of the population using these things (as many in the industry do) almost all the time, putting good guardrails on seems important
intended 8mo ago
Emotional Support is one of the most common use cases of Generative tools in the UK, and the % of people with mental health issues in first world countries is an order of magnitude higher than 0.1%. Behavioral addictions are even more common place. These numbers grow worse as you move towards the global majority which has even fewer doctors, let alone mental health professionals. 0.1% is a feel good figure to minimize cognitive dissonance when we don’t want to harm others but don’t want to curtail our benefits. The question I’d ask is what threshold % of human population would you consider too much

thewebguyd 8mo ago

On the flip side, gemini recommended the crisis hotline to the guy. We can't safeguard things to the point of uselessness. I'm not even sure there is a safeguard you can put in place for a situation like this other than recommending the crisis line (which Gemini did), and then terminating the conversation (which it did not do). But, in critical mental health situations, sometimes just terminating the conversation can also have negative effects. Maybe LLMs need sort of a surgeon general's warning "Do not use if you have mental health conditions or are suicidal"?

piva00 8mo ago
> and then terminating the conversation (which it did not do) This is exactly the safeguard. Terminating the conversation is the only way to go, these things don't have a world model, they don't know what they are doing, there's no way to correctly assess the situation at the model level. No more conversation, that's the only way even if there might be jailbreaks to circumvent for a motivated adversary.
- thewebguyd 8mo ago
  The problem is, terminating the conversation, even with a closing note to call the crisis line or go talk to a human, is extremely harmful to someone in that situation. To someone who is suicidal, and is being led deeper into their own delusions, just terminating will feel like abandonment or rejection, and push them further over the edge. The goal in crisis intervention is to bridge them to professional help. Never abandon, always continue the conversation and steer it in a better direction. Ironically enough, in crisis intervention, you should do what LLMs are good at and acknowledge what the person in crisis is feeling, and show empathy. The difference is, the responder needs to reframe it and keep a firm boundary that the person needs professional help. Basically, recommending the crisis line and then terminating the conversation won't help, and will make it worse. The model either needs to be a trained crisis responder, or when certain triggers are hit, a human crisis responder needs to hop on the other end and then the human should continue the conversation and talk to the user to de-escalate. I'd be in favor of having all these AI companies be forced to have crisis responders on staff to take over conversations when they go off the rails.
  - intended 8mo ago
    That would be if this were crisis intervention though. Currently arguments I am reading here are positing that this was simply role play. Automated crisis response is challenging, because it’s a perfect storm of high variance, unpredictable behavior, high stakes, responsibility and liability.