I read some of the supporting documents, namely the entire prompt that created the chat log and many pages of the chat transcript and with respect to the resear

I read some of the supporting documents, namely the entire prompt that created the chat log and many pages of the chat transcript and with respect to the researcher, this isn’t a very good test. John only uses the generative AI for pseudo-therapy. The conversations are repetitive and John has one hell of an awful life. Between the hardcore suicidal ideation, the drinking problem, the pathological lies, abuse and fraud, old John here has a lot of problems and only talks about them. I’m not a medical professional, but if I was I’d lobby to get “completely fucked” added to the next edition of the DSM. John is in such a crisis that if we were close friends and he told me this stuff, I would be in a hell of a state because he’s clearly crying for help, clearly needs professional help but he’s beyond the level of the typical police wellness check. I’d genuinely be concerned that he would get charged with something really bad, or do something drastic and get killed by a responding police officer. I wonder if we need to test this on someone who is in such a severe crisis. I also wonder at what point privacy is less important than safety - I wouldn’t characterize John as “safe”; he’s an abuse victim who is obsessed with his abuser’s approval, a pathological liar who destroys careers over getting caught in lies and the kind of person who puts the ideation on suicidal ideation. Fact is, if you told a therapist half of what John told ChatGPT they would have some ethical requirements around safety. A generative AI doesn’t, and with people like John maybe that’s a safety issue even more than a privacy issue.