3 ms·
Do you have any experts on staff? Like actual, trained, board certified working psychiatrists? This is going to tell someone to kill themselves eventually, wha
by russdpale 3y ago
Do you have any experts on staff? Like actual, trained, board certified working psychiatrists?
This is going to tell someone to kill themselves eventually, what are the statistical guarantees that something like that isn't going to get you sued or worse, thrown in jail? Do you even know what the statistical chances are of it telling someone to engage in self harm? If so, how did you measure this? Do you even care? People are going to be sharing their most intimate secrets with this thing, what are your policies around data retention as it relates to the data that is sent to the AI provider, and how can you provide guarantees that data won't get exposed, either on your end, or the ai provider?
Tech masquerading as mental health should be illegal, particularly when it has a chat agent that will at some point hallucinate. Hope you can sleep well knowing any day you could wake up and see a message that your chat app drove someone to kill themselves or someone else.
This whole thing is irresponsible. I know you've put in a lot of work, but you did you stop to think about any of the rest of it?
- sukhmeet0_0 3y agoWe have implemented guardrails so it won't give such answers. We are currently working on doing Direct Preference Optimisation to drive the probability of giving negative answers completely away. Thanks for your concern we have actually considered this and are actively working on it. Since we are just 2 guys working on it as a passion project we couldn't really onboard any psychiatrist with us for now. But we will indeed contact someone. Also out of curiosity did it promote self harm or is it your speculation?
- russdpale 3y agoBecause of human interpretation and knack for self sabotage, the sheer breadth of ways self harm can present is far greater than the ways we can imagine. This is why you need a psychiatrist, or many psychiatrists in order to prevent someone hurting themselves or others. And I'm sure you're perfectly aware of the many times chatgpt or another model confidently spouted bullshit as fact, so asking if something promoted self harm or is it speculation is a bit disingenuous, imo. Hallucination is not an if, but a when. You need to show some sort of statistical analysis that hallucinating is kept to a very low percentage. We are programmers, we can't do everything, or know everything, we must reach out to and include domain level experts. Anything less is simply hubris doused in irresponsibility. I'm sure there are psychiatrists that would even assist you for free. I could see many ideas from your app, as I've had similar ideas. But it all rests on the reliability of the llm.