High levels of engagement aren't necessarily a good thing. One problem is that the advice is dangerous, but there's an entirely different problem, which is the

High levels of engagement aren't necessarily a good thing. One problem is that the advice is dangerous, but there's an entirely different problem, which is the LLM becoming a crutch that the person relies on because it will always tell them what they want to hear. Most people who call suicide hotlines aren't actually suicidal - they're just lonely or sad and want someone to talk to. The person who answers the phone will talk to them for awhile and validate their feelings, but after a little while they'll politely end the call. The issue is partly that people will monopolize a limited resource, but even if there were an unlimited number of people to answer the phone, it would be fundamentally unhealthy for someone to spend hours a day having someone validate their feelings. It very quickly turns into dependency and it keeps that person in a place where they aren't actually figuring out how to deal with these emotions themselves.