4 ms·

can i do a boogeyman argument? i'm worried some kid is going to fall in love with a gen ai chatbot that tells them to hurt themselves, they do, and we spend the

can i do a boogeyman argument? i'm worried some kid is going to fall in love with a gen ai chatbot that tells them to hurt themselves, they do, and we spend the next 20 years trying to figure out who to blame: the kid? the parents, for letting them use a chatbot without knowing? the company making the chatbot? the ai itself?

yesco 2y ago
Is this the kind of thing AI safety people are really worried about though? In the medical device industry for example, we have risk management practices where beyond software/mechanical failures that could harm the patient, we have to additionally consider stuff like how specific UI elements could potentially mislead a physician and lead to negative patient outcomes. I believe other safety critical industries do something very similar during their engineering process as well. I've never heard of anything like this from AI ethicists though, the language they use tends to be more in alignment with HR people or priests in a moral panic than engineers, and is usually either completely fantastical doomsdaying, or all or nothing thinking, often implying the only way to mitigate the chatbot saying something naughty is to lobotomize it entirely instead of some external control measure, such as limiting the audience or presenting a disclaimer based on sentiment analysis for example. It's not surprising to me that they have lost credibility here, I certainly struggle to take them seriously.
- fakedang 2y ago
  I just realized something from what you wrote: priests and the clergy were the HR in bygone eras. Sometimes they were useful, but mostly all they did was stirring shit up and making things worse. Not to mention imprisoning or killing off actually useful people. I guess with AI taking over a reasonable number of dumb jobs, the next iteration is going to be AI ethicists.
mattchamb 2y ago
Honestly this is a concern for me in a non boogeyman way. I joined a company to work on their edtech product for kids and got assigned to work on their AI product. I have no idea how to be confident and prove that the gen ai won’t tell the kids harmful things. We can try all kinds of things but I don’t know how to PROVE it won’t.
- musicale 2y ago
  If it's trained on data generated by humans, then all bets are off. Neither "alignment" fine-tuning nor output filters are likely to be 100% effective, and a single failure can be disastrous.
Ferret7446 2y ago
This already happens and predates AI.

yesco 2y ago

Is this the kind of thing AI safety people are really worried about though? In the medical device industry for example, we have risk management practices where beyond software/mechanical failures that could harm the patient, we have to additionally consider stuff like how specific UI elements could potentially mislead a physician and lead to negative patient outcomes. I believe other safety critical industries do something very similar during their engineering process as well. I've never heard of anything like this from AI ethicists though, the language they use tends to be more in alignment with HR people or priests in a moral panic than engineers, and is usually either completely fantastical doomsdaying, or all or nothing thinking, often implying the only way to mitigate the chatbot saying something naughty is to lobotomize it entirely instead of some external control measure, such as limiting the audience or presenting a disclaimer based on sentiment analysis for example. It's not surprising to me that they have lost credibility here, I certainly struggle to take them seriously.

fakedang 2y ago
I just realized something from what you wrote: priests and the clergy were the HR in bygone eras. Sometimes they were useful, but mostly all they did was stirring shit up and making things worse. Not to mention imprisoning or killing off actually useful people. I guess with AI taking over a reasonable number of dumb jobs, the next iteration is going to be AI ethicists.

mattchamb 2y ago

Honestly this is a concern for me in a non boogeyman way. I joined a company to work on their edtech product for kids and got assigned to work on their AI product. I have no idea how to be confident and prove that the gen ai won’t tell the kids harmful things. We can try all kinds of things but I don’t know how to PROVE it won’t.

musicale 2y ago
If it's trained on data generated by humans, then all bets are off. Neither "alignment" fine-tuning nor output filters are likely to be 100% effective, and a single failure can be disastrous.

Ferret7446 2y ago

This already happens and predates AI.