3 ms·

Koko raises ethical concerns using ChatGPT in mental health experiment

32 points by celtoid 4y ago

deleted 4y ago
[deleted]
btown 4y ago
There's an interesting meta-question here: is there a meaningful distinction between a system where (a) content is human-generated from scratch each time vs. one where (b) content is machine-suggested but human-in-the-loop-reviewed for every message? There's a paradox that the better the AI gets in (b), the more its human reviewers will trust its judgment, and inevitably the more the organization will try to scale how rapidly the human reviewers are expected to approve responses... so there may be an "uncanny valley" of edge cases where the human reviewers are letting substandard responses through, in a way that might be worse than if the AI were more error-prone and reviewers expected to do rewrites more often. So (b) may not monotonically approach the quality of (a) as AI gets better. And when ethical or brand-reputational considerations come into play (as is certainly true for minors in crisis, but may also be true for the automations you're considering building into your support/ticketing system!), this dynamic may have serious risks.
- kelseyfrog 4y ago
  I mean it failed because > Once people learned the messages were co-created by a machine, it didn’t work. Simulated empathy feels weird, empty. The distinction between human-authored vs human-edited is only salient so long as it aligns with the "people leaning the messages where co-created by a machine" dimension. A followup experiment would be disentangling the content from the frame. The control would be human-authored messages framed as such, and the three treatments would be human-edited messages framed as such, human-edited messages framed as human-authored, and human-authored messages framed as human-edited. Whether or not the conclusions are actionable is another question entirely. What would we do if human-edited messages framed as human-authored work just as well or even better? Are there ethical lies?
malshe 4y ago
How can both these statements be true at the same time if there was informed consent? > He noted that AI-based messages were "rated significantly higher than those written by humans on their own" > Once people learned the messages were co-created by a machine, it didn’t work.
- rogerkirkness 4y ago
  The job-to-be-done of therapy is 1. Attention from a human and 2. Cognitive loop bebugging. The latter can be done by a chatbot, the former cannot.
- hgsgm 4y ago
  As the article says several times, there was no informed consent.
metadat 4y ago
I clicked through optimistically hoping it might be the Koko the gorilla, which would've been a lot more impressive. https://en.wikipedia.org/wiki/Koko_(gorilla) https://en.wikipedia.org/wiki/Koko_(gorilla) Can't win 'em all.

32 points by celtoid 4y ago

deleted 4y ago

[deleted]

There's an interesting meta-question here: is there a meaningful distinction between a system where (a) content is human-generated from scratch each time vs. one where (b) content is machine-suggested but human-in-the-loop-reviewed for every message? There's a paradox that the better the AI gets in (b), the more its human reviewers will trust its judgment, and inevitably the more the organization will try to scale how rapidly the human reviewers are expected to approve responses... so there may be an "uncanny valley" of edge cases where the human reviewers are letting substandard responses through, in a way that might be worse than if the AI were more error-prone and reviewers expected to do rewrites more often. So (b) may not monotonically approach the quality of (a) as AI gets better. And when ethical or brand-reputational considerations come into play (as is certainly true for minors in crisis, but may also be true for the automations you're considering building into your support/ticketing system!), this dynamic may have serious risks.

kelseyfrog 4y ago
I mean it failed because > Once people learned the messages were co-created by a machine, it didn’t work. Simulated empathy feels weird, empty. The distinction between human-authored vs human-edited is only salient so long as it aligns with the "people leaning the messages where co-created by a machine" dimension. A followup experiment would be disentangling the content from the frame. The control would be human-authored messages framed as such, and the three treatments would be human-edited messages framed as such, human-edited messages framed as human-authored, and human-authored messages framed as human-edited. Whether or not the conclusions are actionable is another question entirely. What would we do if human-edited messages framed as human-authored work just as well or even better? Are there ethical lies?

malshe 4y ago

How can both these statements be true at the same time if there was informed consent? > He noted that AI-based messages were "rated significantly higher than those written by humans on their own" > Once people learned the messages were co-created by a machine, it didn’t work.

rogerkirkness 4y ago
The job-to-be-done of therapy is 1. Attention from a human and 2. Cognitive loop bebugging. The latter can be done by a chatbot, the former cannot.
hgsgm 4y ago
As the article says several times, there was no informed consent.

metadat 4y ago

I clicked through optimistically hoping it might be the Koko the gorilla, which would've been a lot more impressive. https://en.wikipedia.org/wiki/Koko_(gorilla) https://en.wikipedia.org/wiki/Koko_(gorilla) Can't win 'em all.