3 ms·

GPT Overperformance over Humans in Cognitive Reframing of Negative Scenarios

44 points by CharlesW 3y ago

wongarsu 3y ago
I certainly didn't have "GPT-4 scores higher on empathy than actual humans" on my bingo card. That's quite impressive, even if the task played to GPT's strengths, and it competed against people being paid $12/h to fill out studies on prolific.
- ben_w 3y ago
  Neither did I, but I ought to have due to Moravec's paradox, how effective even ELIZA was at that, and how cute animals affect us.
- zeroonetwothree 3y ago
  It seems to be sensitive to exactly how it was scored. Humans did better at certain types of tasks and GPT better at other types. And either way I wouldn’t say this is “empathy”
  - wongarsu 3y ago
    It wasn't empathy, but it did generate reframings that human reviewers scored higher on the metric "this rethinking is empathic". So it was better at generating the impression of empathy. Which is the same standard we generally apply to humans when judging their empathy, even if it is subtly wrong.
    - 082349872349872 3y ago
      > "all watched over / by machines of loving grace" —RGB
- Grimblewald 3y ago
  I dont think it scores higher neccesarily, but it certainly has a more narrow distribution than humans. If the average of the training corpus is closer to the average human than a given random human then it will generally produce more relatable content. If you have a look at the spread of ratings between gpt and human youll see quite heavy tails on human evaluations, while for gpt theyre very light. This is natural, we click with some and dislike others. Why? Because with some the direction our bias takes us from the mean mean we are quite alike and for others we are very distant and cannot relate to one another. Gpt relates to all equally (well/badly) so while i think gpt could be better than a randomly selected person, a carefully selected person would be hard to beat.
Grimblewald 3y ago
I think what what helps here is that gpt4 will be closer to the average human than a random human and so its responses will, on average, be more relatable. I think that when youre paired with tge right human and your biases are both in the same direction from the mean that synergistic effect wont be beatable by an LLM, but it doesnt surprise me that it will out perform humans at being the best on average. Heck, id wager gpt3 would be better as well.
deleted 3y ago
[deleted]
debo_ 3y ago
"Humans are, on average, even bigger assholes than computers"
hedora 3y ago
I guess this is good news for the Voight-Kampff test. https://bladerunner.fandom.com/wiki/Voight-Kampff_test https://bladerunner.fandom.com/wiki/Voight-Kampff_test

44 points by CharlesW 3y ago

I certainly didn't have "GPT-4 scores higher on empathy than actual humans" on my bingo card. That's quite impressive, even if the task played to GPT's strengths, and it competed against people being paid $12/h to fill out studies on prolific.

ben_w 3y ago
Neither did I, but I ought to have due to Moravec's paradox, how effective even ELIZA was at that, and how cute animals affect us.
zeroonetwothree 3y ago
It seems to be sensitive to exactly how it was scored. Humans did better at certain types of tasks and GPT better at other types. And either way I wouldn’t say this is “empathy”
- wongarsu 3y ago
  It wasn't empathy, but it did generate reframings that human reviewers scored higher on the metric "this rethinking is empathic". So it was better at generating the impression of empathy. Which is the same standard we generally apply to humans when judging their empathy, even if it is subtly wrong.
  - 082349872349872 3y ago
    > "all watched over / by machines of loving grace" —RGB
Grimblewald 3y ago
I dont think it scores higher neccesarily, but it certainly has a more narrow distribution than humans. If the average of the training corpus is closer to the average human than a given random human then it will generally produce more relatable content. If you have a look at the spread of ratings between gpt and human youll see quite heavy tails on human evaluations, while for gpt theyre very light. This is natural, we click with some and dislike others. Why? Because with some the direction our bias takes us from the mean mean we are quite alike and for others we are very distant and cannot relate to one another. Gpt relates to all equally (well/badly) so while i think gpt could be better than a randomly selected person, a carefully selected person would be hard to beat.

Grimblewald 3y ago

I think what what helps here is that gpt4 will be closer to the average human than a random human and so its responses will, on average, be more relatable. I think that when youre paired with tge right human and your biases are both in the same direction from the mean that synergistic effect wont be beatable by an LLM, but it doesnt surprise me that it will out perform humans at being the best on average. Heck, id wager gpt3 would be better as well.

deleted 3y ago

[deleted]

debo_ 3y ago

"Humans are, on average, even bigger assholes than computers"

hedora 3y ago

I guess this is good news for the Voight-Kampff test. https://bladerunner.fandom.com/wiki/Voight-Kampff_test https://bladerunner.fandom.com/wiki/Voight-Kampff_test