3 ms·

Is perfect agreement possible? And what is the definition of agreement? Humans don't agree about much.. are we saying agreement means it matches the truth after

• shanusmagnus 3y ago
  Ugh, when I was doing my PhD work we were studying creativity in an experiment, and we needed an assessment for how creative different solutions were, and trying to get inter-rater reliability on this quite simple thing was just agonizing. I wound up abandoning the experiment because getting enough reliability would have required screwing down the standards so tightly that it would have ruined the underlying point of the thing.
  • biomcgary 3y ago
    Kind of hard to consistently evaluate creativity when someone might pull a James T. Kirk (https://en.wikipedia.org/wiki/Kobayashi_Maru https://en.wikipedia.org/wiki/Kobayashi_Maru), which is only creative the first time and just a cheat thereafter.
    • shanusmagnus 3y ago
      Just one of many problems :)
  • drittich 3y ago
    It's surprising to me that you describe creativity as a simple thing. How were you defining and measuring it?
    • shanusmagnus 3y ago
      I don't (and didn't) think creativity was simple, but the task was super simple (alternate uses task). The fact that people couldn't agree on how creative the answers were to such a simple task was the insight, although maybe it was only an insight because I was naive.