3 ms·
80% agreement is high, but the margins between models at the top are so low that even that remaining 20% could be enough to alter the final rankings, depending
by lolinder 3y ago
80% agreement is high, but the margins between models at the top are so low that even that remaining 20% could be enough to alter the final rankings, depending on which direction it errs.
- bagels 3y agoNot just that, but are the 80/20 randomly distributed? Probably not. These comparisons might have more in common with the 20%.
- s17n 3y agoBut 80% agreement is the same as between humans
- lolinder 3y agoTrue, but given that you're using one of the models to judge the others, it's likely that the cases of disagreement will tend to favor GPT-4. You would never use one of the competitors as a judge among humans.
- sdenton4 3y agoIf only there were some way to produce some kind of "interval" of scores where you were confident that the actual score sat, and then had some way of comparing these intervals between the different models...
- bostonsre 3y agoIs perfect agreement possible? And what is the definition of agreement? Humans don't agree about much.. are we saying agreement means it matches the truth after intensive investigation by humans?
- shanusmagnus 3y agoUgh, when I was doing my PhD work we were studying creativity in an experiment, and we needed an assessment for how creative different solutions were, and trying to get inter-rater reliability on this quite simple thing was just agonizing. I wound up abandoning the experiment because getting enough reliability would have required screwing down the standards so tightly that it would have ruined the underlying point of the thing.
- biomcgary 3y agoKind of hard to consistently evaluate creativity when someone might pull a James T. Kirk (https://en.wikipedia.org/wiki/Kobayashi_Maru https://en.wikipedia.org/wiki/Kobayashi_Maru), which is only creative the first time and just a cheat thereafter.
- shanusmagnus 3y agoJust one of many problems :)
- drittich 3y agoIt's surprising to me that you describe creativity as a simple thing. How were you defining and measuring it?
- shanusmagnus 3y agoI don't (and didn't) think creativity was simple, but the task was super simple (alternate uses task). The fact that people couldn't agree on how creative the answers were to such a simple task was the insight, although maybe it was only an insight because I was naive.