4 ms·

57 percent? It's almost as reliable as a coin flip.

• gull 11y ago
  But more reliable than betrayer's judgement who never saw it coming. And what happens if one combines that data with other data?
  • jmnicolas 11y ago
    Look at the second comment from the article (from Matt) : "[...]and you use this computer algorithm to predict betrayal, the computer will predict that you'll be betrayed 4 times during those 10 turns. That's a lot of false positives[...]"
• philh 11y ago
  Unsurprisingly, that 57% is almost meaningless. It looks like they took all the betrayals they had, then matched each betrayal with a similar friendship that didn't end in betrayal, and somehow ended up with a data set that was about 52% lasting friendships and 48% betrayals. I think that "somehow" is that each friendship added a data point per season, and lasting friendships lasted longer than eventual-betrayal friendships. Within that dataset, they got 57% accuracy on the question "will this friendship end in a betrayal?" That doesn't mean they can look at an arbitrary existing friendship and predict betrayal with 57% accuracy, because the base rates will be different. And in any case, it's important to distinguish between false positives and false negatives, which the accuracy score completely fails to do. But they also looked at eventual-betrayal friendships, and attempted to predict when the betrayal would happen. (Or at least "will it happen this turn?") The base rate for that was 14% and they got an F1 score (https://en.wikipedia.org/wiki/F1_score https://en.wikipedia.org/wiki/F1_score) of 0.31, which I think is decent but not fantastic.