2 ms·

Over many predictions we can apply a proper scoring rule to judge calibration. It's certainly the case that we don't learn much from looking at a single predic

• smsm42 9y ago
  Sure, if we have many predictions, that makes sense. But there are not that many presidential elections, even less - ones that 538 predicted on, and even less - ones that 538 predicted on used the same methodology. So how does one evaluate?
  • dllthomas 9y ago
    If we have fewer predictions, we need to be correspondingly less confident in our assessment of the quality of the predictor, to be sure. 538 has made hundreds (at least) of predictions, and across those has seemingly performed well. You can quite reasonably make a claim that most of those don't translate perfectly to the specific setting of a presidential election. But at the worst we should treat them as irrelevant and conclude that we don't know much about 538's quality. That's a far cry from a demonstration that their predictions are poor, which is what was claimed.