4 ms·
That illustrates how useless such predictions are. What "Trump has 30% chance of winning" means? If Trump won, did this prediction work or not? If he lost, did
by smsm42 9y ago
That illustrates how useless such predictions are. What "Trump has 30% chance of winning" means? If Trump won, did this prediction work or not? If he lost, did it work or not? How one could distinguish successful prediction from a failed one, even post-factum? I don't see any way.
Now, if we had 100 identical Trumps running in 100 USA elections, and 30 of them would win and 70 of them would lose, we'd say good work 538, you were spot on. But since we have one Trump winning one election - I do not see any way of seeing how 538 was "better" than NYT since I can't see how you can say whether both were successful or not. NYT gave Trump 1% to win, he won - so that 1% worked. Big deal, people win lottery all the time, with much lower chances. So maybe NYT was completely right and 538 is full of it, maybe it's really an one in a 100 event? How would you know?
- ceejayoz 9y agoWe know 538's model was better because of what happened on election day. Their model gave Trump a dramatically higher chance than other poll aggregators for a variety of reasons (detailed at http://fivethirtyeight.com/features/election-update-why-our-model-is-more-bullish-than-others-on-trump/ http://fivethirtyeight.com/features/election-update-why-our-...) and #4 there (correlated polling errors between states) wound up being very significant. Now, you're right that we can't perfectly determine if 99% or 70% (or something else entirely) was Hillary's actual chance of winning. We can, however, look at the assumptions made by each model and evaluate those on their merits.
- dllthomas 9y agoOver many predictions we can apply a proper scoring rule to judge calibration. It's certainly the case that we don't learn much from looking at a single prediction. https://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_rules https://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_ru...
- smsm42 9y agoSure, if we have many predictions, that makes sense. But there are not that many presidential elections, even less - ones that 538 predicted on, and even less - ones that 538 predicted on used the same methodology. So how does one evaluate?
- dllthomas 9y agoIf we have fewer predictions, we need to be correspondingly less confident in our assessment of the quality of the predictor, to be sure. 538 has made hundreds (at least) of predictions, and across those has seemingly performed well. You can quite reasonably make a claim that most of those don't translate perfectly to the specific setting of a presidential election. But at the worst we should treat them as irrelevant and conclude that we don't know much about 538's quality. That's a far cry from a demonstration that their predictions are poor, which is what was claimed.