3 ms·
It is true that you cannot evaluate a single probabilistic prediction as being right or wrong (so you cannot say that Nate Silver was either wrong or right in s
by caf 8y ago
It is true that you cannot evaluate a single probabilistic prediction as being right or wrong (so you cannot say that Nate Silver was either wrong or right in some example - he gave a probability, and that probability was either accurate or inaccurate, but in isolation that cannot be evaluated).
However, if a consistent process is used to generate a sequence of probabilistic predictions, the accuracy of the predictions generated by that process certainly can be evaluated.
- safgasCVS 8y agoAbsolutely - and my issue is conflating experiments that ARE repeatable (eg a probabalistic game of chance) with experiments that arent (eg election results of two particular candidates). So its one of those heads I win tails you lose bets when it comes to Nate
- caf 8y agoIt works even if the process is "we ask an oracle for a prediction". The more that Nate Silver produces new prior probability estimates for events, the better an idea we can get of the accuracy of his process. This measure is calibration, as the original article mentions. If our oracle tells us that an event has a 70% chance of happening, then regardless of whether the event happens or not we can't say much about the oracle's calibration. But if, out of 100 independent events the oracle has predicted to each have a 70% chance of happening, 69 of those events actually happened, then that tells us quite a bit.