4 ms·

But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once? Unless the prediction is "this won't happen"

by vosper 6y ago

But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once? Unless the prediction is "this won't happen" and then it continuing to not happen makes the predictor look accurate... (slightly joking here) The article seems to claim that a specific event, which has never happened, can be estimated accurately. Is that really possible? How can the accuracy be assessed?

cl42 6y ago
Also worth noting: it's about being better than an existing baseline. Effectiveness is measured in relation to a baseline accuracy, which can already be quite poor. Suppose you make predictions about 100 events, and only 25 of these events actually take place. Your baseline might be in relation to these 25 events, and your 'superforecaster' needs to be better than an existing model, which could be really bad at predicting these odd/weird outcomes. Now you're better than everything else! :)
jessriedel 6y ago
> But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once? No, that's not right. The general idea is that you have forecasters who make predictions on multiple topics A, B, C, and D that have all not happened at the time of the prediction. Then, by looking at the outcomes of A, B, and C you can judge the ability of the forecaster to make prediction on topics what have not yet occurred at the time of prediction. You can then become confident on their ability to predict topic D before it occurs. The argument you're making is the classic skeptical argument about the philosophical problem of induction: just because some procedure (e.g., an astronomical model of the solar system) has made successful predictions in the past, we cannot rule out the possibility that the sun will not rise tomorrow. You can see where the problem with your argument lies by noticing that it depends heavily on the definition of what a "topic" is. If we have made successful predictions in the past on (say) the tides, and I make new predictions in the future, one can always point to some feature of the new predictions that could arguably make it a different topic, e.g., maybe El Nino is this year, or maybe the tides were for a slightly different region, etc. New predictions always differ from previous predictions in some way, but this does not mean the predictions are useless. The resolution is that "topic" is not a precise idea. Rather, if a forecaster is successful on topics A, B, and C that are all in a natural reference class, and D is also in that reference class, then this is evidence — not incontrovertible evidence, but strong compelling evidence —that they are likely to be accurate on D too.
- vosper 6y ago
  > The general idea is that you have forecasters who make predictions on multiple topics A, B, C, and D that have all not happened at the time of the prediction. Then, by looking at the outcomes of A, B, and C you can judge the ability of the forecaster to make prediction on topics what have not yet occurred at the time of prediction. You can then become confident on their ability to predict topic D before it occurs. Thanks, that really clarified the idea for me :)

by vosper 6y ago

cl42 6y ago

Also worth noting: it's about being better than an existing baseline. Effectiveness is measured in relation to a baseline accuracy, which can already be quite poor. Suppose you make predictions about 100 events, and only 25 of these events actually take place. Your baseline might be in relation to these 25 events, and your 'superforecaster' needs to be better than an existing model, which could be really bad at predicting these odd/weird outcomes. Now you're better than everything else! :)

jessriedel 6y ago

> But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once? No, that's not right. The general idea is that you have forecasters who make predictions on multiple topics A, B, C, and D that have all not happened at the time of the prediction. Then, by looking at the outcomes of A, B, and C you can judge the ability of the forecaster to make prediction on topics what have not yet occurred at the time of prediction. You can then become confident on their ability to predict topic D before it occurs. The argument you're making is the classic skeptical argument about the philosophical problem of induction: just because some procedure (e.g., an astronomical model of the solar system) has made successful predictions in the past, we cannot rule out the possibility that the sun will not rise tomorrow. You can see where the problem with your argument lies by noticing that it depends heavily on the definition of what a "topic" is. If we have made successful predictions in the past on (say) the tides, and I make new predictions in the future, one can always point to some feature of the new predictions that could arguably make it a different topic, e.g., maybe El Nino is this year, or maybe the tides were for a slightly different region, etc. New predictions always differ from previous predictions in some way, but this does not mean the predictions are useless. The resolution is that "topic" is not a precise idea. Rather, if a forecaster is successful on topics A, B, and C that are all in a natural reference class, and D is also in that reference class, then this is evidence — not incontrovertible evidence, but strong compelling evidence —that they are likely to be accurate on D too.

vosper 6y ago
> The general idea is that you have forecasters who make predictions on multiple topics A, B, C, and D that have all not happened at the time of the prediction. Then, by looking at the outcomes of A, B, and C you can judge the ability of the forecaster to make prediction on topics what have not yet occurred at the time of prediction. You can then become confident on their ability to predict topic D before it occurs. Thanks, that really clarified the idea for me :)