4 ms·

Ok, that makes sense, but my question isn't "how would I go about coming up with an estimate?" it's "how could I claim my estimate is accurate, given that the t

by vosper 6y ago

Ok, that makes sense, but my question isn't "how would I go about coming up with an estimate?" it's "how could I claim my estimate is accurate, given that the thing I'm estimating has never happened before?". Even if we extrapolate the verified accuracy of predicting other stuff over to this event, we surely can't know we're right? The article seems to be suggesting that this can be accurately estimated, but I feel like this would be a stronger claim if they cited something that was predicted and actually did happen... Perhaps I misinterpreted the paragraph?

cl42 6y ago
In "Superforecasting", it's a bit clearer. What they end up doing is asking you to predict an EVENT by a SPECIFIC TIME. In other words, what is the probability that "x" will happen by "DATE Y". In this case, by the time DATE Y takes place, you have a clearly verifiable binary outcome, which you can use to judge whether the forecasters were correct or not. There is some art to this -- the "event" in question needs to be very well-defined... I think the example they're using is a bit odd and they have much better examples in their book. The example of a counterfactual event (i.e., 'if X happens then estimate the likelihood of Y happening by DATE Z') adds a great deal of confusion here because you're predicting conditional probabilities. Still possible to verify but to your point, much more difficult. At this point, you're asking forecasters to build a probabilistic graph and estimate causality, which is becoming more of a science (e.g., Judea Pearle's work) but still lots to do in that space. Anyway, I'm digressing.
- vosper 6y ago
  Thanks for the explanation, this helped me understand the idea a bit better.
  - jacques_chester 6y ago
    Adding slightly to it: they use Brier scores to say how "good" someone is at prediction. Because these predictions require a probability of confidence, you can punish those who were confidently wrong or reward those who were confidently right. So if I predict "aliens will land on or before December 1st" with 100% and no aliens appear, then my Brier score is 1 (the worst). If I say 0% confidence, then my score is 0 (the best). There are dozens and dozens of ways of scoring prediction ability, as it happens: https://www.cawcr.gov.au/projects/verification/verif_web_page.html https://www.cawcr.gov.au/projects/verification/verif_web_pag...

by vosper 6y ago

cl42 6y ago

In "Superforecasting", it's a bit clearer. What they end up doing is asking you to predict an EVENT by a SPECIFIC TIME. In other words, what is the probability that "x" will happen by "DATE Y". In this case, by the time DATE Y takes place, you have a clearly verifiable binary outcome, which you can use to judge whether the forecasters were correct or not. There is some art to this -- the "event" in question needs to be very well-defined... I think the example they're using is a bit odd and they have much better examples in their book. The example of a counterfactual event (i.e., 'if X happens then estimate the likelihood of Y happening by DATE Z') adds a great deal of confusion here because you're predicting conditional probabilities. Still possible to verify but to your point, much more difficult. At this point, you're asking forecasters to build a probabilistic graph and estimate causality, which is becoming more of a science (e.g., Judea Pearle's work) but still lots to do in that space. Anyway, I'm digressing.

vosper 6y ago
Thanks for the explanation, this helped me understand the idea a bit better.
- jacques_chester 6y ago
  Adding slightly to it: they use Brier scores to say how "good" someone is at prediction. Because these predictions require a probability of confidence, you can punish those who were confidently wrong or reward those who were confidently right. So if I predict "aliens will land on or before December 1st" with 100% and no aliens appear, then my Brier score is 1 (the worst). If I say 0% confidence, then my score is 0 (the best). There are dozens and dozens of ways of scoring prediction ability, as it happens: https://www.cawcr.gov.au/projects/verification/verif_web_page.html https://www.cawcr.gov.au/projects/verification/verif_web_pag...