3 ms·

> Because they predicted other, different, stuff? In fact, yes. Prediction is a skill, skills can be developed and different predictors will have varying level

> Because they predicted other, different, stuff? In fact, yes. Prediction is a skill, skills can be developed and different predictors will have varying levels of skillfulness. The biggest contributor IIRC is Fermi-ization: the skill of decomposing big hard predictions into many smaller predictions. For example: what are the components of the question? They might include: - The likelihood of deaths due to combat action. That can in turn include predictions of the relative effects of ship armour, weapons effectiveness, preparedness for combat conditions and so on. - The number of people involved. If a skirmish involves a hundred ships on each side, 2 deaths is a very low estimate. How many ships are in the area? How many could reach it by December 31st? How many would be likely to be sent? - The causes of conflict. What are the triggering events, and how likely are they to occur? I'm sure smarter and better-qualified folks than me can argue all the details, but that's the point. You don't accept the question as a coin-flip with no priors. You go prior-hunting.

enkid 6y ago
The other big component is using Bayesian inference. For that you need a prior. What's the prior for this? You could look at incidents the US has had in similar freedom of navigation actions, what China has done with other countries, or other similar situations. For example, I can only think of a few similar incidents, and none that escalated to fatalities at sea. So your prior would probably be pretty small, say 1%. After that you would adjust according to new data. For example, China seems willing to do more risky stuff lately, with India being a perfect example. You might raise your estimate based on that. So maybe that raises your estimate to 2%. But it probably shouldn't completely alter your estimate.
vosper 6y ago
Ok, that makes sense, but my question isn't "how would I go about coming up with an estimate?" it's "how could I claim my estimate is accurate, given that the thing I'm estimating has never happened before?". Even if we extrapolate the verified accuracy of predicting other stuff over to this event, we surely can't know we're right? The article seems to be suggesting that this can be accurately estimated, but I feel like this would be a stronger claim if they cited something that was predicted and actually did happen... Perhaps I misinterpreted the paragraph?
- cl42 6y ago
  In "Superforecasting", it's a bit clearer. What they end up doing is asking you to predict an EVENT by a SPECIFIC TIME. In other words, what is the probability that "x" will happen by "DATE Y". In this case, by the time DATE Y takes place, you have a clearly verifiable binary outcome, which you can use to judge whether the forecasters were correct or not. There is some art to this -- the "event" in question needs to be very well-defined... I think the example they're using is a bit odd and they have much better examples in their book. The example of a counterfactual event (i.e., 'if X happens then estimate the likelihood of Y happening by DATE Z') adds a great deal of confusion here because you're predicting conditional probabilities. Still possible to verify but to your point, much more difficult. At this point, you're asking forecasters to build a probabilistic graph and estimate causality, which is becoming more of a science (e.g., Judea Pearle's work) but still lots to do in that space. Anyway, I'm digressing.
  - vosper 6y ago
    Thanks for the explanation, this helped me understand the idea a bit better.
    - jacques_chester 6y ago
      Adding slightly to it: they use Brier scores to say how "good" someone is at prediction. Because these predictions require a probability of confidence, you can punish those who were confidently wrong or reward those who were confidently right. So if I predict "aliens will land on or before December 1st" with 100% and no aliens appear, then my Brier score is 1 (the worst). If I say 0% confidence, then my score is 0 (the best). There are dozens and dozens of ways of scoring prediction ability, as it happens: https://www.cawcr.gov.au/projects/verification/verif_web_page.html https://www.cawcr.gov.au/projects/verification/verif_web_pag...

enkid 6y ago

The other big component is using Bayesian inference. For that you need a prior. What's the prior for this? You could look at incidents the US has had in similar freedom of navigation actions, what China has done with other countries, or other similar situations. For example, I can only think of a few similar incidents, and none that escalated to fatalities at sea. So your prior would probably be pretty small, say 1%. After that you would adjust according to new data. For example, China seems willing to do more risky stuff lately, with India being a perfect example. You might raise your estimate based on that. So maybe that raises your estimate to 2%. But it probably shouldn't completely alter your estimate.

vosper 6y ago

Ok, that makes sense, but my question isn't "how would I go about coming up with an estimate?" it's "how could I claim my estimate is accurate, given that the thing I'm estimating has never happened before?". Even if we extrapolate the verified accuracy of predicting other stuff over to this event, we surely can't know we're right? The article seems to be suggesting that this can be accurately estimated, but I feel like this would be a stronger claim if they cited something that was predicted and actually did happen... Perhaps I misinterpreted the paragraph?

cl42 6y ago
In "Superforecasting", it's a bit clearer. What they end up doing is asking you to predict an EVENT by a SPECIFIC TIME. In other words, what is the probability that "x" will happen by "DATE Y". In this case, by the time DATE Y takes place, you have a clearly verifiable binary outcome, which you can use to judge whether the forecasters were correct or not. There is some art to this -- the "event" in question needs to be very well-defined... I think the example they're using is a bit odd and they have much better examples in their book. The example of a counterfactual event (i.e., 'if X happens then estimate the likelihood of Y happening by DATE Z') adds a great deal of confusion here because you're predicting conditional probabilities. Still possible to verify but to your point, much more difficult. At this point, you're asking forecasters to build a probabilistic graph and estimate causality, which is becoming more of a science (e.g., Judea Pearle's work) but still lots to do in that space. Anyway, I'm digressing.
- vosper 6y ago
  Thanks for the explanation, this helped me understand the idea a bit better.
  - jacques_chester 6y ago
    Adding slightly to it: they use Brier scores to say how "good" someone is at prediction. Because these predictions require a probability of confidence, you can punish those who were confidently wrong or reward those who were confidently right. So if I predict "aliens will land on or before December 1st" with 100% and no aliens appear, then my Brier score is 1 (the worst). If I say 0% confidence, then my score is 0 (the best). There are dozens and dozens of ways of scoring prediction ability, as it happens: https://www.cawcr.gov.au/projects/verification/verif_web_page.html https://www.cawcr.gov.au/projects/verification/verif_web_pag...