4 ms·

> Well-calibrated forecasters, for instance, can estimate the likelihood that a skirmish with the Chinese navy in the South China Sea will result in at least tw

by vosper 6y ago

> Well-calibrated forecasters, for instance, can estimate the likelihood that a skirmish with the Chinese navy in the South China Sea will result in at least two American deaths by December 31. But what policymakers really want to know is the extent to which China will threaten U.S. interests in the coming years and decades. This has never happened (a skirmish, let alone deaths) so how can we think that the forecasters are well-calibrated to predict the likelihood of this event? Because they predicted other, different, stuff?

deleted 6y ago
[deleted]
jacques_chester 6y ago
> Because they predicted other, different, stuff? In fact, yes. Prediction is a skill, skills can be developed and different predictors will have varying levels of skillfulness. The biggest contributor IIRC is Fermi-ization: the skill of decomposing big hard predictions into many smaller predictions. For example: what are the components of the question? They might include: - The likelihood of deaths due to combat action. That can in turn include predictions of the relative effects of ship armour, weapons effectiveness, preparedness for combat conditions and so on. - The number of people involved. If a skirmish involves a hundred ships on each side, 2 deaths is a very low estimate. How many ships are in the area? How many could reach it by December 31st? How many would be likely to be sent? - The causes of conflict. What are the triggering events, and how likely are they to occur? I'm sure smarter and better-qualified folks than me can argue all the details, but that's the point. You don't accept the question as a coin-flip with no priors. You go prior-hunting.
- enkid 6y ago
  The other big component is using Bayesian inference. For that you need a prior. What's the prior for this? You could look at incidents the US has had in similar freedom of navigation actions, what China has done with other countries, or other similar situations. For example, I can only think of a few similar incidents, and none that escalated to fatalities at sea. So your prior would probably be pretty small, say 1%. After that you would adjust according to new data. For example, China seems willing to do more risky stuff lately, with India being a perfect example. You might raise your estimate based on that. So maybe that raises your estimate to 2%. But it probably shouldn't completely alter your estimate.
- vosper 6y ago
  Ok, that makes sense, but my question isn't "how would I go about coming up with an estimate?" it's "how could I claim my estimate is accurate, given that the thing I'm estimating has never happened before?". Even if we extrapolate the verified accuracy of predicting other stuff over to this event, we surely can't know we're right? The article seems to be suggesting that this can be accurately estimated, but I feel like this would be a stronger claim if they cited something that was predicted and actually did happen... Perhaps I misinterpreted the paragraph?
  - cl42 6y ago
    In "Superforecasting", it's a bit clearer. What they end up doing is asking you to predict an EVENT by a SPECIFIC TIME. In other words, what is the probability that "x" will happen by "DATE Y". In this case, by the time DATE Y takes place, you have a clearly verifiable binary outcome, which you can use to judge whether the forecasters were correct or not. There is some art to this -- the "event" in question needs to be very well-defined... I think the example they're using is a bit odd and they have much better examples in their book. The example of a counterfactual event (i.e., 'if X happens then estimate the likelihood of Y happening by DATE Z') adds a great deal of confusion here because you're predicting conditional probabilities. Still possible to verify but to your point, much more difficult. At this point, you're asking forecasters to build a probabilistic graph and estimate causality, which is becoming more of a science (e.g., Judea Pearle's work) but still lots to do in that space. Anyway, I'm digressing.
    - vosper 6y ago
      Thanks for the explanation, this helped me understand the idea a bit better.
      - jacques_chester 6y ago
        Adding slightly to it: they use Brier scores to say how "good" someone is at prediction. Because these predictions require a probability of confidence, you can punish those who were confidently wrong or reward those who were confidently right. So if I predict "aliens will land on or before December 1st" with 100% and no aliens appear, then my Brier score is 1 (the worst). If I say 0% confidence, then my score is 0 (the best). There are dozens and dozens of ways of scoring prediction ability, as it happens: https://www.cawcr.gov.au/projects/verification/verif_web_page.html https://www.cawcr.gov.au/projects/verification/verif_web_pag...
jessriedel 6y ago
There's strong empirical evidence that prediction ability cross generalizes. That is, if a person is good at predicting unrelated topics A, B, and C, they are also like to be good at topic D.
- vosper 6y ago
  But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once? Unless the prediction is "this won't happen" and then it continuing to not happen makes the predictor look accurate... (slightly joking here) The article seems to claim that a specific event, which has never happened, can be estimated accurately. Is that really possible? How can the accuracy be assessed?
  - cl42 6y ago
    Also worth noting: it's about being better than an existing baseline. Effectiveness is measured in relation to a baseline accuracy, which can already be quite poor. Suppose you make predictions about 100 events, and only 25 of these events actually take place. Your baseline might be in relation to these 25 events, and your 'superforecaster' needs to be better than an existing model, which could be really bad at predicting these odd/weird outcomes. Now you're better than everything else! :)
  - jessriedel 6y ago
    > But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once? No, that's not right. The general idea is that you have forecasters who make predictions on multiple topics A, B, C, and D that have all not happened at the time of the prediction. Then, by looking at the outcomes of A, B, and C you can judge the ability of the forecaster to make prediction on topics what have not yet occurred at the time of prediction. You can then become confident on their ability to predict topic D before it occurs. The argument you're making is the classic skeptical argument about the philosophical problem of induction: just because some procedure (e.g., an astronomical model of the solar system) has made successful predictions in the past, we cannot rule out the possibility that the sun will not rise tomorrow. You can see where the problem with your argument lies by noticing that it depends heavily on the definition of what a "topic" is. If we have made successful predictions in the past on (say) the tides, and I make new predictions in the future, one can always point to some feature of the new predictions that could arguably make it a different topic, e.g., maybe El Nino is this year, or maybe the tides were for a slightly different region, etc. New predictions always differ from previous predictions in some way, but this does not mean the predictions are useless. The resolution is that "topic" is not a precise idea. Rather, if a forecaster is successful on topics A, B, and C that are all in a natural reference class, and D is also in that reference class, then this is evidence — not incontrovertible evidence, but strong compelling evidence —that they are likely to be accurate on D too.
scarier 6y ago
Well, yish. It wouldn't be a big stretch from what we've seen perennially in the South China Sea. https://en.wikipedia.org/wiki/Hainan_Island_incident https://en.wikipedia.org/wiki/Hainan_Island_incident
- vosper 6y ago
  That's not my point, though - perhaps I didn't word it clearly. I'm not saying this event can't or won't happen, I'm questioning the claim (maybe I'm misinterpreting the article?) that we can accurately forecast the likelihood of it happening, even though it's never happened.

by vosper 6y ago

deleted 6y ago

[deleted]

jacques_chester 6y ago

> Because they predicted other, different, stuff? In fact, yes. Prediction is a skill, skills can be developed and different predictors will have varying levels of skillfulness. The biggest contributor IIRC is Fermi-ization: the skill of decomposing big hard predictions into many smaller predictions. For example: what are the components of the question? They might include: - The likelihood of deaths due to combat action. That can in turn include predictions of the relative effects of ship armour, weapons effectiveness, preparedness for combat conditions and so on. - The number of people involved. If a skirmish involves a hundred ships on each side, 2 deaths is a very low estimate. How many ships are in the area? How many could reach it by December 31st? How many would be likely to be sent? - The causes of conflict. What are the triggering events, and how likely are they to occur? I'm sure smarter and better-qualified folks than me can argue all the details, but that's the point. You don't accept the question as a coin-flip with no priors. You go prior-hunting.

enkid 6y ago
The other big component is using Bayesian inference. For that you need a prior. What's the prior for this? You could look at incidents the US has had in similar freedom of navigation actions, what China has done with other countries, or other similar situations. For example, I can only think of a few similar incidents, and none that escalated to fatalities at sea. So your prior would probably be pretty small, say 1%. After that you would adjust according to new data. For example, China seems willing to do more risky stuff lately, with India being a perfect example. You might raise your estimate based on that. So maybe that raises your estimate to 2%. But it probably shouldn't completely alter your estimate.
vosper 6y ago
Ok, that makes sense, but my question isn't "how would I go about coming up with an estimate?" it's "how could I claim my estimate is accurate, given that the thing I'm estimating has never happened before?". Even if we extrapolate the verified accuracy of predicting other stuff over to this event, we surely can't know we're right? The article seems to be suggesting that this can be accurately estimated, but I feel like this would be a stronger claim if they cited something that was predicted and actually did happen... Perhaps I misinterpreted the paragraph?
- cl42 6y ago
  In "Superforecasting", it's a bit clearer. What they end up doing is asking you to predict an EVENT by a SPECIFIC TIME. In other words, what is the probability that "x" will happen by "DATE Y". In this case, by the time DATE Y takes place, you have a clearly verifiable binary outcome, which you can use to judge whether the forecasters were correct or not. There is some art to this -- the "event" in question needs to be very well-defined... I think the example they're using is a bit odd and they have much better examples in their book. The example of a counterfactual event (i.e., 'if X happens then estimate the likelihood of Y happening by DATE Z') adds a great deal of confusion here because you're predicting conditional probabilities. Still possible to verify but to your point, much more difficult. At this point, you're asking forecasters to build a probabilistic graph and estimate causality, which is becoming more of a science (e.g., Judea Pearle's work) but still lots to do in that space. Anyway, I'm digressing.
  - vosper 6y ago
    Thanks for the explanation, this helped me understand the idea a bit better.
    - jacques_chester 6y ago
      Adding slightly to it: they use Brier scores to say how "good" someone is at prediction. Because these predictions require a probability of confidence, you can punish those who were confidently wrong or reward those who were confidently right. So if I predict "aliens will land on or before December 1st" with 100% and no aliens appear, then my Brier score is 1 (the worst). If I say 0% confidence, then my score is 0 (the best). There are dozens and dozens of ways of scoring prediction ability, as it happens: https://www.cawcr.gov.au/projects/verification/verif_web_page.html https://www.cawcr.gov.au/projects/verification/verif_web_pag...

jessriedel 6y ago

There's strong empirical evidence that prediction ability cross generalizes. That is, if a person is good at predicting unrelated topics A, B, and C, they are also like to be good at topic D.

vosper 6y ago
But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once? Unless the prediction is "this won't happen" and then it continuing to not happen makes the predictor look accurate... (slightly joking here) The article seems to claim that a specific event, which has never happened, can be estimated accurately. Is that really possible? How can the accuracy be assessed?
- cl42 6y ago
  Also worth noting: it's about being better than an existing baseline. Effectiveness is measured in relation to a baseline accuracy, which can already be quite poor. Suppose you make predictions about 100 events, and only 25 of these events actually take place. Your baseline might be in relation to these 25 events, and your 'superforecaster' needs to be better than an existing model, which could be really bad at predicting these odd/weird outcomes. Now you're better than everything else! :)
- jessriedel 6y ago
  > But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once? No, that's not right. The general idea is that you have forecasters who make predictions on multiple topics A, B, C, and D that have all not happened at the time of the prediction. Then, by looking at the outcomes of A, B, and C you can judge the ability of the forecaster to make prediction on topics what have not yet occurred at the time of prediction. You can then become confident on their ability to predict topic D before it occurs. The argument you're making is the classic skeptical argument about the philosophical problem of induction: just because some procedure (e.g., an astronomical model of the solar system) has made successful predictions in the past, we cannot rule out the possibility that the sun will not rise tomorrow. You can see where the problem with your argument lies by noticing that it depends heavily on the definition of what a "topic" is. If we have made successful predictions in the past on (say) the tides, and I make new predictions in the future, one can always point to some feature of the new predictions that could arguably make it a different topic, e.g., maybe El Nino is this year, or maybe the tides were for a slightly different region, etc. New predictions always differ from previous predictions in some way, but this does not mean the predictions are useless. The resolution is that "topic" is not a precise idea. Rather, if a forecaster is successful on topics A, B, and C that are all in a natural reference class, and D is also in that reference class, then this is evidence — not incontrovertible evidence, but strong compelling evidence —that they are likely to be accurate on D too.

scarier 6y ago

Well, yish. It wouldn't be a big stretch from what we've seen perennially in the South China Sea. https://en.wikipedia.org/wiki/Hainan_Island_incident https://en.wikipedia.org/wiki/Hainan_Island_incident

vosper 6y ago
That's not my point, though - perhaps I didn't word it clearly. I'm not saying this event can't or won't happen, I'm questioning the claim (maybe I'm misinterpreting the article?) that we can accurately forecast the likelihood of it happening, even though it's never happened.