4 ms·

Philip Tetlock in his masterwork Superforecasting does a great job of showing “expert” forecasting is no better than reasonably informed “amateurs.” A must read

Philip Tetlock in his masterwork Superforecasting does a great job of showing “expert” forecasting is no better than reasonably informed “amateurs.” A must read.

Hoasi 8y ago
Philip Tetlock on Superforecasting, on EconTalk podcast, for those interested: http://www.econtalk.org/philip-tetlock-on-superforecasting/ http://www.econtalk.org/philip-tetlock-on-superforecasting/
nabla9 8y ago
Superforecasting focuses point forecasting and uses the simplest forecasting methodologies to predict outcomes. Expert-opinion methods are very just one tool for forecasters. But even Tetlock has later demonstrated multiple techniques that improve expert-opinion forecasting accuracy dramatically. Many similar and more advanced techniques have been used in the military and business for ages. Training exercises , simulations, games, cross-impact analysis, policy capture analysis, scenario methods etc.
- bkohlmann 8y ago
  As a former member of the military who went through very good and relevant training, I can attest to the TACTICAL value of simulations and war games. However, they were basically worthless in training to a highly dynamic strategic environment. Many were designed to validate a system or belief rather than test the unknown. In theory war games should be built to be free play oriented...in reality the aren’t.
  - nabla9 8y ago
    Those are usually just exercises with adversarial component. The war games I'm thinking of are those what flag officers and staff do in RAND and other places. For example, they bring in diplomats and other outsiders to play the opponents. Or they may be pen and paper games (spreadsheets) scenario analysis where sequence of actions are played out and consequence of every decision are updated into the environment.
tedsanders 8y ago
As someone who has spent years trading in forecasting tournaments, I worry this characterization might be misinterpreted as saying know nothingism is correct and we should give up on predicting the future. (Not that you meant it that way, of course.) It's true that "amateurs" can consistently beat "experts". But not all amateurs - only some. This suggests to me there is true prediction expertise, but our system that attaches labels of "expert" does not select for it. Prediction is very much a skill and some people are very much better than others. We just don't regularly measure or select for it explicitly.
- ssivark 8y ago
  What about the complaint/criticism that Brier scores (which Tetlock uses to score forecasters) do not weight the severity/importance of the result eg: forecasting the outcome of a war and forecasting the winner of the super bowl are weighted equally.
  - tedsanders 8y ago
    Totally valid criticism. I think nuances like these were what Tetlock had toughest time articulating in Superforecasting and his public interviews. He chose to emphasize the big, clear points rather than subtler, more complicated issues like how questions were chosen and how they should be compared or weighted.

Philip Tetlock in his masterwork Superforecasting does a great job of showing “expert” forecasting is no better than reasonably informed “amateurs.” A must read.

Hoasi 8y ago

Philip Tetlock on Superforecasting, on EconTalk podcast, for those interested: http://www.econtalk.org/philip-tetlock-on-superforecasting/ http://www.econtalk.org/philip-tetlock-on-superforecasting/

nabla9 8y ago

Superforecasting focuses point forecasting and uses the simplest forecasting methodologies to predict outcomes. Expert-opinion methods are very just one tool for forecasters. But even Tetlock has later demonstrated multiple techniques that improve expert-opinion forecasting accuracy dramatically. Many similar and more advanced techniques have been used in the military and business for ages. Training exercises , simulations, games, cross-impact analysis, policy capture analysis, scenario methods etc.

bkohlmann 8y ago
As a former member of the military who went through very good and relevant training, I can attest to the TACTICAL value of simulations and war games. However, they were basically worthless in training to a highly dynamic strategic environment. Many were designed to validate a system or belief rather than test the unknown. In theory war games should be built to be free play oriented...in reality the aren’t.
- nabla9 8y ago
  Those are usually just exercises with adversarial component. The war games I'm thinking of are those what flag officers and staff do in RAND and other places. For example, they bring in diplomats and other outsiders to play the opponents. Or they may be pen and paper games (spreadsheets) scenario analysis where sequence of actions are played out and consequence of every decision are updated into the environment.

tedsanders 8y ago

As someone who has spent years trading in forecasting tournaments, I worry this characterization might be misinterpreted as saying know nothingism is correct and we should give up on predicting the future. (Not that you meant it that way, of course.) It's true that "amateurs" can consistently beat "experts". But not all amateurs - only some. This suggests to me there is true prediction expertise, but our system that attaches labels of "expert" does not select for it. Prediction is very much a skill and some people are very much better than others. We just don't regularly measure or select for it explicitly.

ssivark 8y ago
What about the complaint/criticism that Brier scores (which Tetlock uses to score forecasters) do not weight the severity/importance of the result eg: forecasting the outcome of a war and forecasting the winner of the super bowl are weighted equally.
- tedsanders 8y ago
  Totally valid criticism. I think nuances like these were what Tetlock had toughest time articulating in Superforecasting and his public interviews. He chose to emphasize the big, clear points rather than subtler, more complicated issues like how questions were chosen and how they should be compared or weighted.