3 ms·

I think this article has two shortcomings that make its sweeping conclusions shaky. First, it identifies forecasting with point forecasting. There are other wa

by kbrkbr 3y ago

I think this article has two shortcomings that make its sweeping conclusions shaky. First, it identifies forecasting with point forecasting. There are other ways to put forecasting questions, e.g. lower and upper level with a certain probability. Also it mentions Tetlock, but only his negative findings, not his positive ones that lead to Good Judgement Project, which suggest the contrary of the conclusion of this article [1]. Thus I think it is not up to the latest research results. See you over at gjopen.com, if you are interested and have lots of time to waste... [1] https://en.m.wikipedia.org/wiki/The_Good_Judgment_Project https://en.m.wikipedia.org/wiki/The_Good_Judgment_Project

btilly 3y ago
Tetlock paints a different picture of his positive findings than you do. Specifically, Tetlock's project opens with key issues of scope about what to even try to forecast. Based on his previous work in expert prediction, he concluded that geopolitics is sufficiently chaotic to be impossible to predict 10 years out. So while he did a lot of work on forecasting, it is generally focused on the next year or to. Which means that Tetlock agrees that we can't predict 10 years out.
- kbrkbr 3y ago
  I agree with the assessment that there are not many systems we can predict 10 years out with great confidence, specifically geopolitics. But I do not think I painted much of a picture of Tetlock's results. I read the article as concluding: let's stop predicting, it does not work. Let's start building. (After stating we cannot predict this, and we cannot predict that.) And I think Tetlock"s result contradict that, as I said. Sometimes and under certain circumstances we can predict quite well.
  - mcshicks 3y ago
    I have a graph from "Expert Political Judgement" that I've kept on a cork board for over a decade. It's from page 55 in my edition. It charts "Objective Frequency" vs "Subjective Probability" It has three curves, Experts (people in Government, paid to make political assessments), Dilettantes (people who are well read, read NYT, WSJ and the like), and College Undergrads. The Expert and Dilettante lines are more or less on top of each other. The undergrads are observably much worse and farther from the "Perfect Calibration line" that is a 45 degree line between objective frequency and subjective probability. So it's not the case that there is no difference in people's ability to predict political events, it's that so called "experts" are no better than people who follow current events closely. This was for me the main takeaway from the book, is that nobody can predict political events very well, but some groups are measurably worse than others. Tetlock has a brief section that somewhat mirrors your argument on page 186 "Misunderstanding what game is being played" where one expert tells him making predictions is all about getting your sound bite out, not about being correct. In this game, stronger, incorrect predictions might be advantageous in that they can change the narrative.
    - qznc 3y ago
      In October was this chaotic election of the Speaker of the House. Indeed, a few people following the news closely where quite good in predicting the outcome: https://marketwise.substack.com/p/speaker-election-interviewing-the https://marketwise.substack.com/p/speaker-election-interview...
    - kqr 3y ago
      Right, and then he followed this up with Superforecasters which is all about the people who are on that 45 degree line. They exist! They just aren't popular.
- kqr 3y ago
  He's not saying you can't predict 10 years out, just that the appropriate prediction is the base rate. In other words, you cannot use information from today to improve predictions beyond long-term statistical generalities. But that doesn't mean the prediction is useless, only that it has great uncertainty.

by kbrkbr 3y ago

btilly 3y ago

Tetlock paints a different picture of his positive findings than you do. Specifically, Tetlock's project opens with key issues of scope about what to even try to forecast. Based on his previous work in expert prediction, he concluded that geopolitics is sufficiently chaotic to be impossible to predict 10 years out. So while he did a lot of work on forecasting, it is generally focused on the next year or to. Which means that Tetlock agrees that we can't predict 10 years out.

kbrkbr 3y ago
I agree with the assessment that there are not many systems we can predict 10 years out with great confidence, specifically geopolitics. But I do not think I painted much of a picture of Tetlock's results. I read the article as concluding: let's stop predicting, it does not work. Let's start building. (After stating we cannot predict this, and we cannot predict that.) And I think Tetlock"s result contradict that, as I said. Sometimes and under certain circumstances we can predict quite well.
- mcshicks 3y ago
  I have a graph from "Expert Political Judgement" that I've kept on a cork board for over a decade. It's from page 55 in my edition. It charts "Objective Frequency" vs "Subjective Probability" It has three curves, Experts (people in Government, paid to make political assessments), Dilettantes (people who are well read, read NYT, WSJ and the like), and College Undergrads. The Expert and Dilettante lines are more or less on top of each other. The undergrads are observably much worse and farther from the "Perfect Calibration line" that is a 45 degree line between objective frequency and subjective probability. So it's not the case that there is no difference in people's ability to predict political events, it's that so called "experts" are no better than people who follow current events closely. This was for me the main takeaway from the book, is that nobody can predict political events very well, but some groups are measurably worse than others. Tetlock has a brief section that somewhat mirrors your argument on page 186 "Misunderstanding what game is being played" where one expert tells him making predictions is all about getting your sound bite out, not about being correct. In this game, stronger, incorrect predictions might be advantageous in that they can change the narrative.
  - qznc 3y ago
    In October was this chaotic election of the Speaker of the House. Indeed, a few people following the news closely where quite good in predicting the outcome: https://marketwise.substack.com/p/speaker-election-interviewing-the https://marketwise.substack.com/p/speaker-election-interview...
  - kqr 3y ago
    Right, and then he followed this up with Superforecasters which is all about the people who are on that 45 degree line. They exist! They just aren't popular.
kqr 3y ago
He's not saying you can't predict 10 years out, just that the appropriate prediction is the base rate. In other words, you cannot use information from today to improve predictions beyond long-term statistical generalities. But that doesn't mean the prediction is useless, only that it has great uncertainty.