2 ms·

There is nothing about what has happened that should surprise anyone who has read the book Superforecasting. It explains that we naturally trust people who sou

by btilly 7y ago

There is nothing about what has happened that should surprise anyone who has read the book Superforecasting. It explains that we naturally trust people who sound smart, well-informed, and CONFIDENT. We don't want to hear uncertainty, probabilities, or the other signs of someone who thinks in a careful quantitative way. We want to accept a cognitively simple answer, then move on. This is what we find comfortable. However this is a good way to select people who are terrible at making actual predictions. They appear to predict, but often with sufficient weasel words that it is hard afterwards to say whether it was violated. (The book gives real examples.) But if you put them in a setting where they can be tested, they perform worse than uninformed monkeys. And the part of the future that they are worst at predicting is exactly what they were supposed to be experts at! The book Superforecasting walks through how this was demonstrated, and the discovery that there are people you will never see on CNN or Fox news who are really good at forecasting. A fact that is extremely interesting to various TLA agencies (one of whom paid for the research in question). The long and short of it? Bayes' Theorem actually works in the real world. The revolution that started with quants on wall street, analytics in baseball and Nate Silver in politics is still ongoing. When you are done with the book and have processed it, hopefully you will understand why the author said in response to an audience question after a talk, Here’s my long-term prediction for Long Now. When the Long Now audience of 2515 looks back on the audience of 2015, their level of contempt for how we go about judging political debate will be roughly comparable to the level of contempt we have for the 1692 Salem witch trials. Hopefully the contempt that some of us have for how talking heads in January and February of 2020 dismissed Coronavirus is a step on the path to that future.

maps7 7y ago
Based on your comment I went to buy Superforecasting on Audible. When I browsed to it I saw I already bought it on 29-04-19. Time to listen to it.
- thoughtstheseus 7y ago
  Great book. Also check out The Signal and the Noise by Nate Silver.
bo1024 7y ago
There is some really interesting interplay here between forecasting and decisionmaking. (Taleb would have a lot to say here, along the lines of "forecasters are poor.") Maybe it makes sense that forecasts should be measured, but decisions should be, well, decisive. A good Bayesian should be able to make confident decisions based on information available at the moment, while acknowledging that lack of information is leading to suboptimal decisions. For example, a leader can be absolutely confident that shelter-in-place is the best decision based on the available information, while acknowledging that there is missing information that would drastically change this assessment.
- pjscott 7y ago
  Fundamentally, decision-making is what predictions are for. We mainly care about information to guide our actions. There are some interesting implications of this for how we should do research. https://www.gwern.net/Research-criticism#beliefs-are-for-actions https://www.gwern.net/Research-criticism#beliefs-are-for-act... A recent example: people have been talking about clinical trials for coronavirus vaccine candidates. In those you want to minimize the bad things that happen to the people in the study, and also get a working vaccine rolled out to the world as quickly as possible. Therefore you might want to accept unusually low levels of certainty that the vaccine is safe, or ramp up trial size faster than usual, because the world is on fire and every day of delay is terrible. For other vaccines with smaller expected benefits, slow-and-cautious might be the way to go. In both cases it's a matter of balancing expected risks with rewards as your probability estimates change over time.
  - bo1024 7y ago
    Yes, exactly. For example, I would think there would be immense value right now in sampling totally random subsets of the population and testing whether they've had COVID-19 in the past. This is the kind of information that could radically change decisions about when to release stay-at-home orders.
    - btilly 7y ago
      I disagree on that. The purposes of sampling are to find out how deadly the disease is, and to find out if herd immunity exists. But we have good evidence that it is likely to be deadly enough to justify stay-at-home orders while community spread exists. And have strong circumstantial evidence that only a small fraction of the population has had it. Therefore breaking quarantine for a random subset of the population is unlikely to change actionable decisions. But it will be likely to spread the disease. I would love to know the answer to the question raised. But it isn't worth human lives to answer it sooner than it will otherwise be answered.
renewiltord 7y ago
This is why these last few weeks on the market have been wonderful. If you can predict (including accounting for the supposed irrationality) then you make money. Otherwise you lose money. If you had a prediction of how things would get hit w/ virus, there was so much money to be made. I watched but my timing was a bit off and I did not commit the majority of my wealth. That's a pity and speaks to my true estimate of the danger whereas a friend of mine pulled out completely of all long-term holdings in mid Feb so we know he believed.
divbzero 7y ago
> It explains that we naturally trust people who sound smart, well-informed, and CONFIDENT. We don't want to hear uncertainty, probabilities, or the other signs of someone who thinks in a careful quantitative way. We want to accept a cognitively simple answer, then move on. This is what we find comfortable. Almost any interview with Robert Shiller is a striking counter example to these smart confident sounding people who aren’t actually good at making predictions. When speaking Shiller does not sound very confident at all and you need to know his background and history and remind yourself to pay attention to what he’s saying.

by btilly 7y ago

maps7 7y ago

Based on your comment I went to buy Superforecasting on Audible. When I browsed to it I saw I already bought it on 29-04-19. Time to listen to it.

thoughtstheseus 7y ago
Great book. Also check out The Signal and the Noise by Nate Silver.

bo1024 7y ago

There is some really interesting interplay here between forecasting and decisionmaking. (Taleb would have a lot to say here, along the lines of "forecasters are poor.") Maybe it makes sense that forecasts should be measured, but decisions should be, well, decisive. A good Bayesian should be able to make confident decisions based on information available at the moment, while acknowledging that lack of information is leading to suboptimal decisions. For example, a leader can be absolutely confident that shelter-in-place is the best decision based on the available information, while acknowledging that there is missing information that would drastically change this assessment.

pjscott 7y ago
Fundamentally, decision-making is what predictions are for. We mainly care about information to guide our actions. There are some interesting implications of this for how we should do research. https://www.gwern.net/Research-criticism#beliefs-are-for-actions https://www.gwern.net/Research-criticism#beliefs-are-for-act... A recent example: people have been talking about clinical trials for coronavirus vaccine candidates. In those you want to minimize the bad things that happen to the people in the study, and also get a working vaccine rolled out to the world as quickly as possible. Therefore you might want to accept unusually low levels of certainty that the vaccine is safe, or ramp up trial size faster than usual, because the world is on fire and every day of delay is terrible. For other vaccines with smaller expected benefits, slow-and-cautious might be the way to go. In both cases it's a matter of balancing expected risks with rewards as your probability estimates change over time.
- bo1024 7y ago
  Yes, exactly. For example, I would think there would be immense value right now in sampling totally random subsets of the population and testing whether they've had COVID-19 in the past. This is the kind of information that could radically change decisions about when to release stay-at-home orders.
  - btilly 7y ago
    I disagree on that. The purposes of sampling are to find out how deadly the disease is, and to find out if herd immunity exists. But we have good evidence that it is likely to be deadly enough to justify stay-at-home orders while community spread exists. And have strong circumstantial evidence that only a small fraction of the population has had it. Therefore breaking quarantine for a random subset of the population is unlikely to change actionable decisions. But it will be likely to spread the disease. I would love to know the answer to the question raised. But it isn't worth human lives to answer it sooner than it will otherwise be answered.

renewiltord 7y ago

This is why these last few weeks on the market have been wonderful. If you can predict (including accounting for the supposed irrationality) then you make money. Otherwise you lose money. If you had a prediction of how things would get hit w/ virus, there was so much money to be made. I watched but my timing was a bit off and I did not commit the majority of my wealth. That's a pity and speaks to my true estimate of the danger whereas a friend of mine pulled out completely of all long-term holdings in mid Feb so we know he believed.

divbzero 7y ago

> It explains that we naturally trust people who sound smart, well-informed, and CONFIDENT. We don't want to hear uncertainty, probabilities, or the other signs of someone who thinks in a careful quantitative way. We want to accept a cognitively simple answer, then move on. This is what we find comfortable. Almost any interview with Robert Shiller is a striking counter example to these smart confident sounding people who aren’t actually good at making predictions. When speaking Shiller does not sound very confident at all and you need to know his background and history and remind yourself to pay attention to what he’s saying.