3 ms·

> A viral article implores us to “flatten the curve of armchair epidemiology”—that is, to listen only to authoritive sources like the CDC, not random people spo

> A viral article implores us to “flatten the curve of armchair epidemiology”—that is, to listen only to authoritive sources like the CDC, not random people spouting on social media. > This was notable to me for being the diametric opposite of the actual lesson of the past two months. So, because some (unspecified, unnamed) experts weren't able to predict the future with perfect accuracy, we should give equal credence to "random people" instead? Medical professionals, scientists, and public health experts are still the best available source of knowledge about preventing the growth of this pandemic. Spreading the message that we should ignore epidemiologists is wrong at best and dangerous at worst.

jasonhansel 7y ago
(I'm sure that epidemiologists reading posts like the OP feel a lot like software devs reading opinion columns about cybersecurity or cryptocurrencies.)
fiblye 7y ago
>So, because some (unspecified, unnamed) experts weren't able to predict the future with perfect accuracy, we should give equal credence to "random people" instead? That wasn't my takeaway. My takeaway is that if you have some intuitive sense that shit's about to hit the fan and authorities keep telling you to stay calm and don't take any special precautions, all while danger is clearly growing, it's best to lean towards following your intuitions gradually just in case things ramp up. You shouldn't buy 10000 rolls of toilet paper, but all those people who thought, "Hmmm, maybe I should buy some masks and a couple extra bottles of hand sanitizer. Maybe not visit the movie theater anytime soon as well" while authorities were saying "Go to work and be normal. Masks are useless"... well, they were right and it would've been better to take those then "unnecessary" precautions. Reminds me of people talking about government surveillance pre and post Snowden. If you wanted to encrypt all your messages and avoid Facebook, etc, you were a conspiracy nut because it was impossible to surveil everything. Overnight it became "Of course they monitor everything. Why wouldn't they? But we have end-to-end encryption now and we can completely trust it."
- deleted 7y ago
  [deleted]
gentleman11 7y ago
The official experts are often partially under the control of their governments and their politics. Eg, the doctors getting arrested in China for warning people. Sometimes, these agencies are just overconfident too, like how all our health agencies saying over and over that we have ample capacity and supplies - then New York happens. The experts are definitely what we need, but not all experts are created equal. It’s like during wwi - it took a while to weed out the useless commanders/generals since the peacetime systems failed to promote based on ability (Edit: Incidentally, I think we have some great people in Alberta at the moment. Very lucky for us)
knzhou 7y ago
Exactly, cherry picking the most optimistic thing the most optimistic expert said two months ago is a tremendously unfair way of evaluating the field.
btilly 7y ago
That isn't what Scott is saying. His default was to listen to those in authority and responsible. Instead he wishes that he listened to those who did independent thinking and their own projections with verifiable reasoning from the data. This doesn't mean don't listen to epidemiologists. There were plenty of epidemiologists who were plenty concerned in early February. It means do not pay particular heed to epidemiologists who were selected to be listened to via a politically motivated process. And my own editorial. Go read the book Superforecasting. The people that Scott is recommending listening to are exactly the kind of people who do better at making short term forecasts than the majority of recognized experts. And the reason why their forecasts are better is exactly the same as why they got it right this time. Quantitative thinking is something we as a society are still learning to trust. We've come to trust it in baseball. (Go read Moneyball.) We've come to respect it in politics. (Nate Silver established a following for a reason.) But we have yet to internalize as second nature the idea that we should sanity check everything with people who clearly think in a quantitative way. And that we should attempt that kind of thought ourselves. In the long run I believe we will. And perhaps reflecting on who got COVID right and who got it wrong will help us get there sooner.
- jasonhansel 7y ago
  Did people "thinking independently" and "making their own projections" do a better job--on average, not just looking at the outliers--than the more established experts? I don't think we have any empirical data to support that claim.
  - thu2111 7y ago
    Probably not but consider the likely reason: it seems there's no data right now of sufficient quality for making projections. The projections we have been seeing are all based on the same dodgy stats that we read about new problems with every day. A truly quantitatively minded person would evaluate the data quality and not try to build models to begin with. I think I agree with btilly. A lot of people are upset right now by the lack of traditional deference to experts, by which they mean academics, not e.g. actuaries working for large reinsurers or even actual working doctors in many cases. In contrast I find it exciting. What we're witnessing here is a total and complete democratisation of data and analysis. An entire planet's worth of brainpower is focused on this subject right now. Every day those people are using the internet to quickly sift through data, statistics, reporting and analysis to try and make sense of the chaotic picture with which we're presented. Some are trying to establish the bounds on what we know about the infection, others are looking at how to rapidly scale ventilator manufacture and so on. If you've followed the replication crisis closely over the years, like I have, one of the overriding themes is how academic analysis is made brittle by: 1. The subtlety and trickiness of statistics. 2. The vast extent to which it's relied on despite this difficulty. 3. The relative lack of cross-discipline collaboration that could solve the combination of 1+2. Lots of fields (not restricted to psychology) have been seriously battered by the collapse of whole research areas. I've heard that one reason VCs prefer to invest in software startups over biotech is the typical biotech startup begins with an academic paper, and around 50% of the time it doesn't replicate. Even AI has had replication issues! One of the conclusions that's been drawn is that many academics don't have enough statistical training to do what they're trying to do, partly because statistics is genuinely extremely hard. In computer science the general maxim that you don't invent your own ciphers because they'll probably break is well known - everyone relies on standard cryptography made by a relatively small community of people these days. Advanced statistics feels about that level of difficulty to me, based on what I've seen, but everyone rolls their own stats. Perhaps there aren't enough statisticians to go round, but there's also a cultural issue. For instance academic modellers rarely work with professional programmers to productionise their models, a practice that's standard in the private sector. So yeah, btilly has it right. We're seeing a slow, slow, veeerry slow recognition that truly quant thinking is so rare and highly valuable that it's more important than the background of the person. Finance already went through this process years ago: the clear-out of traditional finance types with traditional finance culture in favour of quants doing mathematical analysis is large done already. In other areas of the economy it's barely got started. In the UK Cummings did at least try to hire a superforecaster into Number 10, to his credit, but the media immediately went bezerk and engaged in a massive smear campaign. The guy just walked away in disgust. A superforecaster in government sure would be useful right about now :(
vincentmarle 7y ago
This is typical game theory: epidemiologists and other authorities care about the collective outcome (reducing the # of deaths, the economy, PPE for medical workers etc) while individuals mostly care about their own safety, and those goals aren't always aligned. This is why it’s not necessarily wise to listen to their “advice” of not wearing masks or keep going to work, because they’re optimizing for different things than you should do for yourself. For example, look at the disastrous “herd immunity” experiments that countries like the UK and the Netherlands are trying to conduct at the cost of many lives.

jasonhansel 7y ago

(I'm sure that epidemiologists reading posts like the OP feel a lot like software devs reading opinion columns about cybersecurity or cryptocurrencies.)

fiblye 7y ago

>So, because some (unspecified, unnamed) experts weren't able to predict the future with perfect accuracy, we should give equal credence to "random people" instead? That wasn't my takeaway. My takeaway is that if you have some intuitive sense that shit's about to hit the fan and authorities keep telling you to stay calm and don't take any special precautions, all while danger is clearly growing, it's best to lean towards following your intuitions gradually just in case things ramp up. You shouldn't buy 10000 rolls of toilet paper, but all those people who thought, "Hmmm, maybe I should buy some masks and a couple extra bottles of hand sanitizer. Maybe not visit the movie theater anytime soon as well" while authorities were saying "Go to work and be normal. Masks are useless"... well, they were right and it would've been better to take those then "unnecessary" precautions. Reminds me of people talking about government surveillance pre and post Snowden. If you wanted to encrypt all your messages and avoid Facebook, etc, you were a conspiracy nut because it was impossible to surveil everything. Overnight it became "Of course they monitor everything. Why wouldn't they? But we have end-to-end encryption now and we can completely trust it."

deleted 7y ago
[deleted]

gentleman11 7y ago

The official experts are often partially under the control of their governments and their politics. Eg, the doctors getting arrested in China for warning people. Sometimes, these agencies are just overconfident too, like how all our health agencies saying over and over that we have ample capacity and supplies - then New York happens. The experts are definitely what we need, but not all experts are created equal. It’s like during wwi - it took a while to weed out the useless commanders/generals since the peacetime systems failed to promote based on ability (Edit: Incidentally, I think we have some great people in Alberta at the moment. Very lucky for us)

knzhou 7y ago

Exactly, cherry picking the most optimistic thing the most optimistic expert said two months ago is a tremendously unfair way of evaluating the field.

btilly 7y ago

That isn't what Scott is saying. His default was to listen to those in authority and responsible. Instead he wishes that he listened to those who did independent thinking and their own projections with verifiable reasoning from the data. This doesn't mean don't listen to epidemiologists. There were plenty of epidemiologists who were plenty concerned in early February. It means do not pay particular heed to epidemiologists who were selected to be listened to via a politically motivated process. And my own editorial. Go read the book Superforecasting. The people that Scott is recommending listening to are exactly the kind of people who do better at making short term forecasts than the majority of recognized experts. And the reason why their forecasts are better is exactly the same as why they got it right this time. Quantitative thinking is something we as a society are still learning to trust. We've come to trust it in baseball. (Go read Moneyball.) We've come to respect it in politics. (Nate Silver established a following for a reason.) But we have yet to internalize as second nature the idea that we should sanity check everything with people who clearly think in a quantitative way. And that we should attempt that kind of thought ourselves. In the long run I believe we will. And perhaps reflecting on who got COVID right and who got it wrong will help us get there sooner.

jasonhansel 7y ago
Did people "thinking independently" and "making their own projections" do a better job--on average, not just looking at the outliers--than the more established experts? I don't think we have any empirical data to support that claim.
- thu2111 7y ago
  Probably not but consider the likely reason: it seems there's no data right now of sufficient quality for making projections. The projections we have been seeing are all based on the same dodgy stats that we read about new problems with every day. A truly quantitatively minded person would evaluate the data quality and not try to build models to begin with. I think I agree with btilly. A lot of people are upset right now by the lack of traditional deference to experts, by which they mean academics, not e.g. actuaries working for large reinsurers or even actual working doctors in many cases. In contrast I find it exciting. What we're witnessing here is a total and complete democratisation of data and analysis. An entire planet's worth of brainpower is focused on this subject right now. Every day those people are using the internet to quickly sift through data, statistics, reporting and analysis to try and make sense of the chaotic picture with which we're presented. Some are trying to establish the bounds on what we know about the infection, others are looking at how to rapidly scale ventilator manufacture and so on. If you've followed the replication crisis closely over the years, like I have, one of the overriding themes is how academic analysis is made brittle by: 1. The subtlety and trickiness of statistics. 2. The vast extent to which it's relied on despite this difficulty. 3. The relative lack of cross-discipline collaboration that could solve the combination of 1+2. Lots of fields (not restricted to psychology) have been seriously battered by the collapse of whole research areas. I've heard that one reason VCs prefer to invest in software startups over biotech is the typical biotech startup begins with an academic paper, and around 50% of the time it doesn't replicate. Even AI has had replication issues! One of the conclusions that's been drawn is that many academics don't have enough statistical training to do what they're trying to do, partly because statistics is genuinely extremely hard. In computer science the general maxim that you don't invent your own ciphers because they'll probably break is well known - everyone relies on standard cryptography made by a relatively small community of people these days. Advanced statistics feels about that level of difficulty to me, based on what I've seen, but everyone rolls their own stats. Perhaps there aren't enough statisticians to go round, but there's also a cultural issue. For instance academic modellers rarely work with professional programmers to productionise their models, a practice that's standard in the private sector. So yeah, btilly has it right. We're seeing a slow, slow, veeerry slow recognition that truly quant thinking is so rare and highly valuable that it's more important than the background of the person. Finance already went through this process years ago: the clear-out of traditional finance types with traditional finance culture in favour of quants doing mathematical analysis is large done already. In other areas of the economy it's barely got started. In the UK Cummings did at least try to hire a superforecaster into Number 10, to his credit, but the media immediately went bezerk and engaged in a massive smear campaign. The guy just walked away in disgust. A superforecaster in government sure would be useful right about now :(

vincentmarle 7y ago

This is typical game theory: epidemiologists and other authorities care about the collective outcome (reducing the # of deaths, the economy, PPE for medical workers etc) while individuals mostly care about their own safety, and those goals aren't always aligned. This is why it’s not necessarily wise to listen to their “advice” of not wearing masks or keep going to work, because they’re optimizing for different things than you should do for yourself. For example, look at the disastrous “herd immunity” experiments that countries like the UK and the Netherlands are trying to conduct at the cost of many lives.