4 ms·

"First, a bunch of generic smart people on Twitter who got things exactly right... 'Bill Gates, Balaji Srinivasan, Paul Graham, Greg Cochran, Robin Hanson, Sara

"First, a bunch of generic smart people on Twitter who got things exactly right... 'Bill Gates, Balaji Srinivasan, Paul Graham, Greg Cochran, Robin Hanson, Sarah Constantin, Eliezer Yudkowsky, and Nicholas Christakis....'" And "...they didn’t beat the experts in epidemiology. Whatever probability of pandemic the experts and prediction markets gave for coronavirus getting really bad, these people didn’t necessarily give a higher probability. They were just better at probabilistic reasoning, so they had different reactions to the same number..." I noticed that. As a math person, things were exceptionally clear at a point, clearer than many uncertain events tbh. Simple exponential growth this, I can understand ... and so could a significant slice of people. But my question is: if epidemiologists aren't experts at THIS, wtf are they experts in? Isn't this exactly the "classic" circumstance any average epidemiologist should be a able to see in a heartbeat. Apparently not, so what do they train these people in? Question maybe half rhetorical but also serious. What's "expert knowledge" in these circumstances.

__s 7y ago
So, there's a basis for "Goofus" mentality: If you're trying to compile a set of facts, you very likely want a system which is incomplete, but correct. ie false positives have a significant cost to your models, where you essentially have a technical debt of needing to weed out false facts Policy should follow risk analysis, but research which is not time sensitive has compounding factors which favor building certainty before skipping ahead Think of it like software development: planning/testing/etc have high rewards throughout development, but sometimes you need to sell a rushed MVP
tlb 7y ago
Professions exist that effectively have negative knowledge about their field, in the sense that they do worse than smart generalists at making predictions. Closest to home, I'd say Software Engineering mostly fits this description. 90% of people who call themselves that are terrible at actually making software. Or, for example, most of the best fiction is not written by English Literature majors. Perhaps epidemiology is like that. Based on what I've seen in the last few months, I'm not impressed.
- downerending 7y ago
  Arguably 90% of all people who make software are terrible at actually making software.
bsder 7y ago
> "First, a bunch of generic smart people on Twitter who got things exactly right... 'Bill Gates, Balaji Srinivasan, Paul Graham, Greg Cochran, Robin Hanson, Sarah Constantin, Eliezer Yudkowsky, and Nicholas Christakis....'" What you fail to mention is that had the US Federal Government gotten this right, all those people would have been wrong. Many people assumed that the CDC/NIH/etc. was going to react to this like Ebola and SARS. Now THAT was a mistaken assumption.
- xyzzyz 7y ago
  What did CDC do differently for Ebola and SARS? My impression is that not much, it’s just we got lucky back then.
- lonelappde 7y ago
  Totally different transmission model parameters. Ebola especially is much much harder to transmit.
nl 7y ago
epidemiologists aren't experts at THIS, wtf are they experts in? They are experts, but aren't used to rapid analysis with incomplete data. The very early data (early-mid January) was very unclear. This is completely normal with a new disease (just look at how the US has to rely on newspapers and universities for national reporting numbers!) But epidemiologists are reluctant to guess when we weren't even sure about the transmission mechanism, how infectious it is, incubation times, or even hospitalisation numbers (don't forget it's a respiratory disease and there was no test available until well into January) (Not an epidemiologist, but I've had papers published by the CDC on epidemiology)
- yxhuvud 7y ago
  As a bonus, they probably got burnt a bit by the bird flu nothingburger. That will be conductive neither of them speaking up when not being certain and of politicians listening to them.
  - lonelappde 7y ago
    Instead of bird flu being a lucky near-miss wake-up call (it spawned Contagion which laid out the issue in an easy to digest movie format!), it became an excuse to hit the snooze button.
    - nl 7y ago
      That's a bad take: even in the US the CDC built a bunch of knowledge and systems ready to respond to another epidemic like bird flu. Unfortunately it all got thrown away over the last couple of years. Other countries that were hit by it and by SARS or MERS were ready for it.
- abdullahkhalids 7y ago
  A better way to think about this is that an epidemiologists take the inside view (because they are experts at the inside view) while the generalists take the outside view (because that is what they are good at). In scenarios with high uncertainty in parameters, the outside view can perform as well as or better than the inside view. https://wiki.lesswrong.com/wiki/Outside_view https://wiki.lesswrong.com/wiki/Outside_view
kragen 7y ago
I extrapolated simple exponential growth wrong lots of times, including during the pandemic. You can see where it's going until it stops being exponential, and it's ferociously hard to tell when that's going to happen, but it always does happen, roughly. I think you may be misreading the article. It claims epidemiologists did, in fact, see this in a heartbeat.
- lonelappde 7y ago
  >> "They were just better at probabilistic reasoning, so they had different reactions to the same number..."
- joe_the_user 7y ago
  There are a couple parts to the "getting it right" part. One is seeing that exponential growth is involved. The second is seeing the fatality statistics are going to much more reliable than test results and that fatality statistics are effectively delayed 2-3 weeks for the virus to incubate. With this, Tomas Pueyo laid out the case as well as anyone but these numbers are easy to get, why all the listed people were saying this. If you didn't get, sorry about that, it's a failure on a problem no harder than an advanced undergraduate modeling class final. The situation was obvious to that degree of obvious - which I think is much more obvious than the likelihood of most moderately common events in modern society. And further, I'd claim there aren't many places in the world where you can see exponential growth stopping - certainly not in deaths in the US - 1000 death in NYC today. US cases have slowed their growth 'cause US testing is once again a mess. Even California doesn't have much evidence of a peak and reasonable evidence exponential continues. So maybe Italy, Spain, South Korea, China and Washington state are not experiencing exponential. And that's not comforting. And you're wrong about the article - it definitely says the mainstream epidemiologists got it wrong 'cause they did. https://medium.com/@tomaspueyo/coronavirus-act-today-or-people-will-die-f4d3d9cd99ca https://medium.com/@tomaspueyo/coronavirus-act-today-or-peop...
  - kragen 7y ago
    Well, when I wrote https://is.gd/2019nCoV https://is.gd/2019nCoV there weren't a lot of fatality statistics yet, precisely because of the delay you mention. I could argue that that's part of why my predictions were so wrong, but I knew that the test results were pretty unreliable. When Tomas wrote his article a month and half later, on March 10, there were a lot more statistics available. The world and US situation was extremely obvious at that point, although that didn't stop the New York Times and Vox and Donald Trump and such bullshit factories from continuing to dismiss the problem. But, ten days after Tomas's article, on March 20, I was still very concerned about a friend in New Zealand whose partner was due to deliver a child in 5 weeks, because New Zealand's confirmed case numbers were growing exponentially at 30% per day, same as they do in every country at first — but, again, it was too early to have any real death statistics for New Zealand. So I projected 5 weeks into the future and predicted the kind of apocalyptic healthcare breakdown we had seen in China and were starting to see in Italy and Spain: hundreds of thousands of confirmed COVID-19 cases overwhelming the healthcare system in New Zealand. Instead, the exponential growth in New Zealand continued for only another week, and then something stopped it — probably the lockdown measures that were put in place on March 25, although there's still the unlikely possibility of a very large number of asymptomatic cases (as suggested by the recent Stanford preprint) or existing cross-immunity from some less fatal coronavirus. My friend's partner delivered a healthy baby girl a couple of weeks early on April 10, at which point there were only 1283 total confirmed cases in New Zealand, a number growing only 3.6% per day. Currently NZ has 1431 confirmed cases and 12 deaths — worse than Taiwan, but still a shining paragon of pandemic control compared to almost any country in Europe. So, did I "get things exactly right" when I urged my friend to consult obstetricians about the possibility of inducing labor early, and to get whatever training he could in assisting homebirths in case that was necessary? Certainly I was urging him to prepare for a situation that did not in fact come to pass. Could it have come to pass? Certainly nothing I knew about NZ excluded the possibility or made it very unlikely — perhaps there were things he knew about NZ that excluded the possibility. I think the best course of action, faced with uncertainties of such enormity, is usually to make preparations that will be unnecessary 90% or 95% of the time — not so much that you will devote the majority of your resources to those preparations, but enough that you are well prepared on the occasions where catastrophe does strike. But my extrapolation of exponential growth definitely did not come to pass. In that sense, I extrapolated simple exponential growth wrong. I'm very interested to see what projections for this pandemic you committed to in January and February, since you're saying that what would happen was "exceptionally clear" for you. Where did you post them? — ⁂ — As for what the epidemiologists said, you literally quoted the part of the article where Alexander claims that "these people [who got things exactly right] didn't necessarily give a higher probability [than the experts [in epidemiology] and prediction markets gave for coronavirus getting really bad]". That is, the epidemiologists gave the same probability distribution over likely outcomes as the "generic smart people who got things exactly right". To me, that reads as exactly the opposite of saying "the mainstream epidemiologists got it wrong". And in fact the mainstream epidemiologists, if by that we mean the WHO, declared COVID-19 a "public health emergency of international concern" on January 30th, an action they had taken only five times before in history. Ignorant dilettantes writing articles in Vox or the New York Times, or running the country's government like a reality TV show, are not mainstream epidemiologists. So, I don't think either the article or objective facts support your thesis that "the mainstream epidemiologists got it wrong"; what makes you think otherwise?
watwut 7y ago
But ... epidemiologists did seem epidemic back then and were warning about it ... They argued about models and about R0 and so on. But fundamentally, they predicted the spread.
BaronSamedi 7y ago
"First, a bunch of generic smart people on Twitter who got things exactly right" I think it is far too early to declare anyone "exactly right". We won't know for months, maybe years; maybe we will never know because it will be impossible to disentangle the effects of the virus and the response. The more fundamental problem, in my view, is tunnel-vision on the epidemiological models. The epidemiological models represent, or should represent, input into a higher-level model that we might call the covid overall impact model. Such a model would take into account second and third order effects, it would include not just the virus but also the effects produced by the response. It would factor in economic, political, psychological, non-covid health, and broad societal effects. This is the model and perspective that really matters.
shalmanese 7y ago
> But my question is: if epidemiologists aren't experts at THIS, wtf are they experts in? Isn't this exactly the "classic" circumstance any average epidemiologist should be a able to see in a heartbeat. They're experts at remaining epidemiologists. Epidemiology is a lose-lose profession. If you overreact, then you get blamed. If you underreact, then you get blamed. Every epidemiologist who was willing to put their predictions in clear, unambiguous language and put their reputations on the line to urge decisive action was drummed out when MERS didn't turn out to be a big deal or Ebola didn't turn out to be a big deal or when H1N1 didn't turn out to be a big deal or when SARS didn't turn out to be a big deal. Thus, what remains are people putting their predictions in mush mouthed, carefully qualified scientific language that is all technically correct but carefully calibrated not to encourage anyone to do anything. First time entrants have an advantage in the field in that they don't care if they get drummed out.

__s 7y ago

So, there's a basis for "Goofus" mentality: If you're trying to compile a set of facts, you very likely want a system which is incomplete, but correct. ie false positives have a significant cost to your models, where you essentially have a technical debt of needing to weed out false facts Policy should follow risk analysis, but research which is not time sensitive has compounding factors which favor building certainty before skipping ahead Think of it like software development: planning/testing/etc have high rewards throughout development, but sometimes you need to sell a rushed MVP

tlb 7y ago

Professions exist that effectively have negative knowledge about their field, in the sense that they do worse than smart generalists at making predictions. Closest to home, I'd say Software Engineering mostly fits this description. 90% of people who call themselves that are terrible at actually making software. Or, for example, most of the best fiction is not written by English Literature majors. Perhaps epidemiology is like that. Based on what I've seen in the last few months, I'm not impressed.

downerending 7y ago
Arguably 90% of all people who make software are terrible at actually making software.

bsder 7y ago

> "First, a bunch of generic smart people on Twitter who got things exactly right... 'Bill Gates, Balaji Srinivasan, Paul Graham, Greg Cochran, Robin Hanson, Sarah Constantin, Eliezer Yudkowsky, and Nicholas Christakis....'" What you fail to mention is that had the US Federal Government gotten this right, all those people would have been wrong. Many people assumed that the CDC/NIH/etc. was going to react to this like Ebola and SARS. Now THAT was a mistaken assumption.

xyzzyz 7y ago
What did CDC do differently for Ebola and SARS? My impression is that not much, it’s just we got lucky back then.
lonelappde 7y ago
Totally different transmission model parameters. Ebola especially is much much harder to transmit.

nl 7y ago

epidemiologists aren't experts at THIS, wtf are they experts in? They are experts, but aren't used to rapid analysis with incomplete data. The very early data (early-mid January) was very unclear. This is completely normal with a new disease (just look at how the US has to rely on newspapers and universities for national reporting numbers!) But epidemiologists are reluctant to guess when we weren't even sure about the transmission mechanism, how infectious it is, incubation times, or even hospitalisation numbers (don't forget it's a respiratory disease and there was no test available until well into January) (Not an epidemiologist, but I've had papers published by the CDC on epidemiology)

yxhuvud 7y ago
As a bonus, they probably got burnt a bit by the bird flu nothingburger. That will be conductive neither of them speaking up when not being certain and of politicians listening to them.
- lonelappde 7y ago
  Instead of bird flu being a lucky near-miss wake-up call (it spawned Contagion which laid out the issue in an easy to digest movie format!), it became an excuse to hit the snooze button.
  - nl 7y ago
    That's a bad take: even in the US the CDC built a bunch of knowledge and systems ready to respond to another epidemic like bird flu. Unfortunately it all got thrown away over the last couple of years. Other countries that were hit by it and by SARS or MERS were ready for it.
abdullahkhalids 7y ago
A better way to think about this is that an epidemiologists take the inside view (because they are experts at the inside view) while the generalists take the outside view (because that is what they are good at). In scenarios with high uncertainty in parameters, the outside view can perform as well as or better than the inside view. https://wiki.lesswrong.com/wiki/Outside_view https://wiki.lesswrong.com/wiki/Outside_view

kragen 7y ago

I extrapolated simple exponential growth wrong lots of times, including during the pandemic. You can see where it's going until it stops being exponential, and it's ferociously hard to tell when that's going to happen, but it always does happen, roughly. I think you may be misreading the article. It claims epidemiologists did, in fact, see this in a heartbeat.

lonelappde 7y ago
>> "They were just better at probabilistic reasoning, so they had different reactions to the same number..."
joe_the_user 7y ago
There are a couple parts to the "getting it right" part. One is seeing that exponential growth is involved. The second is seeing the fatality statistics are going to much more reliable than test results and that fatality statistics are effectively delayed 2-3 weeks for the virus to incubate. With this, Tomas Pueyo laid out the case as well as anyone but these numbers are easy to get, why all the listed people were saying this. If you didn't get, sorry about that, it's a failure on a problem no harder than an advanced undergraduate modeling class final. The situation was obvious to that degree of obvious - which I think is much more obvious than the likelihood of most moderately common events in modern society. And further, I'd claim there aren't many places in the world where you can see exponential growth stopping - certainly not in deaths in the US - 1000 death in NYC today. US cases have slowed their growth 'cause US testing is once again a mess. Even California doesn't have much evidence of a peak and reasonable evidence exponential continues. So maybe Italy, Spain, South Korea, China and Washington state are not experiencing exponential. And that's not comforting. And you're wrong about the article - it definitely says the mainstream epidemiologists got it wrong 'cause they did. https://medium.com/@tomaspueyo/coronavirus-act-today-or-people-will-die-f4d3d9cd99ca https://medium.com/@tomaspueyo/coronavirus-act-today-or-peop...
- kragen 7y ago
  Well, when I wrote https://is.gd/2019nCoV https://is.gd/2019nCoV there weren't a lot of fatality statistics yet, precisely because of the delay you mention. I could argue that that's part of why my predictions were so wrong, but I knew that the test results were pretty unreliable. When Tomas wrote his article a month and half later, on March 10, there were a lot more statistics available. The world and US situation was extremely obvious at that point, although that didn't stop the New York Times and Vox and Donald Trump and such bullshit factories from continuing to dismiss the problem. But, ten days after Tomas's article, on March 20, I was still very concerned about a friend in New Zealand whose partner was due to deliver a child in 5 weeks, because New Zealand's confirmed case numbers were growing exponentially at 30% per day, same as they do in every country at first — but, again, it was too early to have any real death statistics for New Zealand. So I projected 5 weeks into the future and predicted the kind of apocalyptic healthcare breakdown we had seen in China and were starting to see in Italy and Spain: hundreds of thousands of confirmed COVID-19 cases overwhelming the healthcare system in New Zealand. Instead, the exponential growth in New Zealand continued for only another week, and then something stopped it — probably the lockdown measures that were put in place on March 25, although there's still the unlikely possibility of a very large number of asymptomatic cases (as suggested by the recent Stanford preprint) or existing cross-immunity from some less fatal coronavirus. My friend's partner delivered a healthy baby girl a couple of weeks early on April 10, at which point there were only 1283 total confirmed cases in New Zealand, a number growing only 3.6% per day. Currently NZ has 1431 confirmed cases and 12 deaths — worse than Taiwan, but still a shining paragon of pandemic control compared to almost any country in Europe. So, did I "get things exactly right" when I urged my friend to consult obstetricians about the possibility of inducing labor early, and to get whatever training he could in assisting homebirths in case that was necessary? Certainly I was urging him to prepare for a situation that did not in fact come to pass. Could it have come to pass? Certainly nothing I knew about NZ excluded the possibility or made it very unlikely — perhaps there were things he knew about NZ that excluded the possibility. I think the best course of action, faced with uncertainties of such enormity, is usually to make preparations that will be unnecessary 90% or 95% of the time — not so much that you will devote the majority of your resources to those preparations, but enough that you are well prepared on the occasions where catastrophe does strike. But my extrapolation of exponential growth definitely did not come to pass. In that sense, I extrapolated simple exponential growth wrong. I'm very interested to see what projections for this pandemic you committed to in January and February, since you're saying that what would happen was "exceptionally clear" for you. Where did you post them? — ⁂ — As for what the epidemiologists said, you literally quoted the part of the article where Alexander claims that "these people [who got things exactly right] didn't necessarily give a higher probability [than the experts [in epidemiology] and prediction markets gave for coronavirus getting really bad]". That is, the epidemiologists gave the same probability distribution over likely outcomes as the "generic smart people who got things exactly right". To me, that reads as exactly the opposite of saying "the mainstream epidemiologists got it wrong". And in fact the mainstream epidemiologists, if by that we mean the WHO, declared COVID-19 a "public health emergency of international concern" on January 30th, an action they had taken only five times before in history. Ignorant dilettantes writing articles in Vox or the New York Times, or running the country's government like a reality TV show, are not mainstream epidemiologists. So, I don't think either the article or objective facts support your thesis that "the mainstream epidemiologists got it wrong"; what makes you think otherwise?

watwut 7y ago

But ... epidemiologists did seem epidemic back then and were warning about it ... They argued about models and about R0 and so on. But fundamentally, they predicted the spread.

BaronSamedi 7y ago

"First, a bunch of generic smart people on Twitter who got things exactly right" I think it is far too early to declare anyone "exactly right". We won't know for months, maybe years; maybe we will never know because it will be impossible to disentangle the effects of the virus and the response. The more fundamental problem, in my view, is tunnel-vision on the epidemiological models. The epidemiological models represent, or should represent, input into a higher-level model that we might call the covid overall impact model. Such a model would take into account second and third order effects, it would include not just the virus but also the effects produced by the response. It would factor in economic, political, psychological, non-covid health, and broad societal effects. This is the model and perspective that really matters.

shalmanese 7y ago

> But my question is: if epidemiologists aren't experts at THIS, wtf are they experts in? Isn't this exactly the "classic" circumstance any average epidemiologist should be a able to see in a heartbeat. They're experts at remaining epidemiologists. Epidemiology is a lose-lose profession. If you overreact, then you get blamed. If you underreact, then you get blamed. Every epidemiologist who was willing to put their predictions in clear, unambiguous language and put their reputations on the line to urge decisive action was drummed out when MERS didn't turn out to be a big deal or Ebola didn't turn out to be a big deal or when H1N1 didn't turn out to be a big deal or when SARS didn't turn out to be a big deal. Thus, what remains are people putting their predictions in mush mouthed, carefully qualified scientific language that is all technically correct but carefully calibrated not to encourage anyone to do anything. First time entrants have an advantage in the field in that they don't care if they get drummed out.