8 ms·

This isn't a "study", it's a statistical model. It reflects the assumptions put into it. To wit: > Our analysis follows four key steps. First, for all location

by timr 6y ago

This isn't a "study", it's a statistical model. It reflects the assumptions put into it. To wit: > Our analysis follows four key steps. First, for all locations where weekly or monthly all-cause mortality has been reported since the start of the pandemic, we estimate how much mortality increased compared to the expected death rate....Second, based on a range of studies and consideration of other evidence, we estimate the fraction of excess mortality that is from total COVID-19 deaths as opposed to the five other drivers that influence excess mortality. Third, we build a statistical model that predicts the weekly ratio of total COVID-19 deaths to reported COVID-19 deaths based on covariates and spatial effects. Fourth, we use this statistical relationship to predict the ratio of total to reported COVID-19 deaths in places without data on total COVID-19 deaths and then multiply the reported COVID-19 deaths by this ratio to generate estimates of total COVID-19 deaths for all locations. And, what, you may ask, are the biases of the models' authors with regards to the percentage of excess mortality due to Covid itself? Excellent question: > Deaths that are directly due to COVID-19 are likely underreported in many locations, particularly in settings where COVID-19 testing is in short supply. Most excess mortality is likely misclassified COVID-19 deaths. > Given that there is insufficient evidence to estimate these contributions to excess mortality, for now we assume that total COVID-19 deaths equal excess mortality. For the reasons presented in this section, we believe that this is likely an underestimate. As someone who has now spent the better part of his life creating statistical models, when you assume something is true, your model is likely to confirm your assumptions. This is why statistical models are worthless without prospective validation. If you haven't shown that your assumptions are correct for the future, you're just making castles in the sky.

grumpyautist 6y ago
Yep, and it's a suspect one at best. Personally I wouldn't trust a stastical model from someone who advertises that they own the book "how to lie with statistics" and obviously used it
- seoaeu 6y ago
  I trust someone who acknowledges that statistics can be used deceptively vastly more than I trust someone who doesn't.
  - deleted 6y ago
    [deleted]
- nathancahill 6y ago
  I mean that book is basically required reading for Statistics 101. Doesn't everyone have a copy?
  - deleted 6y ago
    [deleted]
    - deleted 6y ago
      [deleted]
  - morelisp 6y ago
    A comment from a high school math teacher concerning that book (which yes, was commonplace to read even decades ago, even among high school AP students) has stuck with me: "It's definitely possible to lie with statistics, but wait until you see how much you can get away with without them!"
- xhedley 6y ago
  The book "how to lie with statistics" (1954) was written before tags like /s were invented. Referencing "how to lie with statistics" is an indicator that the author is trying to avoid common pitfalls in statistical reasoning. I own an inherited blue Pelican paperbook copy from my pharmacist grandfather. https://en.wikipedia.org/wiki/How_to_Lie_with_Statistics https://en.wikipedia.org/wiki/How_to_Lie_with_Statistics
- tester756 6y ago
  >Personally I wouldn't trust a stastical model from someone who advertises that they own the book "how to lie with statistics" and obviously used it why?
newacct583 6y ago
Surely you'd agree that "2020 excess deaths are going to be predominantly covid" is a very reasonable hypothesis though, right? And you agree that "2020 has an extraordinarily high excess death rate", because that part is easily measured data, right? Basically: your response seems a bit nitpicky to me, which tells me that perhaps you're starting from a perspective where you're assuming the article is wrong and looking for holes to poke. Look, nothing about this paper's conclusions is surprising at all. At all. We all know true covid deaths are higher than reported, because all such metrics underreport, for all diseases (and frankly almost all causes of death). The only question is "By how much?". So you don't like 400k. What's your counter?
- timr 6y ago
  > Surely you'd agree that "2020 excess deaths are going to be predominantly covid" is a very reasonable hypothesis though, right? And you agree that "2020 has an extraordinarily high excess death rate", because that part is easily measured data, right? I don't know the answer, and neither do you. But given that the top-line is asserting that there are 900k excess deaths due to Covid, the choice of assumption here is not some minor detail. There are lots of other plausible alternative hypotheses. Aassuming X, then building a model that shows X is true, is not validating a hypothesis.
  - grecy 6y ago
    > There are lots of other plausible alternative hypotheses. Please list them. We have data about deaths going back many decades. It's very safe to say that if in 2010 you made a prediction about the number of expected deaths in 2011, you would have been very close to the real number. Same if you had predicted the deaths for 2019 in 2018. Suddenly, after decades and decades of being almost perfect in your predictions, you would have been wildly wrong for the predicted number of deaths in 2020. After decades of predictability, there was a whopping 20% increase in 2020 [1]. How can you possibly say it wasn't caused by Covid? [1] https://jamanetwork.com/journals/jama/fullarticle/2778361 https://jamanetwork.com/journals/jama/fullarticle/2778361
    - 6y ago
Sebb767 6y ago
> for now we assume that total COVID-19 deaths equal excess mortality. This part is especially doubtful for me, as there are factors moving it in either direction. For one, the pandemic measures also helped a lot for things like the flu, which had less cases and therefore caused less deaths. On the other hand, problems caused by lack of exercise and social interaction have definitely increased. Lastly, I'm quite sure we have a lot of missclassified deaths in either direction. These are all quite different and I see no reason to assume these factors even out.
- thebruce87m 6y ago
  For my country and some others I monitored, the first wave happened outside of flu season, and excess mortality was higher than COVID deaths at that point and continued to track. Second wave was more difficult since it overlapped the normal flu season. You can see graphs here showing normal flu seasons and where COVID fits in: https://www.euromomo.eu/graphs-and-maps/ https://www.euromomo.eu/graphs-and-maps/
- mattmanser 6y ago
  I wouldn't call it at all doubtful. The fact there's less cars on the road can only push excess mortality down, regardless of lack of exercise, so it seems sensible to attribute it all to the pandemic. Also, suicides aren't significantly up,the physical and psychological effects of lockdown don't happen immediately, while a reduction of car deaths do. e.g. UK suicides have not significantly changed in 2020: https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/quarterlysuicidedeathregistrationsinengland/2001to2019registrationsandquarter1jantomartoquarter3julytosept2020provisionaldata https://www.ons.gov.uk/peoplepopulationandcommunity/birthsde... So overall I think they're pretty safely assume that any excess deaths are coronavirus, especially as flu has all but disappeared too.
- philwelch 6y ago
  Some more factors might be hospital crowding early in the pandemic, people avoiding medical treatment, and increased homicide rates.
  - 6y ago
teachingassist 6y ago
> for now we assume that total COVID-19 deaths equal excess mortality. This seems reasonable, although I'd suggest it's an underestimate of COVID-19 deaths. Excess deaths counts have been consistently showing negative in countries during periods of time where they have coronavirus under control. Other significant causes of death (including suicide, influenza, road traffic accidents) appear to be down. As such, I'd assume that COVID-19 deaths are likely somewhat higher than excess mortality figures. I'm prepared to accept the assumption that they're approximately the same. [Edited to note: the linked article discusses this with further detail and evidence]
- pbreit 6y ago
  Excess deaths could easily be attributed to non-covid factors like Alzheimer’s. Suggesting the US had 900k covid deaths is ridiculous. Only 1 of the 6 considered factors lists covid as a direct cause.
  - joshuamorton 6y ago
    > Excess deaths could easily be attributed to non-covid factors like Alzheimer’s. Why would there suddenly be an 100% YoY increase in the number of Alzheimers deaths?
    - craftinator 6y ago
      Because we like to ignore facts that counter our worldviews.
    - pbreit 6y ago
      Hospitals and doctors were basically not handling any non-covid patients.
      - joshuamorton 6y ago
        Alzeimer's currently doesn't have any effective treatments, so I'd repeat the question: how would the fatality rate from Alzheimer's double year over year?
zeteo 6y ago
At the very least this has no way to distinguish directly between direct and indirect effects of the epidemic. The average person has put on weight, gets less exercise and is less likely to go visit a doctor. I think the distinction is important if this type of statistics are used to consider e.g. the easing of outdoor restrictions.
cycrutchfield 6y ago
It seems like you have an issue with this paper not because of the methodology but rather because the conclusion disagrees with your worldview.
kurikuri 6y ago
But this wasn’t written to confirm that COVID was underreported, nor predict future results, it was written to estimate the mortality rate of the past year and a half. Your criticism doesn’t seem to hold here. Also: The author cited the sources for their assumptions, specifically on COVID mortality classification.
benmarten 6y ago
https://www.USMortality.com https://www.USMortality.com
lamontcg 6y ago
The IHME has been wildly wrong yet highly cited for over a year now. I've still got screencaps of their models that predict around 90,000 total deaths in the USA by the end of 2020. They were lifting a sigmoid curve off of China's numbers and applying it to the United States entirely blindly. It didn't even predict the shape of the decline off the first peak correctly since it was implicitly assuming the US would take similar steps to China (martial law) and drive the R0 down to 0.4 or so.
- timr 6y ago
  There has been a ton of terrible modeling this year, and the IHME is just one example. This week the CDC published an ensemble of models trained on Covid case data up to March 27. The models completely mispredict the actual number of cases since that time: https://pbs.twimg.com/media/E0qZTvVVkAIs484?format=jpg&name=medium https://pbs.twimg.com/media/E0qZTvVVkAIs484?format=jpg&name=... This is a total failure of validation that would make any other practitioner question the whole exercise, but it didn't stop the CDC from publishing them. Nor did it stop the news media from trumpeting the results. The Washington Post and CNBC and others wrote headline articles fawning over them.
  - deleted 6y ago
    [deleted]
- kens 6y ago
  Yes, I don't understand why the IHME is still taken seriously after their awful models a year ago where they were repeatedly, confidently wrong. Link predicting deaths drop to 0 by June 21, 2020 (i.e. last year). Spoiler: they did not. https://ktla.com/news/nationworld/influential-ihme-model-projects-coronavirus-deaths-will-stop-after-june-21-but-experts-are-skeptical/ https://ktla.com/news/nationworld/influential-ihme-model-pro... The original paper https://www.medrxiv.org/content/10.1101/2020.03.27.20043752v1.full-text https://www.medrxiv.org/content/10.1101/2020.03.27.20043752v...

by timr 6y ago

grumpyautist 6y ago

Yep, and it's a suspect one at best. Personally I wouldn't trust a stastical model from someone who advertises that they own the book "how to lie with statistics" and obviously used it

seoaeu 6y ago
I trust someone who acknowledges that statistics can be used deceptively vastly more than I trust someone who doesn't.
- deleted 6y ago
  [deleted]
nathancahill 6y ago
I mean that book is basically required reading for Statistics 101. Doesn't everyone have a copy?
- deleted 6y ago
  [deleted]
  - deleted 6y ago
    [deleted]
- morelisp 6y ago
  A comment from a high school math teacher concerning that book (which yes, was commonplace to read even decades ago, even among high school AP students) has stuck with me: "It's definitely possible to lie with statistics, but wait until you see how much you can get away with without them!"
xhedley 6y ago
The book "how to lie with statistics" (1954) was written before tags like /s were invented. Referencing "how to lie with statistics" is an indicator that the author is trying to avoid common pitfalls in statistical reasoning. I own an inherited blue Pelican paperbook copy from my pharmacist grandfather. https://en.wikipedia.org/wiki/How_to_Lie_with_Statistics https://en.wikipedia.org/wiki/How_to_Lie_with_Statistics
tester756 6y ago
>Personally I wouldn't trust a stastical model from someone who advertises that they own the book "how to lie with statistics" and obviously used it why?

newacct583 6y ago

Surely you'd agree that "2020 excess deaths are going to be predominantly covid" is a very reasonable hypothesis though, right? And you agree that "2020 has an extraordinarily high excess death rate", because that part is easily measured data, right? Basically: your response seems a bit nitpicky to me, which tells me that perhaps you're starting from a perspective where you're assuming the article is wrong and looking for holes to poke. Look, nothing about this paper's conclusions is surprising at all. At all. We all know true covid deaths are higher than reported, because all such metrics underreport, for all diseases (and frankly almost all causes of death). The only question is "By how much?". So you don't like 400k. What's your counter?

timr 6y ago
> Surely you'd agree that "2020 excess deaths are going to be predominantly covid" is a very reasonable hypothesis though, right? And you agree that "2020 has an extraordinarily high excess death rate", because that part is easily measured data, right? I don't know the answer, and neither do you. But given that the top-line is asserting that there are 900k excess deaths due to Covid, the choice of assumption here is not some minor detail. There are lots of other plausible alternative hypotheses. Aassuming X, then building a model that shows X is true, is not validating a hypothesis.
- grecy 6y ago
  > There are lots of other plausible alternative hypotheses. Please list them. We have data about deaths going back many decades. It's very safe to say that if in 2010 you made a prediction about the number of expected deaths in 2011, you would have been very close to the real number. Same if you had predicted the deaths for 2019 in 2018. Suddenly, after decades and decades of being almost perfect in your predictions, you would have been wildly wrong for the predicted number of deaths in 2020. After decades of predictability, there was a whopping 20% increase in 2020 [1]. How can you possibly say it wasn't caused by Covid? [1] https://jamanetwork.com/journals/jama/fullarticle/2778361 https://jamanetwork.com/journals/jama/fullarticle/2778361
  - 6y ago

Sebb767 6y ago

> for now we assume that total COVID-19 deaths equal excess mortality. This part is especially doubtful for me, as there are factors moving it in either direction. For one, the pandemic measures also helped a lot for things like the flu, which had less cases and therefore caused less deaths. On the other hand, problems caused by lack of exercise and social interaction have definitely increased. Lastly, I'm quite sure we have a lot of missclassified deaths in either direction. These are all quite different and I see no reason to assume these factors even out.

thebruce87m 6y ago
For my country and some others I monitored, the first wave happened outside of flu season, and excess mortality was higher than COVID deaths at that point and continued to track. Second wave was more difficult since it overlapped the normal flu season. You can see graphs here showing normal flu seasons and where COVID fits in: https://www.euromomo.eu/graphs-and-maps/ https://www.euromomo.eu/graphs-and-maps/
mattmanser 6y ago
I wouldn't call it at all doubtful. The fact there's less cars on the road can only push excess mortality down, regardless of lack of exercise, so it seems sensible to attribute it all to the pandemic. Also, suicides aren't significantly up,the physical and psychological effects of lockdown don't happen immediately, while a reduction of car deaths do. e.g. UK suicides have not significantly changed in 2020: https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/quarterlysuicidedeathregistrationsinengland/2001to2019registrationsandquarter1jantomartoquarter3julytosept2020provisionaldata https://www.ons.gov.uk/peoplepopulationandcommunity/birthsde... So overall I think they're pretty safely assume that any excess deaths are coronavirus, especially as flu has all but disappeared too.
philwelch 6y ago
Some more factors might be hospital crowding early in the pandemic, people avoiding medical treatment, and increased homicide rates.
- 6y ago

teachingassist 6y ago

> for now we assume that total COVID-19 deaths equal excess mortality. This seems reasonable, although I'd suggest it's an underestimate of COVID-19 deaths. Excess deaths counts have been consistently showing negative in countries during periods of time where they have coronavirus under control. Other significant causes of death (including suicide, influenza, road traffic accidents) appear to be down. As such, I'd assume that COVID-19 deaths are likely somewhat higher than excess mortality figures. I'm prepared to accept the assumption that they're approximately the same. [Edited to note: the linked article discusses this with further detail and evidence]

pbreit 6y ago
Excess deaths could easily be attributed to non-covid factors like Alzheimer’s. Suggesting the US had 900k covid deaths is ridiculous. Only 1 of the 6 considered factors lists covid as a direct cause.
- joshuamorton 6y ago
  > Excess deaths could easily be attributed to non-covid factors like Alzheimer’s. Why would there suddenly be an 100% YoY increase in the number of Alzheimers deaths?
  - craftinator 6y ago
    Because we like to ignore facts that counter our worldviews.
  - pbreit 6y ago
    Hospitals and doctors were basically not handling any non-covid patients.
    - joshuamorton 6y ago
      Alzeimer's currently doesn't have any effective treatments, so I'd repeat the question: how would the fatality rate from Alzheimer's double year over year?

zeteo 6y ago

At the very least this has no way to distinguish directly between direct and indirect effects of the epidemic. The average person has put on weight, gets less exercise and is less likely to go visit a doctor. I think the distinction is important if this type of statistics are used to consider e.g. the easing of outdoor restrictions.

cycrutchfield 6y ago

It seems like you have an issue with this paper not because of the methodology but rather because the conclusion disagrees with your worldview.

kurikuri 6y ago

But this wasn’t written to confirm that COVID was underreported, nor predict future results, it was written to estimate the mortality rate of the past year and a half. Your criticism doesn’t seem to hold here. Also: The author cited the sources for their assumptions, specifically on COVID mortality classification.

benmarten 6y ago

https://www.USMortality.com https://www.USMortality.com

lamontcg 6y ago

The IHME has been wildly wrong yet highly cited for over a year now. I've still got screencaps of their models that predict around 90,000 total deaths in the USA by the end of 2020. They were lifting a sigmoid curve off of China's numbers and applying it to the United States entirely blindly. It didn't even predict the shape of the decline off the first peak correctly since it was implicitly assuming the US would take similar steps to China (martial law) and drive the R0 down to 0.4 or so.

timr 6y ago
There has been a ton of terrible modeling this year, and the IHME is just one example. This week the CDC published an ensemble of models trained on Covid case data up to March 27. The models completely mispredict the actual number of cases since that time: https://pbs.twimg.com/media/E0qZTvVVkAIs484?format=jpg&name=medium https://pbs.twimg.com/media/E0qZTvVVkAIs484?format=jpg&name=... This is a total failure of validation that would make any other practitioner question the whole exercise, but it didn't stop the CDC from publishing them. Nor did it stop the news media from trumpeting the results. The Washington Post and CNBC and others wrote headline articles fawning over them.
- deleted 6y ago
  [deleted]
kens 6y ago
Yes, I don't understand why the IHME is still taken seriously after their awful models a year ago where they were repeatedly, confidently wrong. Link predicting deaths drop to 0 by June 21, 2020 (i.e. last year). Spoiler: they did not. https://ktla.com/news/nationworld/influential-ihme-model-projects-coronavirus-deaths-will-stop-after-june-21-but-experts-are-skeptical/ https://ktla.com/news/nationworld/influential-ihme-model-pro... The original paper https://www.medrxiv.org/content/10.1101/2020.03.27.20043752v1.full-text https://www.medrxiv.org/content/10.1101/2020.03.27.20043752v...