4 ms·

I know it's off topic but I'd love to learn more about how the modeling of COVID went wrong. You can pm me if you want to take the discussion elsewhere.

by alecst 6y ago

rajansaini 6y ago
Seconded!
neolog 6y ago
I would like to read about that too.
nostromo 6y ago
The models tended to overshoot the number of deaths by huge amounts. For example, the Imperial College of London estimated 40m deaths in 2020 instead of the 2m that occurred. https://www.wsws.org/en/articles/2020/03/18/covi-m18.html https://www.wsws.org/en/articles/2020/03/18/covi-m18.html https://www.nature.com/articles/d41586-020-01003-6 https://www.nature.com/articles/d41586-020-01003-6 https://www.imperial.ac.uk/news/196496/coronavirus-pandemic-could-have-caused-40/ https://www.imperial.ac.uk/news/196496/coronavirus-pandemic-... The model authors have since argued that the data was correct, but people responded to the pandemic by changing the way we live. That's OP point: that feedback cycles and corrections exist and they make modeling dynamic systems very difficult.
- mattmanser 6y ago
  But they never learn either, the ICL react studies are still getting it spectacularly wrong, 4 weeks ago they claimed cases were rising in the UK, that R was above 1. Even a cursory glance at the actual data, even the data available at the time, shows they were completely and utterly wrong.
- p1necone 6y ago
  This seems unsurprising, and the correct way to model a situation like this to me. "Lots of people could die if you keep behaving as you currently are" "Okay, lets behave differently" And then less people die. Trying to frame it as "they modelled it wrong" is nonsense. What even is the point of predictions like this if not to change behaviour - predicting outcomes based on everyone taking precautions and not telling people what might happen if they don't would be dangerous and irresponsible.
  - nostromo 6y ago
    You're missing that they modeled all of those scenarios out, "do nothing" was just one of their models. And even their best case scenarios overshot the mark -- and by a lot. This isn't to criticize modeling -- it's only to point out how hard it is to get right.
    - miscorrection 6y ago
      You claimed: > The models tended to overshoot the number of deaths by huge amounts. For example, the Imperial College of London estimated 40m deaths in 2020 instead of the 2m that occurred. The very article you cited pointed out that the 40m figure was based on a "left unchecked" scenario. It was not an attempt to predict the actual number of deaths that would occur. Claiming that this is indicative of overshooting because the actual number of deaths is 2m is completely wrong.
  - TeMPOraL 6y ago
    > Trying to frame it as "they modelled it wrong" is nonsense. It's not that. It's that when the system you model responds to the existence of your model, it becomes anti-inductive. It's no longer like weather, but is now like the stock market[0]. Your model suddenly can't predict the system anymore[2], it can at best determine its operating envelope by estimating the degree to which the system can react to the existence of the model. -- [0] - I use the term anti-inductive per LessWrong nomenclature[1], but I've also been reading "Sapiens" by Yuval Noah Harari, and there he uses terms "first order chaotic" for systems like weather, and "second order chaotic" for systems like the stock market. [1] - Introduced in https://www.lesswrong.com/posts/h24JGbmweNpWZfBkM/markets-are-anti-inductive https://www.lesswrong.com/posts/h24JGbmweNpWZfBkM/markets-ar.... [2] - I think it becomes uncomputable in Turing sense, but I'm not smart enough to reduce this to the Halting Problem.
    - miscorrection 6y ago
      The issue is that the example nostromo gave (40m) was not intended to be predictive of what would actually happen. It was based on a worst case / left unchecked scenario (useful for establishing an upper bound), and therefore irrelevant w/r/t the system responding to the model.
- Blikkentrekker 6y ago
  > The model authors have since argued that the data was correct, but people responded to the pandemic by changing the way we live. That's OP point: that feedback cycles and corrections exist and they make modeling dynamic systems very difficult. Were they actually so naïve that their model did not allow for the possibility that human beings change their behavior in fear of death by pandemic?
- BurningFrog 6y ago
  Assuming present trends continue, even when things are clearly heading for terrible outcomes, is the basis for most doomsday predictions.
cossatot 6y ago
Quanta had a decent write-up of challenges faced by some researchers about a month ago: https://www.quantamagazine.org/the-hard-lessons-of-modeling-the-coronavirus-pandemic-20210128/ https://www.quantamagazine.org/the-hard-lessons-of-modeling-... Of course there are many ways that things went wrong, and not everyone made the same mistakes.

I know it's off topic but I'd love to learn more about how the modeling of COVID went wrong. You can pm me if you want to take the discussion elsewhere.

by alecst 6y ago

I know it's off topic but I'd love to learn more about how the modeling of COVID went wrong. You can pm me if you want to take the discussion elsewhere.

rajansaini 6y ago

Seconded!

neolog 6y ago

I would like to read about that too.

nostromo 6y ago

The models tended to overshoot the number of deaths by huge amounts. For example, the Imperial College of London estimated 40m deaths in 2020 instead of the 2m that occurred. https://www.wsws.org/en/articles/2020/03/18/covi-m18.html https://www.wsws.org/en/articles/2020/03/18/covi-m18.html https://www.nature.com/articles/d41586-020-01003-6 https://www.nature.com/articles/d41586-020-01003-6 https://www.imperial.ac.uk/news/196496/coronavirus-pandemic-could-have-caused-40/ https://www.imperial.ac.uk/news/196496/coronavirus-pandemic-... The model authors have since argued that the data was correct, but people responded to the pandemic by changing the way we live. That's OP point: that feedback cycles and corrections exist and they make modeling dynamic systems very difficult.

mattmanser 6y ago
But they never learn either, the ICL react studies are still getting it spectacularly wrong, 4 weeks ago they claimed cases were rising in the UK, that R was above 1. Even a cursory glance at the actual data, even the data available at the time, shows they were completely and utterly wrong.
p1necone 6y ago
This seems unsurprising, and the correct way to model a situation like this to me. "Lots of people could die if you keep behaving as you currently are" "Okay, lets behave differently" And then less people die. Trying to frame it as "they modelled it wrong" is nonsense. What even is the point of predictions like this if not to change behaviour - predicting outcomes based on everyone taking precautions and not telling people what might happen if they don't would be dangerous and irresponsible.
- nostromo 6y ago
  You're missing that they modeled all of those scenarios out, "do nothing" was just one of their models. And even their best case scenarios overshot the mark -- and by a lot. This isn't to criticize modeling -- it's only to point out how hard it is to get right.
  - miscorrection 6y ago
    You claimed: > The models tended to overshoot the number of deaths by huge amounts. For example, the Imperial College of London estimated 40m deaths in 2020 instead of the 2m that occurred. The very article you cited pointed out that the 40m figure was based on a "left unchecked" scenario. It was not an attempt to predict the actual number of deaths that would occur. Claiming that this is indicative of overshooting because the actual number of deaths is 2m is completely wrong.
- TeMPOraL 6y ago
  > Trying to frame it as "they modelled it wrong" is nonsense. It's not that. It's that when the system you model responds to the existence of your model, it becomes anti-inductive. It's no longer like weather, but is now like the stock market[0]. Your model suddenly can't predict the system anymore[2], it can at best determine its operating envelope by estimating the degree to which the system can react to the existence of the model. -- [0] - I use the term anti-inductive per LessWrong nomenclature[1], but I've also been reading "Sapiens" by Yuval Noah Harari, and there he uses terms "first order chaotic" for systems like weather, and "second order chaotic" for systems like the stock market. [1] - Introduced in https://www.lesswrong.com/posts/h24JGbmweNpWZfBkM/markets-are-anti-inductive https://www.lesswrong.com/posts/h24JGbmweNpWZfBkM/markets-ar.... [2] - I think it becomes uncomputable in Turing sense, but I'm not smart enough to reduce this to the Halting Problem.
  - miscorrection 6y ago
    The issue is that the example nostromo gave (40m) was not intended to be predictive of what would actually happen. It was based on a worst case / left unchecked scenario (useful for establishing an upper bound), and therefore irrelevant w/r/t the system responding to the model.
Blikkentrekker 6y ago
> The model authors have since argued that the data was correct, but people responded to the pandemic by changing the way we live. That's OP point: that feedback cycles and corrections exist and they make modeling dynamic systems very difficult. Were they actually so naïve that their model did not allow for the possibility that human beings change their behavior in fear of death by pandemic?
BurningFrog 6y ago
Assuming present trends continue, even when things are clearly heading for terrible outcomes, is the basis for most doomsday predictions.

cossatot 6y ago

Quanta had a decent write-up of challenges faced by some researchers about a month ago: https://www.quantamagazine.org/the-hard-lessons-of-modeling-the-coronavirus-pandemic-20210128/ https://www.quantamagazine.org/the-hard-lessons-of-modeling-... Of course there are many ways that things went wrong, and not everyone made the same mistakes.