3 ms·

this is quite off the mark. the author sets the bar too low for himself by criticizing the most easily (and to be fair legitimately) dismissed criticisms of the

this is quite off the mark. the author sets the bar too low for himself by criticizing the most easily (and to be fair legitimately) dismissed criticisms of the Imperial College model. here's a better laid-out critique that the OP doesn't speak to: The Imperial College modelers released the source code a couple of days ago to the model that shut down the world economy. It's not the original model code but was rather original source code turned over to volunteer programmers who re-wrote it so that is more readable. I have done some model review of financial models in the past but without the source code I would not be able to do a full review of the Imperial College model. Now that we have the source code (sort of), I can. Any such model ought to have been independently reviewed before it is ever used for real policy decisions. Policy analysis is awash in models but no one ever really checks them. Going forward, health policy makers should ask for and disclose independent validation of any model before using its results to make recommendations of any consequence. Normally, model reviews are long technical documents but there would also be a summary section. Here's what I think a summary should have looked like. Overall conclusion: this model cannot be relied on to guide coronavirus policy. Even if the documentation, coding, and testing problems were fixed, the model logic is fatally flawed, which is evidenced by its poor forecasting performance. https://www.facebook.com/scarlett.strong.1/posts/252437219500976 https://www.facebook.com/scarlett.strong.1/posts/25243721950...

wrsh07 6y ago
It's worth noting that in a pandemic situation, waiting before taking action is "making a decision" Shutting down the economy is costly, yes, but allowing a virus to continue its exponential growth with all of the unknowns that entails is much more costly in expectation (we can only speak in expectation given the unknowns) Whether or not this model should have been the basis for such a decision I don't know, but it's important to keep in mind that taking proactive steps in a pandemic is conservative
- SpicyLemonZest 6y ago
  The government wasn't waiting before taking action. The Imperial report was impactful because it said that the government's actions were insufficient, that if they continued on with only moderate social distancing people would die in the streets for lack of hospital beds.
Majromax 6y ago
> Any such model ought to have been independently reviewed before it is ever used for real policy decisions. Policy analysis is awash in models but no one ever really checks them. Going forward, health policy makers should ask for and disclose independent validation of any model before using its results to make recommendations of any consequence. That's ignoring the time-limited nature of a virus response. "No decision" is itself a decision. Delay has a real cost, and using a potentially imperfect model is simply using the best information at hand. I think everyone would agree that the ideal is to have well-documented, thoroughly-tested, easy-to-use, and bulletproof models to inform public response to every emergency. However, those models can't be built instantly, and that kind of bulletproofing is not directly relevant to the day-to-day research work aided by such models. Adding research capacity to ensure that governments have a stable of well-researched, thoroughly-vetted models for emergencies would be a great thing, but it would also be quite expensive. Keeping any single model up to spec might be the job of 1FTE (so an additional ~$150k/grant/yr -- large for a research budget but small for a government), but that would have to be multiplied by every area where the government might possibly want research-informed decisionmaking on short notice. > Even if the documentation, coding, and testing problems were fixed, the model logic is fatally flawed, which is evidenced by its poor forecasting performance. That sounds like a great scientific criticism! Model validation is the cornerstone of research on computer models of things, and finding a poor forecast opens the door to many great research questions. But without that further research, "poor performance" sounds a (loud) note of caution, but isn't necessarily fatal. The leading-order problem is to understand why the model performed poorly: was it improperly calibrated with information known at the time (such as if the virus behaves differently than assumed)? Was there some out-of-sample feature of the forecast that the model would not expect to do well on (e.g., low death rates in an open society because everything was shut down for severe weather anyway?) Is an overall trend correct but the timing in error? "Flawed," I think, can be easily shown, and this should probably be expected in a research model. "Fatally flawed," however, is a stronger claim that must pass a greater burden of proof.
- enitihas 6y ago
  The quickest decision is random number generation. Should people go with that? Governments across the world had loads and loads of time. > Adding research capacity to ensure that governments have a stable of well-researched, thoroughly-vetted models for emergencies would be a great thing, but it would also be quite expensive. Keeping any single model up to spec might be the job of 1FTE (so an additional ~$150k/grant/yr -- large for a research budget but small for a government), but that would have to be multiplied by every area where the government might possibly want research-informed decisionmaking on short notice. If you have a once in a century crisis, and still are thinking about saving some millions, while being sure your crisis response will be above billions, your policy is already flawed and not much can be done to help you.
raverbashing 6y ago
> the model logic is fatally flawed, which is evidenced by its poor forecasting performance. Where's the proof the model is flawed? The criticism goes both ways, if on one side the model does not have documentation, it's also fair to say that if someone claims it's "fatally flawed" they need to point out in the code where the issues are.

wrsh07 6y ago

It's worth noting that in a pandemic situation, waiting before taking action is "making a decision" Shutting down the economy is costly, yes, but allowing a virus to continue its exponential growth with all of the unknowns that entails is much more costly in expectation (we can only speak in expectation given the unknowns) Whether or not this model should have been the basis for such a decision I don't know, but it's important to keep in mind that taking proactive steps in a pandemic is conservative

SpicyLemonZest 6y ago
The government wasn't waiting before taking action. The Imperial report was impactful because it said that the government's actions were insufficient, that if they continued on with only moderate social distancing people would die in the streets for lack of hospital beds.

Majromax 6y ago

> Any such model ought to have been independently reviewed before it is ever used for real policy decisions. Policy analysis is awash in models but no one ever really checks them. Going forward, health policy makers should ask for and disclose independent validation of any model before using its results to make recommendations of any consequence. That's ignoring the time-limited nature of a virus response. "No decision" is itself a decision. Delay has a real cost, and using a potentially imperfect model is simply using the best information at hand. I think everyone would agree that the ideal is to have well-documented, thoroughly-tested, easy-to-use, and bulletproof models to inform public response to every emergency. However, those models can't be built instantly, and that kind of bulletproofing is not directly relevant to the day-to-day research work aided by such models. Adding research capacity to ensure that governments have a stable of well-researched, thoroughly-vetted models for emergencies would be a great thing, but it would also be quite expensive. Keeping any single model up to spec might be the job of 1FTE (so an additional ~$150k/grant/yr -- large for a research budget but small for a government), but that would have to be multiplied by every area where the government might possibly want research-informed decisionmaking on short notice. > Even if the documentation, coding, and testing problems were fixed, the model logic is fatally flawed, which is evidenced by its poor forecasting performance. That sounds like a great scientific criticism! Model validation is the cornerstone of research on computer models of things, and finding a poor forecast opens the door to many great research questions. But without that further research, "poor performance" sounds a (loud) note of caution, but isn't necessarily fatal. The leading-order problem is to understand why the model performed poorly: was it improperly calibrated with information known at the time (such as if the virus behaves differently than assumed)? Was there some out-of-sample feature of the forecast that the model would not expect to do well on (e.g., low death rates in an open society because everything was shut down for severe weather anyway?) Is an overall trend correct but the timing in error? "Flawed," I think, can be easily shown, and this should probably be expected in a research model. "Fatally flawed," however, is a stronger claim that must pass a greater burden of proof.

enitihas 6y ago
The quickest decision is random number generation. Should people go with that? Governments across the world had loads and loads of time. > Adding research capacity to ensure that governments have a stable of well-researched, thoroughly-vetted models for emergencies would be a great thing, but it would also be quite expensive. Keeping any single model up to spec might be the job of 1FTE (so an additional ~$150k/grant/yr -- large for a research budget but small for a government), but that would have to be multiplied by every area where the government might possibly want research-informed decisionmaking on short notice. If you have a once in a century crisis, and still are thinking about saving some millions, while being sure your crisis response will be above billions, your policy is already flawed and not much can be done to help you.

raverbashing 6y ago

> the model logic is fatally flawed, which is evidenced by its poor forecasting performance. Where's the proof the model is flawed? The criticism goes both ways, if on one side the model does not have documentation, it's also fair to say that if someone claims it's "fatally flawed" they need to point out in the code where the issues are.