4 ms·

I'm finding myself in disagreement with rule #6. Using a model effectively is about a lot more than just the domain knowledge. I'd value analysis from a mathe

by YZF 7y ago

I'm finding myself in disagreement with rule #6. Using a model effectively is about a lot more than just the domain knowledge. I'd value analysis from a mathematician/statistician more highly than from an infectious disease physician. There's the stuff that informs models, i.e. the observations, the experimentation etc. and then there's the science of modelling itself which isn't really in the same domain.

martingoodson 7y ago
I agree with this - but I do think it should be clear that the model is from outside the mainstream. Not to dismiss it but to clarify its status. Check out the New Yorker piece I link to in the article - it's quite shocking the misinformation that's out there.
- garmaine 7y ago
  Well, we could have benefited greatly from the mainstream media and politicians taking the outside predictions seriously at the beginning of this crisis. Instead we had to wait a month for the Imperial College London to say the same exact thing before certain leaders got their heads out of the sand. Likewise now with hydroxychloroquine--if you listen to the epidemiologists all you'd hear is how it's an UNPROVEN drug. What we need instead is coverage of sample sizes, p values, bayesian predictions of effectiveness (in the absence of controlled studies) and serious modeling of the number of ICU beds and ventilators required with and without various levels of treatment, from emergency care to prophylactic use. The epidemiologists have their head in the sand and think we can just wait 6 months for a proper set of randomized trials. It's the less attached data modelers you need to turn to get predictions that are useful for effective policy choices.
  - martingoodson 7y ago
    That's not really a fair representation. (Harvard epidemiologist) Marc Lipsitch raised the alarm back in February: "it's likely we'll see a global pandemic" of coronavirus, with 40 to 70 percent of the world's population likely to be infected this year." https://thehill.com/changing-america/well-being/prevention-cures/482794-officials-say-the-cdc-is-preparing-for https://thehill.com/changing-america/well-being/prevention-c...
- thu2111 7y ago
  I do think it should be clear that the model is from outside the mainstream Which model? Your article mostly dwells on the Oxford model in the FT which is as mainstream as it gets. The Daily Mail article is about a prediction by an Imperial academic. There doesn't appear to be any "mainstream" in epidemiology. This kind of false-consensus argument is one I've seen before and frankly it undermines people's respect in both academic and journalism. How exactly is the "mainstream" established, in your mind? And don't talk about vague, easily manipulated non-metrics like reputation. Mainstream models should come from a single thing: success. Your insistence on rigid academic segmentation is the very thing that makes academia brittle and riven with non-replicable papers. You know which kind of modellers I respect the most? Actuaries. They have skin in the game. They don't try to predict pandemics or sell insurance products against them, implying they think pandemics can't be modelled. So far based on what I've read about the history of epidemiology, I can't disagree with them.
oivey 7y ago
Don’t discount domain experts. Models are meant to predict the real world. In order for them to be accurate, the model itself needs to capture how the real world works, and the math underlying the model has to be correct. Domain experts are the most likely to have experience in both areas. Drawing an example, a physicist models the universe and knows the math and model behind electromagnetism. A mathematician probably knows the math but maybe not the model.
- skat20phys 7y ago
  I'm at the point where it shouldn't matter who did it, what should matter is its integrity. I'd prefer everything was anonymously posted at some level so author background didn't go into consideration regarding how it was received. I agree with you completely, but the flip side of the coin is that experts can be blinded by assumptions that the field has. Sometimes outsiders aren't aware of these basic assumptions and so are less biased. There's also the simple issue that sometimes expertise comes from places you least expect for reasons you might not anticipate. For me there's as many problems in this pandemic related to appeals to authority (at least in the US) -- problems with testing related to FDA regulation and the CDC, problems with lack of healthcare providers due to long-term rent-seeking monopolies in licensing and practice scope, problems related to academic fraudulence and incentives (see: Didier Raoult) -- that I think it's dangerous to raise appeal to authority as anything but a bias. For me there's multiple levels of problems to this, the first of which is the conspiracy and anti-science culture surrounding the pandemic. Above that is an appeal to medical and scientific expertise and authority that has sometimes been helpful but sometimes harmful. Above that still is an appeal to rigorous thinking and risk management, which transcends expertise boundaries.
noobermin 7y ago
Domain knowledge is very, very important in modeling because it helps you figure out what to put in the models. Math is important but certain odds and ends that are of great interest to mathematicians are not helpful for the usual simulation, and the simulations that are most helpful in informing policy.

by YZF 7y ago

martingoodson 7y ago

I agree with this - but I do think it should be clear that the model is from outside the mainstream. Not to dismiss it but to clarify its status. Check out the New Yorker piece I link to in the article - it's quite shocking the misinformation that's out there.

garmaine 7y ago
Well, we could have benefited greatly from the mainstream media and politicians taking the outside predictions seriously at the beginning of this crisis. Instead we had to wait a month for the Imperial College London to say the same exact thing before certain leaders got their heads out of the sand. Likewise now with hydroxychloroquine--if you listen to the epidemiologists all you'd hear is how it's an UNPROVEN drug. What we need instead is coverage of sample sizes, p values, bayesian predictions of effectiveness (in the absence of controlled studies) and serious modeling of the number of ICU beds and ventilators required with and without various levels of treatment, from emergency care to prophylactic use. The epidemiologists have their head in the sand and think we can just wait 6 months for a proper set of randomized trials. It's the less attached data modelers you need to turn to get predictions that are useful for effective policy choices.
- martingoodson 7y ago
  That's not really a fair representation. (Harvard epidemiologist) Marc Lipsitch raised the alarm back in February: "it's likely we'll see a global pandemic" of coronavirus, with 40 to 70 percent of the world's population likely to be infected this year." https://thehill.com/changing-america/well-being/prevention-cures/482794-officials-say-the-cdc-is-preparing-for https://thehill.com/changing-america/well-being/prevention-c...
thu2111 7y ago
I do think it should be clear that the model is from outside the mainstream Which model? Your article mostly dwells on the Oxford model in the FT which is as mainstream as it gets. The Daily Mail article is about a prediction by an Imperial academic. There doesn't appear to be any "mainstream" in epidemiology. This kind of false-consensus argument is one I've seen before and frankly it undermines people's respect in both academic and journalism. How exactly is the "mainstream" established, in your mind? And don't talk about vague, easily manipulated non-metrics like reputation. Mainstream models should come from a single thing: success. Your insistence on rigid academic segmentation is the very thing that makes academia brittle and riven with non-replicable papers. You know which kind of modellers I respect the most? Actuaries. They have skin in the game. They don't try to predict pandemics or sell insurance products against them, implying they think pandemics can't be modelled. So far based on what I've read about the history of epidemiology, I can't disagree with them.

oivey 7y ago

Don’t discount domain experts. Models are meant to predict the real world. In order for them to be accurate, the model itself needs to capture how the real world works, and the math underlying the model has to be correct. Domain experts are the most likely to have experience in both areas. Drawing an example, a physicist models the universe and knows the math and model behind electromagnetism. A mathematician probably knows the math but maybe not the model.

skat20phys 7y ago
I'm at the point where it shouldn't matter who did it, what should matter is its integrity. I'd prefer everything was anonymously posted at some level so author background didn't go into consideration regarding how it was received. I agree with you completely, but the flip side of the coin is that experts can be blinded by assumptions that the field has. Sometimes outsiders aren't aware of these basic assumptions and so are less biased. There's also the simple issue that sometimes expertise comes from places you least expect for reasons you might not anticipate. For me there's as many problems in this pandemic related to appeals to authority (at least in the US) -- problems with testing related to FDA regulation and the CDC, problems with lack of healthcare providers due to long-term rent-seeking monopolies in licensing and practice scope, problems related to academic fraudulence and incentives (see: Didier Raoult) -- that I think it's dangerous to raise appeal to authority as anything but a bias. For me there's multiple levels of problems to this, the first of which is the conspiracy and anti-science culture surrounding the pandemic. Above that is an appeal to medical and scientific expertise and authority that has sometimes been helpful but sometimes harmful. Above that still is an appeal to rigorous thinking and risk management, which transcends expertise boundaries.

noobermin 7y ago

Domain knowledge is very, very important in modeling because it helps you figure out what to put in the models. Math is important but certain odds and ends that are of great interest to mathematicians are not helpful for the usual simulation, and the simulations that are most helpful in informing policy.