4 ms·

Author here: happy to take comments or criticism

dwohnitmok 7y ago
This is a hard problem, mainly because of the problem of motivated reasoning as mentioned by another comment. You're requiring cooperation from multiple different parties here (scientists, journalists, policy makers, readers, etc.) and any of these parties can warp the results in any number of ways regardless of the cooperation of other parties. Climate science still hasn't solved this problem despite trying to implement what you're talking about. It's an age-old problem, if a priori you're looking for something hard enough you're bound to find it.
- martingoodson 7y ago
  I think, at the least, if journalists get some quotes from other scientists before publishing a piece on a new model it would be a major win.
  - dwohnitmok 7y ago
    That's fair. I mean asking folks to be more circumspect in general is probably a good thing. I prefer though to look at problems through the lens of incentive structures (keeping in mind humans generally heavily time discount incentives and what incentivizes people is not always obvious! Death isn't always much of a disincentive beyond a rather short time horizon). And here I'm having a hard time seeing easy ways to tweak the incentive structure.
  - thaumasiotes 7y ago
    Often they do, but that doesn't affect the piece they write. It's not at all difficult to find people complaining "I was interviewed for this article, most of what I said was left out, and to the extent I am quoted, it's to give the impression that my beliefs are the exact opposite of what I explained to the journalist at length".
  - excalibur 7y ago
    Asking more of journalists at this juncture may be a tall order. The bar for them has been trending downward, due largely to a lack of funding and an abundance of competition, many of whom have little regard for journalistic standards.
  - guscost 7y ago
    I don’t think there’s value in doing this, unless the second opinion disagrees with the original model. It’s not hard to find a second “expert” to agree with just about anything, if you look hard enough. The number of people who agree with something tells you, as a rational person, practically nothing - it reminds me of the absurd compilation argument “One Hundred Authors Against Einstein”: https://archive.org/details/HundertAutorenGegenEinstein/mode/2up https://archive.org/details/HundertAutorenGegenEinstein/mode... Einstein’s famous reply: “If I were wrong, then one would have been enough!”
    - noobermin 7y ago
      An article with just opposing positions would just confuse the reader. It's also like how the media has climate skeptics debate climate scientists one-on-one as if there is an actual debate to be had, magnifying the level of climate skepticism in the field.
      - guscost 7y ago
        I think you missed the point of the story.
  - TwoBit 7y ago
    Journalists and their publications don't give a crap about scientific truth. They care about maximizing clicks and traffic. If a group of scientists give them the truth and another group gives them a more sensational story, they'll print the latter. Scientists will never win this battle.
  - thu2111 7y ago
    That would just mean nobody would publish articles about models, as there are no models that don't have some scientists disagreeing with them. Which is fine. It'd be far better if people weren't trying to model COVID-19 at all. It's clearly far beyond our societies capabilities today. We can't even get useful data let alone feed it into a simulation of society.
YZF 7y ago
I'm finding myself in disagreement with rule #6. Using a model effectively is about a lot more than just the domain knowledge. I'd value analysis from a mathematician/statistician more highly than from an infectious disease physician. There's the stuff that informs models, i.e. the observations, the experimentation etc. and then there's the science of modelling itself which isn't really in the same domain.
- martingoodson 7y ago
  I agree with this - but I do think it should be clear that the model is from outside the mainstream. Not to dismiss it but to clarify its status. Check out the New Yorker piece I link to in the article - it's quite shocking the misinformation that's out there.
  - garmaine 7y ago
    Well, we could have benefited greatly from the mainstream media and politicians taking the outside predictions seriously at the beginning of this crisis. Instead we had to wait a month for the Imperial College London to say the same exact thing before certain leaders got their heads out of the sand. Likewise now with hydroxychloroquine--if you listen to the epidemiologists all you'd hear is how it's an UNPROVEN drug. What we need instead is coverage of sample sizes, p values, bayesian predictions of effectiveness (in the absence of controlled studies) and serious modeling of the number of ICU beds and ventilators required with and without various levels of treatment, from emergency care to prophylactic use. The epidemiologists have their head in the sand and think we can just wait 6 months for a proper set of randomized trials. It's the less attached data modelers you need to turn to get predictions that are useful for effective policy choices.
    - martingoodson 7y ago
      That's not really a fair representation. (Harvard epidemiologist) Marc Lipsitch raised the alarm back in February: "it's likely we'll see a global pandemic" of coronavirus, with 40 to 70 percent of the world's population likely to be infected this year." https://thehill.com/changing-america/well-being/prevention-cures/482794-officials-say-the-cdc-is-preparing-for https://thehill.com/changing-america/well-being/prevention-c...
guscost 7y ago
If any of the scenarios from the famous Imperial College model turn out to have been based on just-as-bad assumptions, would you be willing to write a follow up about that?
- martingoodson 7y ago
  I'm not taking a position on the Imperial College model. I'm explicitly advocating that all models should have their assumptions examined. And that policy makers should use a range of model and not depend on just one.
  - gadders 7y ago
    Why do you think the Oxford model compares poorly to the Imperial one? Both are created by eminent people in their field. Neither has been peer-reviewed.
  - guscost 7y ago
    You will have nothing to add if the “2.2 million deaths in the US” scenario, which was blasted across every newspaper front page a few weeks ago, turns out to have been impossible all along? If that scenario was “completely wrong” too, it seems like it would serve as a perfect example of the consequences of this kind of (still hypothetical) misinformation.
    - not2b 7y ago
      We can't rerun the experiment with a control version of the US in which nothing was shut down and we continued to have crowded sports events and night life. So it won't be possible to determine that the 2.2 million deaths scenario is impossible, especially if it's interpreted as 2.2 million extra deaths from either COVID-19 or other causes that could have been treatable by a medical system that wasn't completely overwhelmed.
      - guscost 7y ago
        I’m extremely pissed off at these folks who built an unfalsifiable model, panicked the whole world, and will now pretend that it is “science” as they ignore and downplay the death and destruction that follows in the wake of their foolishness. This will not stand.
garmaine 7y ago
The final point seems like a bunk attack. A lot of epidemiologists were upset at Silicon Valley data scientists and hedge fund quants who were putting together prediction models that disputed what the government was saying. Well, so far these models have been more accurate than the official ones. As we should expect, because data scientists and quants are EXACTLY the people with the set of skills necessary to make such predictive models. At best, it's an ancillary skill for epidemiologist, and we've seen many cases in this pandemic where they have wielded these tools incorrectly.
etangent 7y ago
Your entire post seems to be about #6. And therefore it's entirely wrong. Given that "people with background in infectious diseases" have largely failed, as a group, to warn us in January-February about this pandemic (see this Twitter thread for a slew of concrete examples https://twitter.com/RokoMijicUK/status/1246509433145917443 https://twitter.com/RokoMijicUK/status/1246509433145917443), my conclusion is that unless someone has a background in hard quantitative field (regardless of what that field is), that person should not be let anywhere near quantitative models.
svat 7y ago
This is a great start on how scientists (and journalists) should communicate to the public! Thanks for writing this, and the world would be a better place if everyone remembered to follow these principles. I think in fact it would be better to go even further: not speak from a position of superiority (even if one knows more), but try to acknowledge the audience and persuade effectively. Here's a recent article on the topic: https://undark.org/2020/03/19/coronavirus-myths/ https://undark.org/2020/03/19/coronavirus-myths/ and here's one of my favourites (in a different field of science) from three years ago: https://deansforimpact.org/why-mythbusting-fails-a-guide-to-influencing-education-with-science/ https://deansforimpact.org/why-mythbusting-fails-a-guide-to-...
tomcatfish 7y ago
I very much enjoyed this article you linked with the Times reporter and the law professor https://www.newyorker.com/news/q-and-a/the-contrarian-coronavirus-theory-that-informed-the-trump-administration https://www.newyorker.com/news/q-and-a/the-contrarian-corona...
empath75 7y ago
I think starting with the global warming example was a poor choice because there are a lot of vested interests who would find something to attack and distort even in a flawless paper.

This is a hard problem, mainly because of the problem of motivated reasoning as mentioned by another comment. You're requiring cooperation from multiple different parties here (scientists, journalists, policy makers, readers, etc.) and any of these parties can warp the results in any number of ways regardless of the cooperation of other parties. Climate science still hasn't solved this problem despite trying to implement what you're talking about. It's an age-old problem, if a priori you're looking for something hard enough you're bound to find it.

martingoodson 7y ago
I think, at the least, if journalists get some quotes from other scientists before publishing a piece on a new model it would be a major win.
- dwohnitmok 7y ago
  That's fair. I mean asking folks to be more circumspect in general is probably a good thing. I prefer though to look at problems through the lens of incentive structures (keeping in mind humans generally heavily time discount incentives and what incentivizes people is not always obvious! Death isn't always much of a disincentive beyond a rather short time horizon). And here I'm having a hard time seeing easy ways to tweak the incentive structure.
- thaumasiotes 7y ago
  Often they do, but that doesn't affect the piece they write. It's not at all difficult to find people complaining "I was interviewed for this article, most of what I said was left out, and to the extent I am quoted, it's to give the impression that my beliefs are the exact opposite of what I explained to the journalist at length".
- excalibur 7y ago
  Asking more of journalists at this juncture may be a tall order. The bar for them has been trending downward, due largely to a lack of funding and an abundance of competition, many of whom have little regard for journalistic standards.
- guscost 7y ago
  I don’t think there’s value in doing this, unless the second opinion disagrees with the original model. It’s not hard to find a second “expert” to agree with just about anything, if you look hard enough. The number of people who agree with something tells you, as a rational person, practically nothing - it reminds me of the absurd compilation argument “One Hundred Authors Against Einstein”: https://archive.org/details/HundertAutorenGegenEinstein/mode/2up https://archive.org/details/HundertAutorenGegenEinstein/mode... Einstein’s famous reply: “If I were wrong, then one would have been enough!”
  - noobermin 7y ago
    An article with just opposing positions would just confuse the reader. It's also like how the media has climate skeptics debate climate scientists one-on-one as if there is an actual debate to be had, magnifying the level of climate skepticism in the field.
    - guscost 7y ago
      I think you missed the point of the story.
- TwoBit 7y ago
  Journalists and their publications don't give a crap about scientific truth. They care about maximizing clicks and traffic. If a group of scientists give them the truth and another group gives them a more sensational story, they'll print the latter. Scientists will never win this battle.
- thu2111 7y ago
  That would just mean nobody would publish articles about models, as there are no models that don't have some scientists disagreeing with them. Which is fine. It'd be far better if people weren't trying to model COVID-19 at all. It's clearly far beyond our societies capabilities today. We can't even get useful data let alone feed it into a simulation of society.

YZF 7y ago

I'm finding myself in disagreement with rule #6. Using a model effectively is about a lot more than just the domain knowledge. I'd value analysis from a mathematician/statistician more highly than from an infectious disease physician. There's the stuff that informs models, i.e. the observations, the experimentation etc. and then there's the science of modelling itself which isn't really in the same domain.

martingoodson 7y ago
I agree with this - but I do think it should be clear that the model is from outside the mainstream. Not to dismiss it but to clarify its status. Check out the New Yorker piece I link to in the article - it's quite shocking the misinformation that's out there.
- garmaine 7y ago
  Well, we could have benefited greatly from the mainstream media and politicians taking the outside predictions seriously at the beginning of this crisis. Instead we had to wait a month for the Imperial College London to say the same exact thing before certain leaders got their heads out of the sand. Likewise now with hydroxychloroquine--if you listen to the epidemiologists all you'd hear is how it's an UNPROVEN drug. What we need instead is coverage of sample sizes, p values, bayesian predictions of effectiveness (in the absence of controlled studies) and serious modeling of the number of ICU beds and ventilators required with and without various levels of treatment, from emergency care to prophylactic use. The epidemiologists have their head in the sand and think we can just wait 6 months for a proper set of randomized trials. It's the less attached data modelers you need to turn to get predictions that are useful for effective policy choices.
  - martingoodson 7y ago
    That's not really a fair representation. (Harvard epidemiologist) Marc Lipsitch raised the alarm back in February: "it's likely we'll see a global pandemic" of coronavirus, with 40 to 70 percent of the world's population likely to be infected this year." https://thehill.com/changing-america/well-being/prevention-cures/482794-officials-say-the-cdc-is-preparing-for https://thehill.com/changing-america/well-being/prevention-c...

guscost 7y ago

If any of the scenarios from the famous Imperial College model turn out to have been based on just-as-bad assumptions, would you be willing to write a follow up about that?

martingoodson 7y ago
I'm not taking a position on the Imperial College model. I'm explicitly advocating that all models should have their assumptions examined. And that policy makers should use a range of model and not depend on just one.
- gadders 7y ago
  Why do you think the Oxford model compares poorly to the Imperial one? Both are created by eminent people in their field. Neither has been peer-reviewed.
- guscost 7y ago
  You will have nothing to add if the “2.2 million deaths in the US” scenario, which was blasted across every newspaper front page a few weeks ago, turns out to have been impossible all along? If that scenario was “completely wrong” too, it seems like it would serve as a perfect example of the consequences of this kind of (still hypothetical) misinformation.
  - not2b 7y ago
    We can't rerun the experiment with a control version of the US in which nothing was shut down and we continued to have crowded sports events and night life. So it won't be possible to determine that the 2.2 million deaths scenario is impossible, especially if it's interpreted as 2.2 million extra deaths from either COVID-19 or other causes that could have been treatable by a medical system that wasn't completely overwhelmed.
    - guscost 7y ago
      I’m extremely pissed off at these folks who built an unfalsifiable model, panicked the whole world, and will now pretend that it is “science” as they ignore and downplay the death and destruction that follows in the wake of their foolishness. This will not stand.

garmaine 7y ago

The final point seems like a bunk attack. A lot of epidemiologists were upset at Silicon Valley data scientists and hedge fund quants who were putting together prediction models that disputed what the government was saying. Well, so far these models have been more accurate than the official ones. As we should expect, because data scientists and quants are EXACTLY the people with the set of skills necessary to make such predictive models. At best, it's an ancillary skill for epidemiologist, and we've seen many cases in this pandemic where they have wielded these tools incorrectly.

etangent 7y ago

Your entire post seems to be about #6. And therefore it's entirely wrong. Given that "people with background in infectious diseases" have largely failed, as a group, to warn us in January-February about this pandemic (see this Twitter thread for a slew of concrete examples https://twitter.com/RokoMijicUK/status/1246509433145917443 https://twitter.com/RokoMijicUK/status/1246509433145917443), my conclusion is that unless someone has a background in hard quantitative field (regardless of what that field is), that person should not be let anywhere near quantitative models.

svat 7y ago

This is a great start on how scientists (and journalists) should communicate to the public! Thanks for writing this, and the world would be a better place if everyone remembered to follow these principles. I think in fact it would be better to go even further: not speak from a position of superiority (even if one knows more), but try to acknowledge the audience and persuade effectively. Here's a recent article on the topic: https://undark.org/2020/03/19/coronavirus-myths/ https://undark.org/2020/03/19/coronavirus-myths/ and here's one of my favourites (in a different field of science) from three years ago: https://deansforimpact.org/why-mythbusting-fails-a-guide-to-influencing-education-with-science/ https://deansforimpact.org/why-mythbusting-fails-a-guide-to-...

tomcatfish 7y ago

I very much enjoyed this article you linked with the Times reporter and the law professor https://www.newyorker.com/news/q-and-a/the-contrarian-coronavirus-theory-that-informed-the-trump-administration https://www.newyorker.com/news/q-and-a/the-contrarian-corona...

empath75 7y ago

I think starting with the global warming example was a poor choice because there are a lot of vested interests who would find something to attack and distort even in a flawless paper.