3 ms·
I've read a lot of public health papers in the past year, probably more than 50. Here are some of the problems that cropped up in papers that were used by gover
by mike_hearn 6y ago
I've read a lot of public health papers in the past year, probably more than 50. Here are some of the problems that cropped up in papers that were used by governments to support policy decisions. Many of those policy decisions later turned out to be incorrect, like lockdowns and mask mandates, both of which easily count as drastic and neither were textbook. In fact pre-COVID pandemic control plans by the WHO explicitly recommended against lockdowns.
By the way, be careful to separate the question of mask mandates from masks themselves. Mask mandates don't work: just look at case curves when mandates were introduced or removed and observe the lack of inflection points. If they worked people would have hundreds of examples by now of case curves which obviously inflected right after a mask mandate was changed, but no such graphs are ever cited because those inflections don't reliably happen. Texas provides a recent example (mandate removed, curve continues prior trend) but this problem was obvious from within a week of the first mandates being introduced. Look at [1] for some case graphs with mandate change dates drawn on them to see the problem.
Anyway, errors seen in public health papers:
1. Circular reasoning.
2. Invalid citations.
3. Programming errors in models.
4. Use of extremely out of date numbers.
5. Absurd or obviously invalid assumptions and results in models being ignored.
That's not a comprehensive list. Unfortunately these aren't rare problems. Virtually every public health paper I've read has had at least one of these issues, often multiple. Circular logic in particular is mind-numbingly common, to an extent I've never seen before. For example, a common "validation" technique for models is to compare them to other models and declare their outputs to be similar (e.g. [2] or [3]). It's almost unheard of to compare model outputs to actual observed data, probably because doing validation right would invalidate virtually all public health models (this problem was admitted in a 2012 paper [4]).
For a specific example of these problems see the paper by Flaxman et al from Imperial College London [5]. This paper argued that lockdowns work using a statistical model, but they actually don't work, so to get this result required a combination of:
1. Circular logic: the model took as a starting assumption that case curves could only be changed by government intervention. In other words the paper encoded its own conclusions in its assumptions. The assumption epidemics can only be affected by lockdowns has no rational basis given the long history of epidemics starting and ending naturally.
2. The paper included Sweden in its data set, which attracted attention because Sweden appears to prove that the models generating the counterfactual were wrong. It managed to conclude lockdowns worked despite this because it concluded Sweden was a freak coincidence with an only 1 in 2000 chance of existing at all; in the graph that showed the different per-country fudge factors the model was allowed to calculate, Sweden was simply hidden to obscure what had happened. The truth was discovered later by people who studied the tables of prior probabilities uploaded to GitHub.
3. The paper admitted half way through that its scenario was "illustrative only" and that "in reality" the results would be different.
There were other problems too, but none of them stopped the authors telling the press that lockdowns had "saved millions of lives" (i.e. the drastic policy pushed for by that very same research team). Nor did it stop international press agencies citing this paper in "fact checks".
After reading so many papers with really basic and blatant problems, it's hard not to conclude that open access is going to seriously damage academia's credibility. Being able to just download and read the output of academic scientists is a very new thing, and one of the few highlights of the time I've spent reading COVID research is that open access is real now: I've hardly ever hit paywalls. Only for old papers. Unfortunately open access is a double edged sword. Now we can all read what we're paying for and observe the dangerously low quality. The outcome will probably be a large expansion of the battle between "science believers" and "science skeptics". Up until now that has been mostly restricted to debates on climatology, but now I think it will widen considerably. There's just no way to read the literature and retain your confidence in academic science when so much of it is entirely un-scientific.
[1] https://rationalground.com/mask-charts/ https://rationalground.com/mask-charts/
[2] https://github.com/ptti/ptti/blob/master/README.md https://github.com/ptti/ptti/blob/master/README.md (see the paragraph starting with "Formalism-agnostic").
[3] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3001435/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3001435/ ("There is agreement in the literature that comparing the results of different models provides important
evidence of validity and increases model credibility")
[4] ibid; ("few models in healthcare could ever be validated for predictive use. This, however, does not disqualify such models from being used as aids to decision making")
[5] https://nicholaslewis.org/did-lockdowns-really-save-3-million-covid-19-deaths-as-flaxman-et-al-claim/ https://nicholaslewis.org/did-lockdowns-really-save-3-millio...
- newacct583 6y ago> This paper argued that lockdowns work using a statistical model, but they actually don't work Sorry, cite for "actually don't work"?! I don't know what you mean by "lockdown". That's not a term of art, it's something used mostly among right wing people arguing against mitigation strategies via hyperbole. Are you talking about mask requirements? Restaurant closures? Hard curfews? Obviously "lockdowns" (in this sense of "broad covid mitigation strategies") do work. Arguing otherwise is nonsensical. Just look at the data for what is happening in influenza in areas where mitigation was common. Flu cases and deaths fell through the floor. Clearly mitigation worked, just not perfectly (in large part because of poor compliance by people who believe "lockdowns don't work").
- mike_hearn 6y agoHere are 32 studies, papers and articles of varying quality that support that claim: https://www.aier.org/article/lockdowns-do-not-control-the-coronavirus-the-evidence/ https://www.aier.org/article/lockdowns-do-not-control-the-co... I don't know what you mean by "lockdown". That's not a term of art, it's something used mostly among right wing people arguing against mitigation strategies via hyperbole. The word lockdown has been widely used for the entire past year by people across the political spectrum, to refer to mandatory stay-at-home orders, business closures and so on. You know this already of course, but are determined to view this through ideological lenses. As for obviously working and being nonsensical to claim otherwise, it's the opposite: the data is extremely clear that they have no effect whatsoever and quite a lot of all-country statistical analyses have been done that show that rigorously. For example, here is one such paper: https://www.thelancet.com/journals/eclinm/article/PIIS2589-5370(20)30208-X/fulltext https://www.thelancet.com/journals/eclinm/article/PIIS2589-5... "Rapid border closures, full lockdowns, and wide-spread testing were not associated with COVID-19 mortality per million people" But again, to see this, you don't really need sophisticated analysis. You can just look at the case curves for different regions that did or did not use these tactics. Re: influenza. We're talking about COVID here, so influenza is irrelevant, controlling the flu wasn't the goal of lockdowns. However, there is some evidence that suggests you can't be infected with more than one respiratory virus at once, so it's possible SARS-CoV-2 simply kicked influenza out. But given the extremely lax symptom classification of COVID it's also possible influenza cases have simply been reclassified. Regardless of the explanation it's not actually relevant: neither lockdowns nor mask mandates had any impact on COVID and there is abundant evidence to this effect. The real question we should be asking is why not, and there unfortunately there are quite a few plausible explanations but nothing conclusive. And the people who are paid to figure that out (epidemiologists) all seem to be in denial and still pretending their models were never invalidated, so they're not much help.