9 ms·

Re running the authors code with their data will likely just repeat any methodological issues they had (either accidentally or fraudulently). In medical studie

Re running the authors code with their data will likely just repeat any methodological issues they had (either accidentally or fraudulently). In medical studies, one level is to reevaluate the data from the same set of patients to see if any bias or errors wasn't included, perhaps with a different or newer statistical idealogy. The best is for the study to be repeated with a fresh set of patient data to see if the underlying conclusions were valid.

mike_hearn 5y ago
"Re running the authors code with their data will likely just repeat any methodological issues they had" Yes, but that's useful too. COVID-19 lockdowns largely kicked off in the west due to an epidemiological model from Imperial College London, written over a period of many years by the now notorious Professor Neil Ferguson. When his team uploaded a pre-print of his paper and started sending it to government ministers, the code for his model wasn't available. They spent months fighting FOIA requests by claiming they were about to release it, but just had to tidy things up a bit first. When the code was finally uploaded to GitHub the world discovered the reason for delay: the model was a 15,000 lines of C trash fire of race conditions, floating point accumulation errors and memory corruptions in which basically every variable was in global scope and with a single letter name. It was a textbook case of how not to write software. In fact it was a textbook case of why variable names matter because one of the memory corruptions was caused by the author apparently losing track of what a variable named 'k' was meant to contain at that point in the code. Not surprisingly, the model didn't work. Although it had command line flags to set the PRNG seeds these flags were useless: the model generated totally different output every time you ran it, even with fixed seeds. In fact hospital bed demand prediction changed from run to run by more than the size of the entire NHS Nightingale crash hospital building programme, purely due to bugs. And as we now know the model was entirely wrong and the results were catastrophic. Lockdowns had no impact on mortality. There are many people who looked at the data and saw this but here's just the latest meta-analysis of published studies showing that to be true [1]. They destroyed the NHS which now has a cancer treatment backlog and pool of 'missing' patients so large that it cannot possibly catch up, meaning people will die waiting for treatment from the system they supported with their taxes for their entire lives. They destroyed the tax base, leaving the government with an unpayable debt that can be eliminated only via inflation meaning they will have soon destroyed people's savings too. It's just a catastrophe of incompetence and hubris. The incorrectness of the model wasn't due only to programming bugs. The underlying biological assumptions were roughly GCSE level or a bit lower (GCSE is the exams you take at 15/16 in the UK), and it's quite evident that high school germ theory is woefully incomplete. In particular it has nothing to say on the topic of aerosol vs droplet transmission, which appears to be a critical error in the way these models are constructed. Nonetheless, even if the assumptions were correct such a model should never have been used. Anyone outside the team who had access to the original code would have seen this problem immediately and could have sounded the alarm, but: 1. Nobody did have access. 2. ICL lied to the press by claiming the code had been published and peer reviewed years earlier (so where was it?) 3. Then when it was revealed the model wasn't reproducible, they teamed up with another academic at Cambridge and lied again by publishing a report+press release claiming it actually was reproducible and claims otherwise were misinformation. 4. And then the journal Nature and the BBC repeated these false claims of reproducibility. All whilst anyone who looked at the GitHub issues list could see it filling up with determinism bugs. If you want citations for all these claims look here, at a summary report I wrote for a friendly MP [2]. So. It's good that the Royal Society is telling people not to engage in censorship, but their justifications for taking that stance reveal they're still living in lala land. By far the deadliest and most dangerous scientific misinformation throughout COVID has come from the formal institutions of science themselves. You could sum all the Substacks together and they would amount to 1% of the misinformation that has been published by government-backed "scientists", zero of whom have been banned from anything or received any penalty whatsoever. For as long as the scientific institutions are in denial about how utterly dishonest their culture has become we will continue to see a weird inversion in which random outsiders point out basic errors in their work and they respond by yelling "disinformation". [1] https://sites.krieger.jhu.edu/iae/files/2022/01/A-Literature-Review-and-Meta-Analysis-of-the-Effects-of-Lockdowns-on-COVID-19-Mortality.pdf https://sites.krieger.jhu.edu/iae/files/2022/01/A-Literature... [2] https://plan99.net/~mike/epidemiology.pdf https://plan99.net/~mike/epidemiology.pdf
- tsimionescu 5y ago
  The meta-study you cite has quite bizarre conclusions. It basically states that isolation behavior is highly effective at preventing Covid deaths, but that lockdowns were a bad predictor of such behavior in people. But, they still seem to attribute huge economic impact to the lockdowns, not the pandemic and natural response (voluntary behavioral changes) to it. Overall it seems that they would have been better served by adding lockdown compliance to their models, which would likely explain much of the difference. It's absurd to claim that lockdowns don't work when a country like Vietnam (100 million people, first detected cases of COVID-19 community spread outside China) had <100 total COVID-19 deaths in 2020. Strict targeted lockdowns, strict isolation requirements after every identified case, for contacts up to the third degree ( contact of someone who was a contact of someone who came in contact with a patient) - these all must have played a role, and any study that fails to explain such extreme success is simply flawed.
  - mike_hearn 5y ago
    Are you sure it states that? Maybe you mean this paragraph: "If this interpretation is true, what Björk et al. (2021) find is that information and signaling is far more important than the strictness of the lockdown. There may be other interpretations, but the point is that studies focusing on timing cannot differentiate between these interpretations. However, if lockdowns have a notable effect, we should see this effect regardless of the timing, and we should identify this effect more correctly by excluding studies that exclusively analyze timing." "It's absurd to claim that lockdowns don't work when a country like Vietnam (100 million people, first detected cases of COVID-19 community spread outside China) had <100 total COVID-19 deaths in 2020" Many countries claim an improbably low level of COVID death. That doesn't mean their numbers can be taken at face value, although poorer countries do seem to be less worse hit simply because they have far fewer weak, obese and elderly people to begin with. It's certainly not absurd to claim lockdowns don't work. You're attempting to refute a meta-analysis of studies looking at all the data with the example of a single country. That's meaningless. A single data point can falsify a theory but it cannot prove a theory. To prove lockdowns work there has to be a consistent impact of the policy.
atrettel 5y ago
Reading this, I am reminded of a quote by Edsger W. Dijkstra from 1975 [1]: > In the good old days physicists repeated each other's experiments, just to be sure. Today they stick to FORTRAN, so that they can share each other's programs, bugs included. [1] https://www.cs.utexas.edu/users/EWD/transcriptions/EWD04xx/EWD498.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD04xx/E...
maximus-decimus 5y ago
The code should still be shared so you can run it on new data (or new test subjects in this case.)

mike_hearn 5y ago

"Re running the authors code with their data will likely just repeat any methodological issues they had" Yes, but that's useful too. COVID-19 lockdowns largely kicked off in the west due to an epidemiological model from Imperial College London, written over a period of many years by the now notorious Professor Neil Ferguson. When his team uploaded a pre-print of his paper and started sending it to government ministers, the code for his model wasn't available. They spent months fighting FOIA requests by claiming they were about to release it, but just had to tidy things up a bit first. When the code was finally uploaded to GitHub the world discovered the reason for delay: the model was a 15,000 lines of C trash fire of race conditions, floating point accumulation errors and memory corruptions in which basically every variable was in global scope and with a single letter name. It was a textbook case of how not to write software. In fact it was a textbook case of why variable names matter because one of the memory corruptions was caused by the author apparently losing track of what a variable named 'k' was meant to contain at that point in the code. Not surprisingly, the model didn't work. Although it had command line flags to set the PRNG seeds these flags were useless: the model generated totally different output every time you ran it, even with fixed seeds. In fact hospital bed demand prediction changed from run to run by more than the size of the entire NHS Nightingale crash hospital building programme, purely due to bugs. And as we now know the model was entirely wrong and the results were catastrophic. Lockdowns had no impact on mortality. There are many people who looked at the data and saw this but here's just the latest meta-analysis of published studies showing that to be true [1]. They destroyed the NHS which now has a cancer treatment backlog and pool of 'missing' patients so large that it cannot possibly catch up, meaning people will die waiting for treatment from the system they supported with their taxes for their entire lives. They destroyed the tax base, leaving the government with an unpayable debt that can be eliminated only via inflation meaning they will have soon destroyed people's savings too. It's just a catastrophe of incompetence and hubris. The incorrectness of the model wasn't due only to programming bugs. The underlying biological assumptions were roughly GCSE level or a bit lower (GCSE is the exams you take at 15/16 in the UK), and it's quite evident that high school germ theory is woefully incomplete. In particular it has nothing to say on the topic of aerosol vs droplet transmission, which appears to be a critical error in the way these models are constructed. Nonetheless, even if the assumptions were correct such a model should never have been used. Anyone outside the team who had access to the original code would have seen this problem immediately and could have sounded the alarm, but: 1. Nobody did have access. 2. ICL lied to the press by claiming the code had been published and peer reviewed years earlier (so where was it?) 3. Then when it was revealed the model wasn't reproducible, they teamed up with another academic at Cambridge and lied again by publishing a report+press release claiming it actually was reproducible and claims otherwise were misinformation. 4. And then the journal Nature and the BBC repeated these false claims of reproducibility. All whilst anyone who looked at the GitHub issues list could see it filling up with determinism bugs. If you want citations for all these claims look here, at a summary report I wrote for a friendly MP [2]. So. It's good that the Royal Society is telling people not to engage in censorship, but their justifications for taking that stance reveal they're still living in lala land. By far the deadliest and most dangerous scientific misinformation throughout COVID has come from the formal institutions of science themselves. You could sum all the Substacks together and they would amount to 1% of the misinformation that has been published by government-backed "scientists", zero of whom have been banned from anything or received any penalty whatsoever. For as long as the scientific institutions are in denial about how utterly dishonest their culture has become we will continue to see a weird inversion in which random outsiders point out basic errors in their work and they respond by yelling "disinformation". [1] https://sites.krieger.jhu.edu/iae/files/2022/01/A-Literature-Review-and-Meta-Analysis-of-the-Effects-of-Lockdowns-on-COVID-19-Mortality.pdf https://sites.krieger.jhu.edu/iae/files/2022/01/A-Literature... [2] https://plan99.net/~mike/epidemiology.pdf https://plan99.net/~mike/epidemiology.pdf

tsimionescu 5y ago
The meta-study you cite has quite bizarre conclusions. It basically states that isolation behavior is highly effective at preventing Covid deaths, but that lockdowns were a bad predictor of such behavior in people. But, they still seem to attribute huge economic impact to the lockdowns, not the pandemic and natural response (voluntary behavioral changes) to it. Overall it seems that they would have been better served by adding lockdown compliance to their models, which would likely explain much of the difference. It's absurd to claim that lockdowns don't work when a country like Vietnam (100 million people, first detected cases of COVID-19 community spread outside China) had <100 total COVID-19 deaths in 2020. Strict targeted lockdowns, strict isolation requirements after every identified case, for contacts up to the third degree ( contact of someone who was a contact of someone who came in contact with a patient) - these all must have played a role, and any study that fails to explain such extreme success is simply flawed.
- mike_hearn 5y ago
  Are you sure it states that? Maybe you mean this paragraph: "If this interpretation is true, what Björk et al. (2021) find is that information and signaling is far more important than the strictness of the lockdown. There may be other interpretations, but the point is that studies focusing on timing cannot differentiate between these interpretations. However, if lockdowns have a notable effect, we should see this effect regardless of the timing, and we should identify this effect more correctly by excluding studies that exclusively analyze timing." "It's absurd to claim that lockdowns don't work when a country like Vietnam (100 million people, first detected cases of COVID-19 community spread outside China) had <100 total COVID-19 deaths in 2020" Many countries claim an improbably low level of COVID death. That doesn't mean their numbers can be taken at face value, although poorer countries do seem to be less worse hit simply because they have far fewer weak, obese and elderly people to begin with. It's certainly not absurd to claim lockdowns don't work. You're attempting to refute a meta-analysis of studies looking at all the data with the example of a single country. That's meaningless. A single data point can falsify a theory but it cannot prove a theory. To prove lockdowns work there has to be a consistent impact of the policy.

atrettel 5y ago

Reading this, I am reminded of a quote by Edsger W. Dijkstra from 1975 [1]: > In the good old days physicists repeated each other's experiments, just to be sure. Today they stick to FORTRAN, so that they can share each other's programs, bugs included. [1] https://www.cs.utexas.edu/users/EWD/transcriptions/EWD04xx/EWD498.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD04xx/E...

maximus-decimus 5y ago

The code should still be shared so you can run it on new data (or new test subjects in this case.)