4 ms·

There's lots of science that can't be reproduced before publishing or where reproducing doesn't help: * Theoretical work * Computer simulations (re-run the sim

There's lots of science that can't be reproduced before publishing or where reproducing doesn't help: * Theoretical work * Computer simulations (re-run the simulation? Well that's not going to detect most issues. Re-create the program from scratch? expensive and hard to make a re-write meaningfully different systematically) * Cosmological observations (can't re-run that supernova!) * Medical case studies (can't just find a new patient) * Large scale studies (build a second LHC? Do results based on the Framingham Study have to kick off another 30 years of data collection before they publish) In fact it's only small and self contained experiments where this is really practical.

Mountain_Skies 4y ago
>Computer simulations (re-run the simulation? Well that's not going to detect most issues It would have detected the fraud going on at ICL with their covid model that was used by the governments of the world to set the course of pandemic response. I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here. Running the same simulation with the same inputs produced wildly different results between runs. That would have been an easily detected red flag but because no one was able to do so until long after the pandemic response was set in motion and changes would have been politically uncomfortable for the powers that be, we ended with a terrible response that many still cannot bring themselves to admit was both poor and instigated by a terrible model that should have been easily detected as being garbage.
- eesmith 4y ago
  There were two popular models, in rough agreement. https://publichealth.jhu.edu/2020/transparency-reproducibility-and-validation-of-covid-19-projection-models https://publichealth.jhu.edu/2020/transparency-reproducibili... > Among the two popular models, we found that the ICL model is more transparent and reproducible compared to the IHME model. The former sometimes over-predicted future deaths while the latter clearly under-predicted post-peak deaths. Both models predicted the timing of peaks reasonably well using data until one week prior. The ICL model produced a much wider band of uncertainty for New York state, possibly because the pattern did not conform well with their internal training data used from European countries. Nature (on 08 June 2020) reported "a computational neuroscientist has reported that he has independently rerun the simulation and reproduced its results. And other scientists have told Nature that they had already privately verified that the code is reproducible." - https://www.nature.com/articles/d41586-020-01685-y https://www.nature.com/articles/d41586-020-01685-y citing the CODECHECK certificate at https://zenodo.org/record/3865491 https://zenodo.org/record/3865491 from May 29, 2020. You should also be asking why public health is so under-funded world-wide that many countries looked to the ICL model - which in turn was a repurposed influenza model - rather than use local expertise or be able to use better validated models. > Running the same simulation with the same inputs produced wildly different results between runs. As I recall, the replication issues were the normal issues related to parallelization and random number generation, and not things which greatly affected the results. There were a number of HN threads on the topic back in 2020. Eg, https://news.ycombinator.com/item?id=23212268 https://news.ycombinator.com/item?id=23212268 , which linked to a couple of those issues and their resolution. > I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here. I can. The Reinhart-Rogoff spreadsheet error which was widely used as the excuse for "austerity" cuts, rather than the real reason of wanting to lower taxes on the rich. https://truthout.org/articles/excel-spreadsheet-error-lessons-from-the-reinhart-rogoff-controversy/ https://truthout.org/articles/excel-spreadsheet-error-lesson... I wonder where that tax money could have been used. Perhaps into public health measures?
rgrieselhuber 4y ago
If it’s not reproducible then assertions such as “the science is settled” and “trust the science” are completely inane and certainly should not be used as policy cudgels.

Mountain_Skies 4y ago

>Computer simulations (re-run the simulation? Well that's not going to detect most issues It would have detected the fraud going on at ICL with their covid model that was used by the governments of the world to set the course of pandemic response. I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here. Running the same simulation with the same inputs produced wildly different results between runs. That would have been an easily detected red flag but because no one was able to do so until long after the pandemic response was set in motion and changes would have been politically uncomfortable for the powers that be, we ended with a terrible response that many still cannot bring themselves to admit was both poor and instigated by a terrible model that should have been easily detected as being garbage.

eesmith 4y ago
There were two popular models, in rough agreement. https://publichealth.jhu.edu/2020/transparency-reproducibility-and-validation-of-covid-19-projection-models https://publichealth.jhu.edu/2020/transparency-reproducibili... > Among the two popular models, we found that the ICL model is more transparent and reproducible compared to the IHME model. The former sometimes over-predicted future deaths while the latter clearly under-predicted post-peak deaths. Both models predicted the timing of peaks reasonably well using data until one week prior. The ICL model produced a much wider band of uncertainty for New York state, possibly because the pattern did not conform well with their internal training data used from European countries. Nature (on 08 June 2020) reported "a computational neuroscientist has reported that he has independently rerun the simulation and reproduced its results. And other scientists have told Nature that they had already privately verified that the code is reproducible." - https://www.nature.com/articles/d41586-020-01685-y https://www.nature.com/articles/d41586-020-01685-y citing the CODECHECK certificate at https://zenodo.org/record/3865491 https://zenodo.org/record/3865491 from May 29, 2020. You should also be asking why public health is so under-funded world-wide that many countries looked to the ICL model - which in turn was a repurposed influenza model - rather than use local expertise or be able to use better validated models. > Running the same simulation with the same inputs produced wildly different results between runs. As I recall, the replication issues were the normal issues related to parallelization and random number generation, and not things which greatly affected the results. There were a number of HN threads on the topic back in 2020. Eg, https://news.ycombinator.com/item?id=23212268 https://news.ycombinator.com/item?id=23212268 , which linked to a couple of those issues and their resolution. > I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here. I can. The Reinhart-Rogoff spreadsheet error which was widely used as the excuse for "austerity" cuts, rather than the real reason of wanting to lower taxes on the rich. https://truthout.org/articles/excel-spreadsheet-error-lessons-from-the-reinhart-rogoff-controversy/ https://truthout.org/articles/excel-spreadsheet-error-lesson... I wonder where that tax money could have been used. Perhaps into public health measures?

rgrieselhuber 4y ago

If it’s not reproducible then assertions such as “the science is settled” and “trust the science” are completely inane and certainly should not be used as policy cudgels.