9 ms·

> I just throw out papers that don’t publish datasets and code There was an interesting research paper that claimed you could up your IQ with a computer game c

> I just throw out papers that don’t publish datasets and code There was an interesting research paper that claimed you could up your IQ with a computer game called dual n back and a lot of paper tried to replicate it, but my mind was blown when I realized none of them used the same code. None of them actually shared the code they used for their research and then they all claimed they were testing the same thing when they obviously weren't. To me, refusing to share the source code to a program that could be used to replicate your research seems like a big middle finger to science itself. It shows a total disregard to study replication.

bikingbismuth 5y ago
I have been adjacent to industrial and academic partnerships where both the university and company wanted to maintain the IP of the work and in their minds that extended to the software. Paper was published and the source code was closely guarded. I wondered how people could replicate the findings without the fairly complex system the researchers used.
judge2020 5y ago
Exactly, it'd be like omitting the "methodology" section of the paper. There's no way to verify or prove wrong a paper if you don't know how they reached their conclusions.
- swayvil 5y ago
  Remove description of methodology and science becomes mere dogma.
  - 6510 5y ago
    and dogma science!
- nextos 5y ago
  Even more than sharing code, I think the key is to share raw data, postprocessed data and describe methods as exceptionally clear equations and pseudocode. Some well intentioned papers share code which is hard to run years afterwards. It's sometimes much simpler to reimplement things that to get the code to run if they are described accurately. Besides, some articles hide ugly things, nasty tricks and lies in the code, which make their results a lot less believable and valid. Being super upfront about models in terms of equations and pseudocode is important. Of course, we should also have standards to make code reproducible. Perhaps depositing a VM with everything ready to run.
  - zrm 5y ago
    > Perhaps depositing a VM with everything ready to run. The danger then is that the VM has so much undocumented complexity that if anything goes wrong, or goes "well" when it shouldn't, no one can explain why. Which also reintroduces a vector to hide nasty tricks.
    - ungamedplayer 5y ago
      But you can at least figure that out. If U don't have it all results are suspect.
      - kergonath 5y ago
        Results that haven't been independently replicated are suspect. There are just too many factors that can lead an experiment to give some results that are not transferrable or not relevant. The worst aspect of this is the lack of will or funding to replicate, replicate, and replicate again all significant results that get published. Post-processed data can be altered, but a TB of raw data is meaningless as well if it hasn't been produced properly, has been obfuscated, or is weirdly formatted. Data availability is a red herring for the vast majority of the science being made right now (almost everything that does not depend on a multi-millions dollars experiment). If data availability is an end in itself, we would just have moved the goalposts and have a data quality problem instead of a reproducibility problem.
    - hirako2000 5y ago
      this. the point of sharing reproducible steps and not the experiment itself is that it can be fully reproduced independently. not just independently verify that the result show what the paper claims.
  - mrandish 5y ago
    > Perhaps depositing a VM with everything ready to run. Yes, this is hugely important. We need clearer requirements for what constitutes properly 'published' data, code and methods. It should include all raw data (both used and discarded) as well as complete snapshots of each preliminary and intermediate processing step in a time-stamped chronology. This is an area where the expertise and best practices of software development, documentation and source control could help inform standards and tooling which would dramatically improve the quality of scientific publishing and replication.
ransom1538 5y ago
"To me, refusing to share the source code to a program that could be used to replicate your research seems like a big middle finger to science itself." It is done on purpose.
- lstodd 5y ago
  > It is done on purpose. The sad thing is that it isn't. It's just how the things are done. src: friends in HEP
  - NmAmDa 5y ago
    Sharing source code for a HEP experiment is not that easy or possible at sometimes. A lot of stuff is done by different group and whole framework used and overall a lot of raw data collected and reconstructed involved. To analyze those even if available you willl need a lot of people with a lot of resources. So even it made available (which would be a lot of efforts itself) it wouldn't make much sense for replication purpose.
    - patcassidy2000 5y ago
      How is it possible to peer review a paper when the only people qualified to do so are the one's involved in the research. Seems like a massive issue. Don't want to cast to much shade here, but there's a fair number of people that called out high energy physics for having problematic methodology. See Constructing quarks by Andrew Pickering. Ultimately, it should be CERNS job to release the data, yes even if it is terabytes in size, because that's the whole point of science.
      - HideousKojima 5y ago
        >How is it possible to peer review a paper when the only people qualified to do so are the one's involved in the research. Seems like a massive issue. Peer review isn't what it's chalked up to be. Basically it's two other people in the same field saying "There's no blatantly obvious issues with this paper" and even that isn't always guaranteed. Reviewers don't make any efforts to actually replicate the research, crunch the data, etc.
Retric 5y ago
Sharing code and data can be as harmful to science as it is beneficial. You don’t want to reuse the same instruments that conducted the last experiment to validate it. Independent replication is inherently expensive, but also critical to the field at large. Some sort of code vault that releases the code and data after a period could be a solid compromise.
- kortex 5y ago
  Well yes, but no. You want to avoid any systematic bias, which if you reuse tooling/instruments/code, you run the risk of, but source code is also something which can be peer-reviewed. Reproducing the code base from scratch just isn't going to be tractable for large scientific pieces of software (e.g. CERN). Better to have the source code and dig through it carefully as part of the review and replication than insist everyone write their own cleanroom copy just to avoid systematic bias.
  - deleted 5y ago
    [deleted]
  - Retric 5y ago
    Suppose someone decided to build another LHC, but to save money they would use exactly the same design, constriction crew, and code used to build the first one. Would you consider that a perfectly reasonable choice or would it seem risky? That said I am all for peer reviewing code, but that’s not where this ends up. People are going to reuse code they don’t bother to dig into because people are people. The obvious next step would be to reject any paper that’s reusing code from a different team, but that’s excessive. That said reusing code or data is much safer than reusing code and data.
    - rurban 5y ago
      For that reason a special multimedia sharing tool was created, which is now called WWW. So the international physicist community could share their code, papers and CAD designs for the LHC experiments. Quite a success for them, but academia is still resistant
skybrian 5y ago
Ideally you would re-implement the code to the same spec and compare the outputs. This, too, is a form of replication. If the results are sufficiently different, then look at the source?
helsinkiandrew 5y ago
Re running the authors code with their data will likely just repeat any methodological issues they had (either accidentally or fraudulently). In medical studies, one level is to reevaluate the data from the same set of patients to see if any bias or errors wasn't included, perhaps with a different or newer statistical idealogy. The best is for the study to be repeated with a fresh set of patient data to see if the underlying conclusions were valid.
- mike_hearn 5y ago
  "Re running the authors code with their data will likely just repeat any methodological issues they had" Yes, but that's useful too. COVID-19 lockdowns largely kicked off in the west due to an epidemiological model from Imperial College London, written over a period of many years by the now notorious Professor Neil Ferguson. When his team uploaded a pre-print of his paper and started sending it to government ministers, the code for his model wasn't available. They spent months fighting FOIA requests by claiming they were about to release it, but just had to tidy things up a bit first. When the code was finally uploaded to GitHub the world discovered the reason for delay: the model was a 15,000 lines of C trash fire of race conditions, floating point accumulation errors and memory corruptions in which basically every variable was in global scope and with a single letter name. It was a textbook case of how not to write software. In fact it was a textbook case of why variable names matter because one of the memory corruptions was caused by the author apparently losing track of what a variable named 'k' was meant to contain at that point in the code. Not surprisingly, the model didn't work. Although it had command line flags to set the PRNG seeds these flags were useless: the model generated totally different output every time you ran it, even with fixed seeds. In fact hospital bed demand prediction changed from run to run by more than the size of the entire NHS Nightingale crash hospital building programme, purely due to bugs. And as we now know the model was entirely wrong and the results were catastrophic. Lockdowns had no impact on mortality. There are many people who looked at the data and saw this but here's just the latest meta-analysis of published studies showing that to be true [1]. They destroyed the NHS which now has a cancer treatment backlog and pool of 'missing' patients so large that it cannot possibly catch up, meaning people will die waiting for treatment from the system they supported with their taxes for their entire lives. They destroyed the tax base, leaving the government with an unpayable debt that can be eliminated only via inflation meaning they will have soon destroyed people's savings too. It's just a catastrophe of incompetence and hubris. The incorrectness of the model wasn't due only to programming bugs. The underlying biological assumptions were roughly GCSE level or a bit lower (GCSE is the exams you take at 15/16 in the UK), and it's quite evident that high school germ theory is woefully incomplete. In particular it has nothing to say on the topic of aerosol vs droplet transmission, which appears to be a critical error in the way these models are constructed. Nonetheless, even if the assumptions were correct such a model should never have been used. Anyone outside the team who had access to the original code would have seen this problem immediately and could have sounded the alarm, but: 1. Nobody did have access. 2. ICL lied to the press by claiming the code had been published and peer reviewed years earlier (so where was it?) 3. Then when it was revealed the model wasn't reproducible, they teamed up with another academic at Cambridge and lied again by publishing a report+press release claiming it actually was reproducible and claims otherwise were misinformation. 4. And then the journal Nature and the BBC repeated these false claims of reproducibility. All whilst anyone who looked at the GitHub issues list could see it filling up with determinism bugs. If you want citations for all these claims look here, at a summary report I wrote for a friendly MP [2]. So. It's good that the Royal Society is telling people not to engage in censorship, but their justifications for taking that stance reveal they're still living in lala land. By far the deadliest and most dangerous scientific misinformation throughout COVID has come from the formal institutions of science themselves. You could sum all the Substacks together and they would amount to 1% of the misinformation that has been published by government-backed "scientists", zero of whom have been banned from anything or received any penalty whatsoever. For as long as the scientific institutions are in denial about how utterly dishonest their culture has become we will continue to see a weird inversion in which random outsiders point out basic errors in their work and they respond by yelling "disinformation". [1] https://sites.krieger.jhu.edu/iae/files/2022/01/A-Literature-Review-and-Meta-Analysis-of-the-Effects-of-Lockdowns-on-COVID-19-Mortality.pdf https://sites.krieger.jhu.edu/iae/files/2022/01/A-Literature... [2] https://plan99.net/~mike/epidemiology.pdf https://plan99.net/~mike/epidemiology.pdf

bikingbismuth 5y ago

I have been adjacent to industrial and academic partnerships where both the university and company wanted to maintain the IP of the work and in their minds that extended to the software. Paper was published and the source code was closely guarded. I wondered how people could replicate the findings without the fairly complex system the researchers used.

judge2020 5y ago

Exactly, it'd be like omitting the "methodology" section of the paper. There's no way to verify or prove wrong a paper if you don't know how they reached their conclusions.

swayvil 5y ago
Remove description of methodology and science becomes mere dogma.
- 6510 5y ago
  and dogma science!
nextos 5y ago
Even more than sharing code, I think the key is to share raw data, postprocessed data and describe methods as exceptionally clear equations and pseudocode. Some well intentioned papers share code which is hard to run years afterwards. It's sometimes much simpler to reimplement things that to get the code to run if they are described accurately. Besides, some articles hide ugly things, nasty tricks and lies in the code, which make their results a lot less believable and valid. Being super upfront about models in terms of equations and pseudocode is important. Of course, we should also have standards to make code reproducible. Perhaps depositing a VM with everything ready to run.
- zrm 5y ago
  > Perhaps depositing a VM with everything ready to run. The danger then is that the VM has so much undocumented complexity that if anything goes wrong, or goes "well" when it shouldn't, no one can explain why. Which also reintroduces a vector to hide nasty tricks.
  - ungamedplayer 5y ago
    But you can at least figure that out. If U don't have it all results are suspect.
    - kergonath 5y ago
      Results that haven't been independently replicated are suspect. There are just too many factors that can lead an experiment to give some results that are not transferrable or not relevant. The worst aspect of this is the lack of will or funding to replicate, replicate, and replicate again all significant results that get published. Post-processed data can be altered, but a TB of raw data is meaningless as well if it hasn't been produced properly, has been obfuscated, or is weirdly formatted. Data availability is a red herring for the vast majority of the science being made right now (almost everything that does not depend on a multi-millions dollars experiment). If data availability is an end in itself, we would just have moved the goalposts and have a data quality problem instead of a reproducibility problem.
  - hirako2000 5y ago
    this. the point of sharing reproducible steps and not the experiment itself is that it can be fully reproduced independently. not just independently verify that the result show what the paper claims.
- mrandish 5y ago
  > Perhaps depositing a VM with everything ready to run. Yes, this is hugely important. We need clearer requirements for what constitutes properly 'published' data, code and methods. It should include all raw data (both used and discarded) as well as complete snapshots of each preliminary and intermediate processing step in a time-stamped chronology. This is an area where the expertise and best practices of software development, documentation and source control could help inform standards and tooling which would dramatically improve the quality of scientific publishing and replication.

ransom1538 5y ago

"To me, refusing to share the source code to a program that could be used to replicate your research seems like a big middle finger to science itself." It is done on purpose.

lstodd 5y ago
> It is done on purpose. The sad thing is that it isn't. It's just how the things are done. src: friends in HEP
- NmAmDa 5y ago
  Sharing source code for a HEP experiment is not that easy or possible at sometimes. A lot of stuff is done by different group and whole framework used and overall a lot of raw data collected and reconstructed involved. To analyze those even if available you willl need a lot of people with a lot of resources. So even it made available (which would be a lot of efforts itself) it wouldn't make much sense for replication purpose.
  - patcassidy2000 5y ago
    How is it possible to peer review a paper when the only people qualified to do so are the one's involved in the research. Seems like a massive issue. Don't want to cast to much shade here, but there's a fair number of people that called out high energy physics for having problematic methodology. See Constructing quarks by Andrew Pickering. Ultimately, it should be CERNS job to release the data, yes even if it is terabytes in size, because that's the whole point of science.
    - HideousKojima 5y ago
      >How is it possible to peer review a paper when the only people qualified to do so are the one's involved in the research. Seems like a massive issue. Peer review isn't what it's chalked up to be. Basically it's two other people in the same field saying "There's no blatantly obvious issues with this paper" and even that isn't always guaranteed. Reviewers don't make any efforts to actually replicate the research, crunch the data, etc.

Retric 5y ago

Sharing code and data can be as harmful to science as it is beneficial. You don’t want to reuse the same instruments that conducted the last experiment to validate it. Independent replication is inherently expensive, but also critical to the field at large. Some sort of code vault that releases the code and data after a period could be a solid compromise.

kortex 5y ago
Well yes, but no. You want to avoid any systematic bias, which if you reuse tooling/instruments/code, you run the risk of, but source code is also something which can be peer-reviewed. Reproducing the code base from scratch just isn't going to be tractable for large scientific pieces of software (e.g. CERN). Better to have the source code and dig through it carefully as part of the review and replication than insist everyone write their own cleanroom copy just to avoid systematic bias.
- deleted 5y ago
  [deleted]
- Retric 5y ago
  Suppose someone decided to build another LHC, but to save money they would use exactly the same design, constriction crew, and code used to build the first one. Would you consider that a perfectly reasonable choice or would it seem risky? That said I am all for peer reviewing code, but that’s not where this ends up. People are going to reuse code they don’t bother to dig into because people are people. The obvious next step would be to reject any paper that’s reusing code from a different team, but that’s excessive. That said reusing code or data is much safer than reusing code and data.
  - rurban 5y ago
    For that reason a special multimedia sharing tool was created, which is now called WWW. So the international physicist community could share their code, papers and CAD designs for the LHC experiments. Quite a success for them, but academia is still resistant

skybrian 5y ago

Ideally you would re-implement the code to the same spec and compare the outputs. This, too, is a form of replication. If the results are sufficiently different, then look at the source?

helsinkiandrew 5y ago

Re running the authors code with their data will likely just repeat any methodological issues they had (either accidentally or fraudulently). In medical studies, one level is to reevaluate the data from the same set of patients to see if any bias or errors wasn't included, perhaps with a different or newer statistical idealogy. The best is for the study to be repeated with a fresh set of patient data to see if the underlying conclusions were valid.

mike_hearn 5y ago
"Re running the authors code with their data will likely just repeat any methodological issues they had" Yes, but that's useful too. COVID-19 lockdowns largely kicked off in the west due to an epidemiological model from Imperial College London, written over a period of many years by the now notorious Professor Neil Ferguson. When his team uploaded a pre-print of his paper and started sending it to government ministers, the code for his model wasn't available. They spent months fighting FOIA requests by claiming they were about to release it, but just had to tidy things up a bit first. When the code was finally uploaded to GitHub the world discovered the reason for delay: the model was a 15,000 lines of C trash fire of race conditions, floating point accumulation errors and memory corruptions in which basically every variable was in global scope and with a single letter name. It was a textbook case of how not to write software. In fact it was a textbook case of why variable names matter because one of the memory corruptions was caused by the author apparently losing track of what a variable named 'k' was meant to contain at that point in the code. Not surprisingly, the model didn't work. Although it had command line flags to set the PRNG seeds these flags were useless: the model generated totally different output every time you ran it, even with fixed seeds. In fact hospital bed demand prediction changed from run to run by more than the size of the entire NHS Nightingale crash hospital building programme, purely due to bugs. And as we now know the model was entirely wrong and the results were catastrophic. Lockdowns had no impact on mortality. There are many people who looked at the data and saw this but here's just the latest meta-analysis of published studies showing that to be true [1]. They destroyed the NHS which now has a cancer treatment backlog and pool of 'missing' patients so large that it cannot possibly catch up, meaning people will die waiting for treatment from the system they supported with their taxes for their entire lives. They destroyed the tax base, leaving the government with an unpayable debt that can be eliminated only via inflation meaning they will have soon destroyed people's savings too. It's just a catastrophe of incompetence and hubris. The incorrectness of the model wasn't due only to programming bugs. The underlying biological assumptions were roughly GCSE level or a bit lower (GCSE is the exams you take at 15/16 in the UK), and it's quite evident that high school germ theory is woefully incomplete. In particular it has nothing to say on the topic of aerosol vs droplet transmission, which appears to be a critical error in the way these models are constructed. Nonetheless, even if the assumptions were correct such a model should never have been used. Anyone outside the team who had access to the original code would have seen this problem immediately and could have sounded the alarm, but: 1. Nobody did have access. 2. ICL lied to the press by claiming the code had been published and peer reviewed years earlier (so where was it?) 3. Then when it was revealed the model wasn't reproducible, they teamed up with another academic at Cambridge and lied again by publishing a report+press release claiming it actually was reproducible and claims otherwise were misinformation. 4. And then the journal Nature and the BBC repeated these false claims of reproducibility. All whilst anyone who looked at the GitHub issues list could see it filling up with determinism bugs. If you want citations for all these claims look here, at a summary report I wrote for a friendly MP [2]. So. It's good that the Royal Society is telling people not to engage in censorship, but their justifications for taking that stance reveal they're still living in lala land. By far the deadliest and most dangerous scientific misinformation throughout COVID has come from the formal institutions of science themselves. You could sum all the Substacks together and they would amount to 1% of the misinformation that has been published by government-backed "scientists", zero of whom have been banned from anything or received any penalty whatsoever. For as long as the scientific institutions are in denial about how utterly dishonest their culture has become we will continue to see a weird inversion in which random outsiders point out basic errors in their work and they respond by yelling "disinformation". [1] https://sites.krieger.jhu.edu/iae/files/2022/01/A-Literature-Review-and-Meta-Analysis-of-the-Effects-of-Lockdowns-on-COVID-19-Mortality.pdf https://sites.krieger.jhu.edu/iae/files/2022/01/A-Literature... [2] https://plan99.net/~mike/epidemiology.pdf https://plan99.net/~mike/epidemiology.pdf