3 ms·

The problem is that the "direct and economical way" tends to basically enforce bit rot and lack a reproducibility. It also tends to screw whoever inherits the p

The problem is that the "direct and economical way" tends to basically enforce bit rot and lack a reproducibility. It also tends to screw whoever inherits the project.

coliveira 11y ago
Reproducibility is an issue in all scientific fields, it is not something unique to computer science. Can someone easily reproduce experiments performed in a particle accelerator? What about an extremely complex chemical reaction? The answer to these problems is to have more people doing research in that area to make sure that the problem is well understood, instead of requiring that scientists slow down the research because it needs to first achieve some kind of "engineering reproduction" standard.
- bbgm 11y ago
  Speaking for the chemical reaction - yes. That is exactly what should be possible. Some reactions are hard to reproduce and labs spend years trying to figure out how to do so. Without that you really don't have good science. The key is .. can someone read your paper and reproduce your experiment and the results. If not, it's not valid; it's just a paper.
  - dalke 11y ago
    That's not true. Some types of chemistry cannot (legally) be reproduced, but they are still good and valid science. For a trivial example, consider the many chemistry papers based on measuring fallout effects from nuclear testing. More prosaically, some things can be verified even when a chemical process cannot be reproduced. Protein crystallization is a tricky process with many irreducible results. Yet if you have a crystal, and can use the crystal to determine the protein structure, then you can verify that the crystal indeed contains that protein - even if you are unable to reproduce the chemical process used to create the crystal in the first place. Others can use the same sample to re-verify the resulting x-ray structure even though they also might be unable to reproduce the crystallization step.
- angersock 11y ago
  My critique isn't aimed at computer science research--it applies equally well to other fields using computation to produce results. There are a lot of projects (and I won't name names, because I've got friends here who work on them) whose results cannot be reproduced, because they ran on some weird-ass combination of a weird libc, using a weird build system, running on weird hardware environments (especially common in HPC), with weird one-off data sets. They report the results using this setup, and then the setup is gone forever. The code is probably not kept correctly in source control, and generally the community just takes them at their word that the results they get are authentic. When you run millions of iterations on some crazy quantum solver, it's hard to spot if you're wrong. If others can't at least run the same result, science becomes a matter of anecdote.
  - coliveira 11y ago
    As hard to believe as it is for software engineers, science is not in the code, it is in the thought process that leads to the code and in the interpretation of the results. The code for an experiment can be lost forever, but if you did your job correctly you have described why, how, and where the experiment works. Future scientists can now "build on the shoulders" of this knowledge and use it to go further and validate the previous result or discredit if it is indeed incorrect. People complain that published papers are full of incorrect results, but they don't understand that this is just part of the process. Incorrect results will be forgotten because they will never be validated. Correct results will be validated and used (cited) by new scientists as they try discover even more complicated things.
    - collyw 11y ago
      Things would work so much better if the software was written in a better way.

The problem is that the "direct and economical way" tends to basically enforce bit rot and lack a reproducibility. It also tends to screw whoever inherits the project.

coliveira 11y ago

Reproducibility is an issue in all scientific fields, it is not something unique to computer science. Can someone easily reproduce experiments performed in a particle accelerator? What about an extremely complex chemical reaction? The answer to these problems is to have more people doing research in that area to make sure that the problem is well understood, instead of requiring that scientists slow down the research because it needs to first achieve some kind of "engineering reproduction" standard.

bbgm 11y ago
Speaking for the chemical reaction - yes. That is exactly what should be possible. Some reactions are hard to reproduce and labs spend years trying to figure out how to do so. Without that you really don't have good science. The key is .. can someone read your paper and reproduce your experiment and the results. If not, it's not valid; it's just a paper.
- dalke 11y ago
  That's not true. Some types of chemistry cannot (legally) be reproduced, but they are still good and valid science. For a trivial example, consider the many chemistry papers based on measuring fallout effects from nuclear testing. More prosaically, some things can be verified even when a chemical process cannot be reproduced. Protein crystallization is a tricky process with many irreducible results. Yet if you have a crystal, and can use the crystal to determine the protein structure, then you can verify that the crystal indeed contains that protein - even if you are unable to reproduce the chemical process used to create the crystal in the first place. Others can use the same sample to re-verify the resulting x-ray structure even though they also might be unable to reproduce the crystallization step.
angersock 11y ago
My critique isn't aimed at computer science research--it applies equally well to other fields using computation to produce results. There are a lot of projects (and I won't name names, because I've got friends here who work on them) whose results cannot be reproduced, because they ran on some weird-ass combination of a weird libc, using a weird build system, running on weird hardware environments (especially common in HPC), with weird one-off data sets. They report the results using this setup, and then the setup is gone forever. The code is probably not kept correctly in source control, and generally the community just takes them at their word that the results they get are authentic. When you run millions of iterations on some crazy quantum solver, it's hard to spot if you're wrong. If others can't at least run the same result, science becomes a matter of anecdote.
- coliveira 11y ago
  As hard to believe as it is for software engineers, science is not in the code, it is in the thought process that leads to the code and in the interpretation of the results. The code for an experiment can be lost forever, but if you did your job correctly you have described why, how, and where the experiment works. Future scientists can now "build on the shoulders" of this knowledge and use it to go further and validate the previous result or discredit if it is indeed incorrect. People complain that published papers are full of incorrect results, but they don't understand that this is just part of the process. Incorrect results will be forgotten because they will never be validated. Correct results will be validated and used (cited) by new scientists as they try discover even more complicated things.
  - collyw 11y ago
    Things would work so much better if the software was written in a better way.