4 ms·
My critique isn't aimed at computer science research--it applies equally well to other fields using computation to produce results. There are a lot of projects
by angersock 11y ago
My critique isn't aimed at computer science research--it applies equally well to other fields using computation to produce results.
There are a lot of projects (and I won't name names, because I've got friends here who work on them) whose results cannot be reproduced, because they ran on some weird-ass combination of a weird libc, using a weird build system, running on weird hardware environments (especially common in HPC), with weird one-off data sets.
They report the results using this setup, and then the setup is gone forever. The code is probably not kept correctly in source control, and generally the community just takes them at their word that the results they get are authentic.
When you run millions of iterations on some crazy quantum solver, it's hard to spot if you're wrong. If others can't at least run the same result, science becomes a matter of anecdote.
- coliveira 11y agoAs hard to believe as it is for software engineers, science is not in the code, it is in the thought process that leads to the code and in the interpretation of the results. The code for an experiment can be lost forever, but if you did your job correctly you have described why, how, and where the experiment works. Future scientists can now "build on the shoulders" of this knowledge and use it to go further and validate the previous result or discredit if it is indeed incorrect. People complain that published papers are full of incorrect results, but they don't understand that this is just part of the process. Incorrect results will be forgotten because they will never be validated. Correct results will be validated and used (cited) by new scientists as they try discover even more complicated things.
- collyw 11y agoThings would work so much better if the software was written in a better way.