9 ms·
This is par for the course with computational research. I discovered a bug in my code the last week before submitting my PhD dissertation. Luckily, all of my co
by Xcelerate 8y ago
This is par for the course with computational research. I discovered a bug in my code the last week before submitting my PhD dissertation. Luckily, all of my code was organized in a pipeline that could automatically regenerate everything on the fly, but I needed a supercomputer. The queue was too long for Titan, so I set up an impromptu cluster on Azure (it was the only cloud provider with Infiniband at the time) and paid $200 out of pocket to regenerate the correct figures.
I wouldn’t be surprised if a significant portion of published computational research has bugs that totally invalidate the conclusions. I think we need to push hard to require all taxpayer-funded research to make any code that results in a journal article publicly available.
- thrmsforbfast 8y agoOpen source doesn't ensure quality code. Ideally the code would be part of the peer review process, but code review is really expensive, so who knows how that would play out.
- shkkmo 8y ago> Open source doesn't ensure quality code. Yes, but closed source helps ensure that low quality code is hidden from sight. It also means that people who distrust or doubt the conclusions have no chance to identify any bug(s) and disprove the results or conclusions.
- deleted 8y ago[deleted]
- fifnir 8y agoIt's simple: We stop publishing in papers, and instead adopt smaller chunks of our work as the core publishing units. Each figure should be an individually published entity which contains the entire computational pipeline. Figures are our observations on which we apply logic/philosophy/whatyouwannacallit. Publishing them alongside their relevant code makes the process transparent, reproducible and individually reviewable, as it should be. We can then "publish" comments, observations, conclusions etc on those Figures as a separate thing. Now the logic of the conclusions can be reviewed separately from the statistics and code of the figure.
- no_identd 8y agoI have two words for you: Ted. Nelson.
- chiefalchemist 8y agoA comparable solution would be for all involved to value all research, not just the ground breaking, earth shattering type. As it is, research that yields a "failure" is buried. That means wheels are being reinvented and re-failed. That means there's no opportunity to compare similar "failures", be inspired, and come up with the magic that others overlooked. Unfortunately, I would imagine, even if you can get researchers to agree to this the lawyers are going to have a shit fit. Imagine Google using an IBM "failure" for something truly innovative.
- tokai 8y agoWhat you are proposing sounds a lot like the concept of the least publishable unit. https://en.wikipedia.org/wiki/Least_publishable_unit https://en.wikipedia.org/wiki/Least_publishable_unit
- d0mine 8y agowww was created to publish information in CERN but we can use it in other contexts too ;) http://info.cern.ch/Proposal.html http://info.cern.ch/Proposal.html
- agumonkey 8y agosimilar to reproducible builds or nix research just jumped onto jupyter notebooks, it's halfway there, someone helps the remaining step
- ChrisFoster 8y agoTrue, but it does provide at least some measure of reproducibility. Quality of implementation and reproducibility are orthogonal and both very valuable in their own right.
- darpa_escapee 8y agoIt's not about code quality, it's about transparency and ease of reproduction.
- lvh 8y agoOf course it won’t ensure anything, but currently being completely unable to reproduce results, even as the author but just a year from now, is par for the course.
- deleted 8y ago[deleted]
- BurningFrog 8y agoCode review is cheap. I do it for fun. But it doesn't prove anything. Science should prove things...
- m_mueller 8y agoscience can never prove anything as a matter of principle. it can only disprove all the alternatives. math and logic can prove, but only within the model it has built up, which has been shown to contain unprovable axioms that one must simply accept.
- BurningFrog 8y agoYeah, I'm aware of the strict theory.
- Fomite 8y agoLittle of what I do, even with the most rigorous methods available and the best practices from both software development and computational science, proves anything.
- BurningFrog 8y agoI know. And I think it's a problem for science... Logical proofs will never happen for software development, but surely standards for scientific programming can be tightened up a few levels! I think I heard of some reform proposals from the Reproducibility Crisis reformers.
- Fomite 8y agoI more mean there are whole aspects of science that aren't provable without being able to actually obtain counterfactuals, and that means time machines
- andyv 8y agoComputational research is incredibly difficult because it's usually hard to see the effect of a bug. A triangle drawn on the wrong place of a screen can be easy to see, but a typo in your integration subroutine? Hard to spot if you don't catch it when it is born. I also had a few do-overs at the end of my thesis, but fortunately had a cluster standing by...
- BeetleB 8y ago>A triangle drawn on the wrong place of a screen can be easy to see, but a typo in your integration subroutine? Hard to spot if you don't catch it when it is born. Well, there's also this notion of testing and regression. As I said in another comment a few days ago: >A few weeks ago I had a conversation with a friend of mine who is wrapping up his PhD. He pointed out that not one of his colleagues is concerned whether anyone can reproduce their work. They use a home grown simulation suite which only they have access to, and is constantly being updated with the worst software practices you can think of. No one in their team believes that the tool will give the same results they did 4 years ago. The troubling part is, no one sees that as being a problem. They got their papers published, and so the SW did its job.
- fouc 8y agoWow. If it's not reproducible, can it even be called science?
- BurningFrog 8y agoPerhaps it's a reproducible way to build careers?
- slavik81 8y agoIt's not really much different from a physical experiment, where the expectation is that you'll have to rebuild the experimental apparatus yourself to reproduce the experiment. An independent implementation of the experiment is neccessary for a full reproduction anyways. If you just run their code again, you'll end up with all their bugs again. (But, don't get me wrong. I like when researchers release their code. It's still very useful.)
- nestorD 8y agoA few days ago in the description of a paper found on hackernews (https://news.ycombinator.com/item?id=18346943 https://news.ycombinator.com/item?id=18346943) : > We also noticed significant improvements in performance of RND every time we discovered and fixed a bug [...]. Getting such details right was a significant part of achieving high performance even with algorithms conceptually similar to prior work. They call bugs 'details' which, I find, is a frightening state of mind for someone publishing an algorithm.
- geezerjay 8y ago> They call bugs 'details' which, I find, is a frightening state of mind for someone publishing an algorithm. It really depends on the algorithm. For example, a bug in the random number generator of a stochastic search algorithm that affects, say, the variance of a distribution won't have a relevant impact on the outcome.
- mlthoughts2018 8y agoThat could have a huge impact. For example if it affected random draws of a hyperparameter in a Bayesian model, leading to incorrect credible intervals ultimately in the posterior distribution. Or worse, if the RNG bug was affecting the variance of randon samples in some deep component of an MCMC algorithm like NUTS or even simple Metropolis. Depending on the exact nature of the bug, it could even cause the model to violate Detailed Balance, and entirely invalidate sampling based inferences or conclusions.
- riskneutral 8y agoWell that is why you would always run statistical validity tests on the RNG and other intermediate values whenever using a Monte Carlo model in production. Ideally the tests should run as part of a Continuous Integration workflow.
- mlthoughts2018 8y agoIf you (dubiously) write your own RNG, sure. But you should never do that. And for library RNGs, you should execute the unit tests of the library. Frankly, running it as part of CI is at best overkill and at worst adds complexity that costs you. If you pin versions of your dependency and isolate the artifact into an in-house artifact repository, so that the library code is literally never changing unless you explicitly modify the version, then you should test it up front, then again only occasionally if you actually have evidence of a bug. And as part of any code review of the code change that introduces a version change.
- badpun 8y agoThis is why I'm a bit skeptical about the global warming predictions. AFAIK they're all based on multi-million lines of code models. Not only it's monumentally hard to have such large code base without bugs, but also, in case of a software implementing scientific models, finding bugs is extra hard (compared to software which works in a visible way and has millions of users, such as Linux kernel, games, car's embedded software etc.). An effort to have such model verified (let's say to a NASA standard) to be bug-free would probably cost billions of dollars. And that's only bugs, let's not forget that all the models are approximations, plus the numerical methods used all have their quirks and limitations etc. All in all, the problem seems too hard to tackle for humanity right now.
- nafey 8y agoThe counterpoint is that we have already seen a steady temperature rise. So even though the specifics of the various simulations might not play out as predicted we can expect temperatures to rise.
- badpun 8y agoKnowing that the temperature will rise is not enough to make policy decisions though. You need to predict the increase's magnitude, as well as practical consequences, such as climate change, how much the sea level will rise etc. And for that you need resonable and bug-free models.
- leeoniya 8y ago> how much the sea level will rise etc this will be the least of our problems. we understand very little about nature to have any reasonable prediction model. we don't know the inflection point which will cause massive collapse of major ecosystems we depend on. all we know is that things are changing fast. faster than many non-human organisms are able to adapt.
- femidav 8y agoSteady? We have seen 20-years rise after a colder period. Nothing out of order.
- bad_news_bears 8y agoWas there a large, I guess “significant” difference between your results pre- and post-bug?
- deleted 8y ago[deleted]
- fghtr 8y ago> I think we need to push hard to require all taxpayer-funded research to make any code that results in a journal article publicly available. "Already 18824 SIGNATURES – sign the open letter now!" https://publiccode.eu https://publiccode.eu