28 ms·

Challenge to scientists: does your ten-year-old code still run?

• ecmascript 6y ago
  I am not a scientist, but actually I think most of the code I wrote 10 years ago still is in production at different companies.
  • dhosek 6y ago
    If it's still in production, it's most likely still getting some level of maintenance attention as well. When I was an undergrad I did some coding for some of the professors at the college. A lot of scientific programming is stuff that gets written and run once and never run again. Try dusting off some 10-year-old C++ and try compiling it with the current version of your compiler.
  • jnxx 6y ago
    Companies with code in production have a short-term real incentive to keep that code running. This is different from code from research projects, which is on many cases just run a few times, and in other times, written by somebody who has, if he / she wants to make any kind of career in the field, to change to a new workplace and will not have any time to maintain that old code. There are a few long-running mayor science projects, say, in particle physics or astronomy, which are forced to work differently. And in these environments, there are actually people who have knowledge on both science and software engineering.
  • hex1848 6y ago
    I just shut down a VB6 app that had been running since 1998 at the company I work for last month. The leadership team finally decided they didn't want to sell that particular feature anymore. We still have a handful of apps from around that time period that do various small tasks. One day it will be a priority to get rid of them.
• dhosek 6y ago
  Back in the 80s/90s I was heavily into TeX/LaTeX—I was responsible for a major FTP archive that predated CTAN, wrote ports for some of the utilities to VM/CMS and VAX/VMS and taught classes in LaTeX for the TeX Users Group. I wrote most of a book on LaTeX based on those classes that a few years back I thought I'd resurrect. Even something as stable as LaTeX has evolved enough that just getting the book to recompile with a contemporary TeX distribution was a challenge. (On the other hand, I've also found that a lot of what I knew from 20+ years ago is still valid and I'm able to still be helpful on the TeX stack exchange site).
• lordnacho 6y ago
  You often run into code of the "just get it to work" variety, which has the problem that when it was written, maintainability was bottom of the list of priorities. Often the author has a goal that isn't described in terms of software engineering terms: calculate my option model, work out the hedge amounts, etc. And the people who write this kind of code tend not to think about version control, documentation, dependency management, deployment, and so forth. The result is you get these fragile pieces holding up some very complex logic, which takes a lot of effort to understand. IMO there should be a sort of code literacy course that everyone who writes anything needs to do. In a way it's the modern equivalent of everyone who writes needing to understand not just grammar but style and other writing related hygiene.
  • throwanem 6y ago
    The fundamental problem here, as you note, is that scientists are rarely also engineers, and don't really share our desiderata. The point is to develop and publish a result, and engineering analysis code for resiliency is of secondary concern at best when that code isn't likely to need to be used again once the paper is finished. The "Software Carpentry" movement [1] has in the past decade tried to address this, as I recall. It's very much in the vein of the "basic literacy" course you suggest. I can't say how far they've gotten, and I'm no longer adjacent to academia, but based on what I do still see of academics' code, there's a long way still to go. [1] https://software-carpentry.org/ https://software-carpentry.org/
    • detaro 6y ago
      And that scientists also are rarely supported by programmers, or if they are it's an unstable and unappreciated position.
      • non-entity 6y ago
        Ive seen job listings for "scientific programmers" where what they're asking for is a scientist who happens to know a little programming.
• matsemann 6y ago
  Would an abandoned project I wrote 10 years ago still run? The code is probably fine, but getting it to actually run by linking up whatever libraries, sdks and environment correctly could be troublesome. Even a small pipeline a wrote a few weeks ago I had trouble re-running, because I forgot there was a manual step I had to do on the input file. Expecting more rigid software practices of scientists than software engineers would be wrong. I don't think they should have to tangle with this, tools should aid them somehow.
  • tyingq 6y ago
    It's interesting that it's often easier to get something 25+ years old running because I need fewer things. Not so hard to find, say "DosBox" and and old version of Turbo Pascal.
    • dhosek 6y ago
      When I was in my 20s I managed to get a contract updating some control software for a contact lens company on the basis of my happening to own an old copy of Borland C++ 1.0.
      • eythian 6y ago
        Had a similar experience getting a contract updating a mass spectrometer control system because I had extensive high school experience in Turbo Pascal.
    • lebuffon 6y ago
      Sounds like simplicity for the win. The complex house of cards we currently stand on seems fragile by comparison.
      • tyingq 6y ago
        We also benefit, for that old stuff, from enthusiasts that build cool stuff. Like DosBox, Floppy Emulators, etc. I doubt there are going to be folks nostalgic for the complex mess we have now.
• cube00 6y ago
  The day when code used to produce a paper must also be published can not come soon enough.
  • goalieca 6y ago
    Arguably, data is just as important. Academics hoard their data and try to milk out every paper they can from it. The reward system is based on publishing as many papers as possible rather than just making a meaningful contribution.
    • jgeada 6y ago
      The reward system also prevents dead ends from being identified, publication of approaches that did not lead to the expected results or got nul results, publishing confirmations of prior papers, etc. Basically, the reward system is designed to be easy to measure and administer, but is not actually useful in any way to the advancement of science.
    • belval 6y ago
      Data is much trickier because your data source for medical, education or even just regular businesses don't want the added legal weight of making data freely available. This is obviously a shame, I was working on segmentation of open wounds and most papers include a "we are currently in talks with the hospital to make the data available". If you contact the authors directly they will tell you that their committee blocked it because the information is too sensitive.
      • abathur 6y ago
        It seems like there can be a balance between "the results are unverifiable because no one else can touch the data" and "effectively open-source the dataset"? Something like: "To make it easier to verify the code behind this paper, we've used <accepted standard project/practice> to generate a synthetic dataset with the same fields as the original and included it with the source code. The <data-owning institution> isn't comfortable with publishing the full dataset, but they did agree to provide the same data to groups working on verification studies as long as they're willing to sign a data privacy agreement. Send a query to <blahblahblah> ..."
• dcolkitt 6y ago
  I mean, Python 2->3 alone is gonna kill this challenge for most people.
  • djsumdog 6y ago
    You can always run old Python2 stuff in a Docker container, so long as the dependencies haven't disappeared.
    • jnxx 6y ago
      As long as it does not use some CUDA hardware which is using tensorflow and Numba which is using a version of llvmlite which does not support Python2 any more..... This isn't a theoretical example.
      • therealx 6y ago
        Then you make a vm or whatnot and install all the old versions of everything. I haven't seen an open source project in a while that doesn't have old versions for download easily. Still, annoying if you can't stand that kind of stuff. (people seem to be in two camps: either they hate it or have almost no problem with it)
        • jnxx 6y ago
          Well, where do you download the hardware ? ;-)
        • PeterisP 6y ago
          To clarify, the issue is that the old version of software won't work with the new libraries, and the old libraries won't work with the current GPU models, so you can't run the old code without modification unless you have old hardware as well, and you can't virtualize the GPUs.
    • closeparen 6y ago
      Most of the "requirements.txt" I come across in the real world do not actually lock down all deps to Python 2.7 compatible versions. I've been able to get most of them running again, but it's a long porcess looking through changelogs to find the last 2.7-compatible version of each dependency.
• awkward 6y ago
  Scientific programming is a perfect storm of extremely smart people, with strong abilities to do it themselves, distain for the subject matter, and no direct experience with the price of failing to write portable code. In some circumstances, even parameterizing scripts so that they aren't re-edited with new values for every experiment is an uphill fight, never mind having promotion through environments.
• djsumdog 6y ago
  This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f4002279cff6cf8673208b5cd387/src/SetupModel.cpp https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College for COVID-19 predictions. It has race conditions, seeds the model multiple times, and therefore has totally non-deterministic results[0]. Also, this is the cleaned up repo. The original is not available[1]. A lot of my homework from over 10 years ago still runs (Some require the right Docker container: https://github.com/sumdog/assignments/ https://github.com/sumdog/assignments/). If journals really care about the reproducibility crisis, artifact reviews need to be part of the editorial process. Scientific code needs to have tests, a minimal amount of test coverage, and code/data used really need to be published and run by volunteers/editors in the same way papers are reviewed, even for non-computer science journals. [0] https://lockdownsceptics.org/code-review-of-fergusons-model/ https://lockdownsceptics.org/code-review-of-fergusons-model/ [1] https://github.com/mrc-ide/covid-sim/issues/179 https://github.com/mrc-ide/covid-sim/issues/179
  • arcanus 6y ago
    > Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. You are blaming scientists but speaking from my personal experience as a computational scientist, this exists because there are few structures in place that incentivize strong programming practices. * Funding agencies do not provide support for verification and validation of scientific software (typically) * Few journals require assess code reproducibility and few require public code (few require even public data) * There are few funded studies to reproduce major existing studies Until these structural challenges are addressed, scientists will not have sufficient incentive to change their behavior. > Scientific code needs to have tests, a minimal amount of test coverage, and code/data used really need to be published and run by volunteers/editors in the same way papers are reviewed, even for non-computer science journals. I completely agree.
• lumost 6y ago
  I'm continuously surprised that Code Review isn't a part of the review process for journal acceptance. The majority of academic code for a given paper isn't particularly large - and the benefits are significant.
• userbinator 6y ago
  I still use Windows binaries daily which I wrote and last modified over 20 years ago. I don't expect that to change in the next ten years either.
  • dr-detroit 6y ago
    sounds neat please link the open source repo so we can check it out
• xipho 6y ago
  Yes. 110% attributed to learning about unit-tests and gems/CPAN in grad school. IMO there is a big fallacy about the "just get it to work" approach. Most serious scientific code, i.e. supporting months-years of research, is used and modified a lot. It's also not really one-off, it's a core part of a dissertation, or research program, if it fails- you do. I'd argue that (and I found that), using unit-tests, a deployment strategy, etc. ultimately allowed me to do more, and better science because in the long run I didn't spend as much time figuring out why my code didn't run when I tweaked stuff. This is really liberating stuff. I suspect this is all obvious to those who have gone down that path. Frankly, every reasonably tricky problem benefits from unit-tests as well for another reason. Don't know how to code it, but know the answer? Assert lots of stuff, not just one at a time red-green style. Then code, and see what happens. So powerful for scientific approaches.
  • xorfish 6y ago
    And bugs can have quite big implications: https://smw.ch/article/doi/smw.2020.20336 https://smw.ch/article/doi/smw.2020.20336
• slhck 6y ago
  The two main problems in academia are that a) few researchers have formal training in best practices of software engineering, and that b) time pressure leads to "whatever worked two minutes before submission deadline" becoming what is kept for posteriority. When I started working as a full-time researcher, I had come from working two years in a software shop, only to find people at the research lab having never used VCS, object-oriented programming, etc. Everyone just put together a few text files and Python or MATLAB scripts that output some numbers that went into Excel or gnuplot scripts that got copy-pasted into LaTeX documents with suffixes like "v2_final_modified.tex", shared over Dropbox. Took a long time to establish some coding standards, but even then it took me a while to figure out that that alone didn't help: you need a proper way to lock dependencies, which, at the time, was mostly unknown (think requirements.txt, packrat for R, …).
  • justinmeiners 6y ago
    Don't you think docker, dependencies, unit test frameworks, etc actually increase the need for ongoing maintenance as opposed to spitting out some C files or python scripts which last "forever"?
    • slhck 6y ago
      I don't think so. The source code is the same but there's now metadata that helps in setting up the same environment again, even years later. You still have the original code in case, e.g. Docker is no longer available. For instance, if you just have a Python script importing a statistical library, what version are you going to use? Scipy had a pretty nasty change in one of its statistical functions, changing the outcome of significance tests in our project. Depending on which version you happened to have installed it'd give you a positive or negative result.
      • justinmeiners 6y ago
        It makes sense that having more information is better than less. I would argue that they should use no dependencies to avoid this problem entirely, or download them and include them as source in the project, or at least include a note of which version of a major library they used in a README or comment. I think this is what is often done in practice currently. Perhaps as you are saying, docker is just a stable way to document this stuff formally. But it is a large moving part that assumes a lot of stuff is still on the internet. What if the docker hub image is removed or dramatically changed? What if that OS package manager no longer exists? It just doesn't seem like our software is getting more longevity, but less. I don't know why we would bring that extra complexity to academic research if the goal is longevity.
• Sebb767 6y ago
  As someone who worked with bits of scientific code: Does the code you write right now work on another machine might be the more appropriate challenge. If seen a lot of hardcoded paths, unmentioned dependencies and monkey-patched libraries downloaded from somewhere; just getting the new code to work is hard enough. And let's not even begin to talk about versioning or magic numbers. Similar to other comments I don't mean to fault scientists for that - their job is not coding and some of the dependencies come from earlier papers or proprietary cluster setups and are therefore hard to avoid - but the situation is not good.
  • TheJoeMan 6y ago
    I emailed an author of a 5 year old paper and they said they had lost their original MATLAB code, certainly brings into question their paper.
    • James_Henry 6y ago
      Definitely makes you question it more. Does the paper not explain the contents of the MATLAB code? That's all that is usually needed for reproducibility. You should be able to get the same results no matter who writes the code to do what is explained in their methods. Of course, I have no idea about the paper you're talking about and just want to say that reproducibility isn't dependent on releasing code. There could even be a case were it's better if someone reproduces a result without having been biased by someone else's code.
  • BeetleB 6y ago
    > their job is not coding To me, that's like a theoretical physicist saying "My job is not to do mathematics" when asked for a derivation of a formula he put in the paper. Or an experimental physicist saying "My job is not mechanical engineering" when asked for details of their lab equipment (almost all of which is typically custom built for the experiment).
    • Sebb767 6y ago
      On one hand, yes. But on the other hand, reuseable code, dependency management, linting, portability etc are not that easy problems and something junior developers tend to struggle with (and its not like that problem never pops up for seniors, either). I really can't fault non-compsci scientist for not handling that problem well. Of course, part of it (like publishing the relevant code) is far easier and should be done, but some aspects are really hard. IMO the incentive problem in science (basically number of papers and new results is what counts) also plays into this, as investing tons of time in your code gives you hardly any reward.
• hprotagonist 6y ago
  Mine does. I swear 40% of the idiocy of science code is because people fundamentally don’t understand how file paths work. Stop hardcoding paths to data and the world gets better by an order.
• akerro 6y ago
  Code written in Oak still works in Java 14. You can still write `public abstract interface BlaBla{}` and it still works. If it doesn't work (due to reflection safety changes in Java9), it sill surely compile with newer compiler. Another thing, are tools used to compile still available? I tried to compile my BCS Android+native OpenCV project and failed quickly. Gradle removed some plugin for native code integration, another plugin was no longer maintained, it had internal check for gradle version and it said "I'm designed to work with gradle >= 1.x < 3.x" and just refused to run under 6.x ... I would have to fork that plugin, make it work with newer Gradle or find replacement. I was obviously too lazy and stopped working on that project before I even started. I'm sure if (would) put more effort into making the build process reproducible, it would work effortlessly, but I didn't care at the point. I wrote it using beta release of OpenCV that's also no longer maintained, because there are better, faster official alternatives available.
  • therealx 6y ago
    Or use the old version of Gradle? It sounds like creating a vm/container/whatever with the old versions of everything is the fastest path, although I understand not wanting to do it after some point.
    • akerro 6y ago
      >Or use the old version of Gradle? Intellij / Android Studio are nagging me to update gradle
• drummer 6y ago
  This is why backwards compatibility is important. Many people have a problem when this is raised as a primary concern and goal of the C and C++ languages, but it is a must have feature.
• vikramkr 6y ago
  In addition - does your ten year old protocol still work? Do your 10 year old results replicate? This isn't isolates to just programming - making robust and reproducible tools, code, equiptment, protocols, and results is undervalued across all areas of research, leading to situations where protocols published weren't robust so a change in reagent supplier leads to failure, or to protocols so dependent on weird local or unreported environmental conditions or random extra steps that attempting to replicate them leaves you nowhere. Robustness needs to be improved in general.
• goalieca 6y ago
  > Today, researchers can use Docker containers (see also ref. 7) and Conda virtual environments (see also ref. 8) to package computational environments for reuse. Docker is also flawed. You can perfectly reproduce it today but what about in 10 years. I can barely go back to our previous release for some dockerfiles.
  • Sulfolobus 6y ago
    Similarly, conda envs can break in weeks due to package changes. Even if you remove build versioning and all transitive dependencies from your env (making it less reproducible...) they will break pretty damn quick.
  • jnxx 6y ago
    Guix is arguably better.
• dekhn 6y ago
  I wrote a C++ implementation of the AMBER force field in 2003. Still have the source code with its original modification times. Let's see: /usr/bin/g++ -I/home/dek/sw/rh9/gsl-1.3/include -c -o NBEnergy.o NBEnergy.cpp NBEnergy.cpp: In member function ‘virtual double NBEnergy::Calculate(Coordinates&, std::vector<Force*>)’: NBEnergy.cpp:20:68: error: no matching function for call to ‘find(std::vector<atom*>::const_iterator, std::vector<atom*>::const_iterator, const atom*&)’ 20 | if (std::find(at1->Excluded.begin(), at1->Excluded.end(), at2) != at1->Excluded.end()) { | ^ In file included from /usr/include/c++/9/bits/locale_facets.h:48, from /usr/include/c++/9/bits/basic_ios.h:37, from /usr/include/c++/9/ios:44, from /usr/include/c++/9/ostream:38, from GeneralParameters.h:6, from NBEnergy.h:6, from NBEnergy.cpp:1: /usr/include/c++/9/bits/streambuf_iterator.h:373:5: note: candidate: ‘template<class _CharT2> typename __gnu_cxx::__enable_if<std::__is_char<_CharT2>::__value, std::istreambuf_iterator<_CharT> >::__type std::find(std::istreambuf_iterator<_CharT>, std::istreambuf_iterator<_CharT>, const _CharT2&)’ 373 | find(istreambuf_iterator<_CharT> __first, | ^~~~ /usr/include/c++/9/bits/streambuf_iterator.h:373:5: note: template argument deduction/substitution failed: NBEnergy.cpp:20:68: note: ‘__gnu_cxx::__normal_iterator<atom* const*, std::vector<atom*> >’ is not derived from ‘std::istreambuf_iterator<_CharT>’ 20 | if (std::find(at1->Excluded.begin(), at1->Excluded.end(), at2) != at1->Excluded.end()) { | ^ make: *** [<builtin>: NBEnergy.o] Error 1 I still have a hardcoded reference to RedHat 9 apparently. But the only error has to do with an iterator, so clearly, something in C++ changed. Looks like a 1-2 line change.
  • josefx 6y ago
    You probably didn't include the algorithm header that defines find directly and it stopped compiling once the standard library maintainers cleaned up their own includes. The iostreams headers you include define their own stream iterator specific overload of find and that doesn't match.
    • dekhn 6y ago
• nanddalal 6y ago
  GitHub offers a free tier for GitHub actions with 2,000 Actions minutes/month [1]. This could be useful: 1. write some unit tests which don't use too much compute resources (so you can stick to the free tier) 2. package your code into a docker where the tests can be run 3. wire up the docker with tests to GitHub Actions This way now you have continuous testing and can make sure your codes keep running. References: [1] https://github.com/pricing https://github.com/pricing
  • MattGaiser 6y ago
    Even if it broke, who would go back and fix it? I do not see that happening, especially with complex library bugs.
  • jnxx 6y ago
    > package your code into a docker docker is not a general solution for this. What is needed is a way to re-generate everything from source and from scratch.
• myself248 6y ago
  Plenty of actual professional programmers can't manage this, how is it a fair standard to hold scientists to, when the code is just one of the many tools they're trying to use to get their real job done? I think moving away from the cesspool of imported remote libraries that update at random times and can vanish off the internet without warning, would help a lot of both cases.
  • rudolph9 6y ago
    Professional programmers should adopt package manager that focus on reproducibility like Guix and Nix and make them accessible enough for non programmers to use. Neither of these are perfect but in my experience they are worlds better than apk, Dockerfiles, and many other commonly used solutions. http://guix.gnu.org/ http://guix.gnu.org/ https://nixos.org/ https://nixos.org/
    • konjin 6y ago
      I've had the pleasure of setting up a guix system in production. The next guy to come along didn't understand it and threw it all away.
      • rudolph9 6y ago
        That’s very unfortunate.
  • proverbialbunny 6y ago
    >Plenty of actual professional programmers can't manage this, how is it a fair standard to hold scientists to That's a good point. On a tangential note, prototype code tends to be at a higher level than production code, so there is a higher chance 10 year old code will continue to run on the scientist side, as long as the libraries imported haven't vanished.
  • minkzilla 6y ago
    I think we have to hold scientists to higher standards for code quality because it has a direct impact on the findings of their results. How many off by one or other subtle errors that are found later in testing have most software engineers written in their career? Is it fine to just say eh, scientific results can be off by one because the standards should be lower?
• westurner 6y ago
  "Ten Simple Rules for Reproducible Computational Research" http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fjournal.pcbi.1003285 http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fj... : > Rule 1: For Every Result, Keep Track of How It Was Produced > Rule 2: Avoid Manual Data Manipulation Steps > Rule 3: Archive the Exact Versions of All External Programs Used > Rule 4: Version Control All Custom Scripts > Rule 5: Record All Intermediate Results, When Possible in Standardized Formats > Rule 6: For Analyses That Include Randomness, Note Underlying Random Seeds > Rule 7: Always Store Raw Data behind Plots > Rule 8: Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected > Rule 9: Connect Textual Statements to Underlying Results > Rule 10: Provide Public Access to Scripts, Runs, and Results ... You can get a free DOI for and archive a tag of a Git repo with FigShare or Zenodo. ... re: [Conda and] Docker container images https://news.ycombinator.com/item?id=24226604 https://news.ycombinator.com/item?id=24226604 : > - repo2docker (and thus BinderHub) can build an up-to-date container from requirements.txt, environment.yml, install.R, postBuild and any of the other dependency specification formats supported by REES: Reproducible Execution Environment Standard; which may be helpful as Docker Hub images will soon be deleted if they're not retrieved at least once every 6 months (possibly with a GitHub Actions cron task) BinderHub builds a container with the specified versions of software and installs a current version of Jupyter Notebook with repo2docker, and then launches an instance of that container in a cloud. “Ten Simple Rules for Creating a Good Data Management Plan” http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004525 http://journals.plos.org/ploscompbiol/article?id=10.1371/jou... : > Rule 6: Present a Sound Data Storage and Preservation Strategy > Rule 8: Describe How the Data Will Be Disseminated ... DVC: https://github.com/iterative/dvc https://github.com/iterative/dvc > Data Version Control or DVC is an open-source tool for data science and machine learning projects. Key features: > - Simple command line Git-like experience. Does not require installing and maintaining any databases. Does not depend on any proprietary online services. Management and versioning of datasets and machine learning models. Data is saved in S3, Google cloud, Azure, Alibaba cloud, SSH server, HDFS, or even local HDD RAID. > - Makes projects reproducible and shareable; helping to answer questions about how a model was built. There are a number of great solutions for storing and sharing datasets. ... "#LinkedReproducibility"
  • 6y ago
• dekhn 6y ago
  The longest-running code I wrote as a scientist was a sandwich ordering system. I worked for a computer graphics group at UCSF and while taking a year off from grad school while my simulations ran on a supercomputer, and we had a weekly group meeting where everybody ordered sandwiches from a local deli. It was 2000, so I wrote a cgi-bin in Python (2?) with a MySQL backend. The menu was stored in MySQL, as were the orders. I occasionally check back to see if it's still running, and it is- a few code changes to port to Python3, a data update since they changed vendors, and a mysql update or two as well. It's not much but at least it was honest work.
  • unnouinceput 6y ago
    My Tetris, first "serious" game I made, written in 1996, can play just as well in DosBox.
• bluetwo 6y ago
  18-year old code still runs. And generates revenue.
• wdwvt1 6y ago
  An excellent article full of good suggestions. I appreciated that it's less certain of the Best Practices TM than many comments on this subject. I am curious how the goals/techniques for reproducibility change with the percentage of software/computational work that a scientific project contains. It feels like as the percentage of a paper's ultimate conclusions that are computationally derived increases, the importance of strict "the tests pass and the numerical results are identical" reproducibility also increases. Most of my projects are mixed wet-lab/dry-lab - a fair amount of custom code is required, but it's usually less than 50% of the work. When I'm relying on other papers that have a similar mix of things, I'm often not interested if the continuous integration tests of their code pass. I am more interested in understanding well the specific steps they take computationally and in a sensitivity analysis of their computational portion (if you slightly alter your binning threshold do you still get that fantastic clustering?). I believe this is because in my field (microbiology), computational tools can guide, but physical reality and demonstrated biology are the only robust evidence of a phenomenon/mechanism/etc. For most research I do not demand tests of all the analytical pieces they are relying on (was their incubator actually set to 37C? was the pH of the media +- 0.2? etc) - I trust they've done good science. Why would I demand their code meet a higher standard?
• stillsut 6y ago
  It's not academia but Kaggle that's really been on the forefront of building portable and reproducible computational pipelines. The real key is incentives and there are two that standout to me: - Incentive to get others to "star" and fork your code makes the coder compete to not only have an accurate result, but also prioritize producing code/notebooks that are digestible and instructive. That includes liberal commenting/markup, idiomatic syntax and patterns, diagnostic figures, and the use of modern and standard libraries. - There is an incentive to move with the community on best practices for the libraries while still allowing experimental libraries. Traditionally, there is the incentive of inertia: e.g. "I always do my modelling in Lisp, and I won't change because then I'd be less productive". But with kaggle, to learn from the insights and advances of others, you need to have an ability to work with the developing common toolset. In academia, if these incentives were given weight on par with publication and citation then we'd see the tools and practices fall into place.
• jnxx 6y ago
  Very related to this, see also Hinsens blog post: http://blog.khinsen.net/posts/2017/11/16/a-plea-for-stability-in-the-scipy-ecosystem/ http://blog.khinsen.net/posts/2017/11/16/a-plea-for-stabilit... I think that GNU Guix is extremely well-suited to improve this situation. Also, one could think this is an academic problem, in the sense of am otherwise unimportant niche problem. It really isn't, it is just like in many other topics that academics get confronted first with this issue. I am sure that in many medium or large companies there are some Visual Basic or Excel code bases which are important but could turn out extremely hard to reproduce. This issue will only get more burning with today's fast-moving ecosystems where backward-compatibility is more a moral ideal than an enforced requirement. It is well known that ransomware can wipe-out businesses if critical business data is lost. But more and more businesses and organizations also have critical, and non-standard, software.
  • neuromantik8086 6y ago
    Guix is one of several solutions that has been touted as a solution. Another one that is quite popular in HPC circles is Spack (https://spack.readthedocs.io/en/latest/ https://spack.readthedocs.io/en/latest/). At my institute, we actually tried out Spack for a little bit, but consistently felt like it was implemented more as a research project rather than something that was production-level and maintainable. In large part, this was due to the dependency resolver, which attempts to tackle some very interesting CS problems I gather (although this is a bit above me at the moment; these problems are discussed in detail at https://extremecomputingtraining.anl.gov//files/2018/08/ATPESC_2018_Dinner_Talk_5_8-2_Gamblin-Spack.pdf https://extremecomputingtraining.anl.gov//files/2018/08/ATPE...), but which produces radically different dependency graphs when invoked with the same command across different versions of Spack. I've since come to regard Spack as the kind of package manager that science deserves, with conda being the more pragmatic / maintainable package manager that we get instead . Spack/Guix/nix are the best solution in theory, but they come with a host of other problems that made them less desirable.
    • jnxx 6y ago
      > Spack/Guix/nix are the best solution in theory, but they come with a host of other problems that made them less desirable. I would be quite interested to learn more what these problems are, in your experience. I've only tried Guix (on top of Debian and Arch) and while it is definitively more resource-hungry (especially in terms of disk space), I don't percive it as impractical.
• hpcjoe 6y ago
  Short answer: Yes, my 30 year old Fortran code runs (with a few minor edits between f77 and modern fortran), as did my ancient Perl codes. Watching the density functional theory based molecular dynamics zip along at ~2 seconds per time step on my 2 year old laptop, versus the roughly 6k seconds per time step on an old Sun machine back in 1991. I remember the same code getting down to 60 seconds per time step on my desktop R8k machine in the late 90s. Whats been really awesome about that has been the fact that I've written some binary data files on big endian machines in the early 90s, and re-read them on the laptop (little endian) adding a single compiler switch. Perl code that worked with big XML file input in the mid 2000s continues to work, though I've largely abandoned using XML for data interchange. C code I wrote in the mid 90s compiled, albeit with errors that needed to be corrected. C++ code was less forgiving. Over the past 4 months, I had to forward port a code from Boost 1.41 to Boost 1.65. Enough changes over 9 years (code was from 2011) that it presented a problem. So I had to follow the changes in the API and fix it. I am quite thankful I've avoided the various fads in platforms and languages over the years. Keep inputs in simple textual format that can be trivially parsed.
  • atrettel 6y ago
    > Whats been really awesome about that has been the fact that I've written some binary data files on big endian machines in the early 90s, and re-read them on the laptop (little endian) adding a single compiler switch. I want to second the idea of just dumping your floating point data as binary. It's basically the CSV of HPC data. It doesn't require any libraries, which could break or change, and even if the endianness changes you can still read it decades later. I've been writing a computational fluid dynamics code recently and decided to only write binary output for those reasons. I'm not convinced of the long-term stability of other formats. I've seen colleagues struggle to read data in proprietary formats even a few years after creating it. Binary is just simple and avoids all of that. Anybody can read it if needed.
    • petschge 6y ago
      Counter argument: Binary dumps are horrible because usually the documentation that allows you to read the data is missing. Using a self-documenting format such as HDF5 is far superior. It will tell you of the bit are floating point numbers in single or double precision, which endianess and what the layout of the 3d array was. (No surprise that HDF was invented for the Voyager mission where they had to ensure readability of the data for half a century).
• adornedCupcake 6y ago
  "Python 2.7 puts “at our disposal an advanced programming language that is guaranteed not to evolve anymore”, Rougier writes1." Oh no. That's not at all what was intended. Regarding my own research: I'm doing theoretical biophysics. Often I do simulations. If conda stays stable enough, my code should be reproducible. There's however some external binaries(like lammps) I did not turn into a conda package yet. There's no official package that fits my use-case in conda since compilation is fine-grained to each user's needs.
  • rekado 6y ago
    I added different variants of lammps to a Guix channel we maintain at our institute: https://github.com/BIMSBbioinfo/guix-bimsb/blob/master/bimsb/packages/staging.scm#L791 https://github.com/BIMSBbioinfo/guix-bimsb/blob/master/bimsb... Thankfully, Guix makes it easy to take an existing package definition and create an altered variant of it.
• tenYearOldCode 6y ago
  Yes it does! 10 PRINT "HELP" 20 GOTO 10
• driverdan 6y ago
  This is a challenge with many types of code. Earlier this year it took me a weekend to get a 7 year old Rails project running again. It's a simple project but the packages it used had old system dependencies that were no longer available. I ended up having to upgrade a lot of things, including code, just to get it running again.
  • therealx 6y ago
    I ran into this too. Rails has changed a lot in 7 years, even if you don't see it. My friend wanted to learn and somehow found the original demo/getting started page and was frustrated.
• noisy_boy 6y ago
  Not a scientist but 13+ year old Perl code I wrote is still running (based on my catchup chats with ex-colleagues) to generate MIS reports.
• _wldu 6y ago
  IMO, this is why ISO standard programming languages are so important and will be around forever. One can always compile with --std=c++11 (or whatever) and be certain it will work.
  • Kenji 6y ago
    Hahaha you would be surprised. Compiling complex C++ projects is incredibly difficult.
• magv 6y ago
  An interesting concern is that there often is no single piece of code that has produced the results of a given paper. Often it is a mixture of different (and evolving) versions of different scripts and programs, with manual steps in between. Often one starts the calculation with one version of the code, identifies edge cases where it is slow or inaccurate, develops it further while the calculations are running, does the next step (or re-does a previous one) with the new version, possibly modifying intermediate results manually to fit the structure of the new code, and so on -- the process it interactive, and not trivially repeatable. So the set of code one has at the end is not the code the results were obtained with: it is just the code with the latest edge case fixed. Is it able to reproduce the parts of the results that were obtained before it was written? One hopes so, but given that advanced research may take months of computer time and machines with high memory/disk/CPU/GPU/network speed requirements only available in a given lab -- it is not at all easy to verify.
  • i-am-curious 6y ago
    And any such "research" should go in the bin. Reproducibility of final results a me d their review is key.
    • James_Henry 6y ago
      No, you should publish this research and be clear with how it all worked out and someone will reproduce it in their own way. Reproducibility isn't usually about having a button to press that magically gives you the researchers' results. It's also not always a set of perfect instructions. More often it is a documentation of what happen and what was observed as the researcher's believe is important to the understanding of the research questions. Sometimes we don't know what's important to document so we try to document as much as possible. This isn't always practical and sometimes it is obviously unnecessary.
  • vharuck 6y ago
    >the process it interactive, and not trivially repeatable. The kind of interaction you're describing should be frowned upon. It requires the audience to trust the manual data edits are no different than rerunning the analysis. But the researcher should just rerun the analysis. Also, mixing old and new results is a common problem in manually updated papers. It can be avoided by using reproducible research tools like R Markdown.
• jonathanstrange 6y ago
  I think it's unfair to expect from anyone to maintain code forever when the code rot is completely beyond your control, let alone to expect this from scientists who have better things to do. Anything with a GUI is bound to self-destruct, for example, and it's not the programmer's fault. Blame the OS makers and framework/3rd party library suppliers. The damage can be limited by choosing a programming language that provides good long compatibility. Languages like ANSI C, Ada, CommonLisp, and Fortran fit the bill. There are many more. Heck, you could use Chipmunk Basic. Anything fancy and trendy will stop working soon, though, sometimes even within a year.
  • jnxx 6y ago
    > CommonLisp Common Lisp has fantastic long-term stability. I think that deserves more recognition, as Common Lisp is often almost as fast as C, but is (by default) not riddled with undefined behavior. It would be superb if Rust could take C's space in computational science and libraries.
    • deleted 6y ago
      [deleted]
• Gatsky 6y ago
  I think code is remarkably persistent in the scheme of things. Try reproducing a wet lab experimental technique from 5 years ago.
• fourseventy 6y ago
  Forget 10 year old code. Try to get your 2 year old javascript + webpack + react set up running...
• m3kw9 6y ago
  It always runs if you used the same computer with the same environment you last time ran it. So yes.
• shadowgovt 6y ago
  I'm not sure how interesting the question is, given how few software engineers outside academic sciences have 10-year-old code that still runs (unless they've maintained a dedicated hardware platform for it without regular software updates).
• emerged 6y ago
  Any scientist with good foresight would've implemented their code in 6502 for the NES. The emulators are nearly flawless and will probably be around until the end of time.
  • yjftsjthsd-h 6y ago
    I once had a thought, that if I wanted to write something that would last forever and run anywhere, I should write it to target DOS, and make sure to test it on FreeDOS in a VM and on DOSBox. That way it would run on a stable ABI with loads of emulators, and via DOSBox it will happily run on all modern desktop OSs (and some non-desktops; IIRC there's at least an Android port).
• rougier 6y ago
  For those interested, the results of the challenge are published here: https://rescience.github.io/read/ https://rescience.github.io/read/ (volume 6, issue 1).
• rudolph9 6y ago
  This was in the Guix-science mail list today > Hello! In an article entitled “Challenge to scientists: does your ten-year-old code still run?”, Nature reports on the Ten Years Reproducibility Challenge organized by ReScience C, led by Nicolas P. Rougier and Konrad Hinsen: https://www.nature.com/articles/d41586-020-02462-7 It briefly mentions Guix as well as the many obstacles that people encountered and solutions they found, including using Software Heritage and floppy disks. :-) You can read the papers (and reviews!) at: https://rescience.github.io/read/#issue-1-ten-years-reproducibility-challenge Ludo’.
• neuromantik8086 6y ago
  Just as a quick bit of context here, Konrad Hinsen has a specific agenda that he is trying to push with this challenge. It's not clear from this summary article, but if you look at the original abstract soliciting entries for the challenge (https://www.nature.com/articles/d41586-019-03296-8 https://www.nature.com/articles/d41586-019-03296-8), it's a bit clearer that Hinsen is using this to challenge the technical merits of Common Workflow Language (https://www.commonwl.org/; https://www.commonwl.org/; currently used in bioinformatics by the Broad Institute via the Cromwell workflow manager). Hinsen has created his own DSL, Leibniz (https://github.com/khinsen/leibniz https://github.com/khinsen/leibniz ; http://dirac.cnrs-orleans.fr/~hinsen/leibniz-20161124.pdf http://dirac.cnrs-orleans.fr/~hinsen/leibniz-20161124.pdf), which he believes is a better alternative to Common Workflow Language. This reproducibility challenge is in support of this agenda in particular, which is worth keeping in mind; it is not an unbiased thought experiment.
  • jnxx 6y ago
    Konrad Hinsen is an expert in molecular bioinformatics and also has significantly contributed to Numerical Python, for example, and has extensively published around the topic of reproducible science and algorithms - see his blog. The fact that he might favor different solutions from you does not mean that he is pushing some kind of hidden agenda. If you think that Common Workflow Language is a better solution, you are free to explain in a blog why you think this. Are you saying that the reproductive challenge poses a difficulty to Common Workflow Language? If this is so, would that not rather support Hinsen's point - without implying that what he suggests is already a perfect solution?
    • neuromantik8086 6y ago
      I never said that Konrad Hinsen's agenda was hidden; in fact, it's not at all hidden (which is why I linked the abstract). It's just that this context isn't at all clear in the Nature write-up, and it's relevant to take into account. I haven't taken the time to seriously contemplate the merits of CWL vs Leibniz, although my gut instinct is that we don't really need another domain-specific language for science given the profusion of such languages that already exist (Mathematica, Maple, R, MATLAB, etc). That's the extent of my bias, but again, it's a gut instinct and not a comprehensive well-reasoned argument against Leibniz.
• suyjuris 6y ago
  I wrote a tool to visualise algorithms for binary decision diagrams [1], also in an academic context, where the problem was basically the same: Does the code still run in ten years? In particular, the assumption is that I will not be around then, and no one will have any amount of time to spend on maintenance. In the end, I chose to write it in C++ with minimal dependencies (only X11, OpenGL and stb_truetype.h), with custom GUI, and packed all resources into a single executable. A lot of effort, but if it causes the application to survive 5x as long then it is probably worth spending twice the time. [1] https://github.com/suyjuris/obst https://github.com/suyjuris/obst
• yummypaint 6y ago
  Not to disagree with any points in the article, but i would point out that the sciences also have cases of very old code being maintained and used in production successfully. For example we still use a kinematics code written in fortran over half a century ago. In practice parts of it get reimplemented in newer projects, but the original still sees use.
• bloak 6y ago
  I had an 18-year-old Python script. But it didn't work! And I couldn't make it work! Fortunately I had an even older version of the code in Perl, which did work after some very minor changes. This wasn't scientific code. It was some snarly private code for generating the index for a book and I didn't look at it between one edition and the next. I hope I don't have to fix it again in another 18 years. Applying some version of the "doomsday argument", Perl 5 might be a good choice if you're writing something now that you want to work (without a great tower of VMs) in 10 or 20 years' time. C would only be a reasonable choice if you have a way of checking that your program does not cause any undefined behaviour. A C program that causes undefined behaviour can quite easily stop working with a newer version of the compiler.
• proverbialbunny 6y ago
  This seems like a fluff piece because: 1) Prototype code scientists write tends to be written at a high level, so barring imported libraries not up and disappearing, there is a high chance that code written by scientists will run 10 years later. There is a higher chance it will run than production code written at a lower level. 2) The article dives into documentation but scientists code in the Literate Programming Paradigm[0] where the idea is you're writing a book and the code is used as examples to support what you're presenting. Of course scientists write documentation. Presenting findings is a primary goal. 3) Comments here have mentioned unit testing. Some of you may scoff at this but when prototyping, every time you run your code, the output from it teaches you something, and that turns into an iterative feedback loop, so every time you learn something you want to change the code. Unit tests are not super helpful when you're changing what the code should be doing every time you run it. Unit tests are better once the model has been solidified and is being productionized. Having a lack of unit testing does not make 10 year old prototype code harder to run. [0] https://en.wikipedia.org/wiki/Literate_programming https://en.wikipedia.org/wiki/Literate_programming
  • comicjk 6y ago
    > scientists code in the Literate Programming Paradigm I wish. In my career as a computational scientist I have never seen this in practice, either in academia or industry. On unit testing, I half agree. Most unit tests get quickly thrown out as the code changes, so it's a depressing way to write research code. But tests absolutely help someone trying to run old code - they show what parts still work and how to use them.
    • proverbialbunny 6y ago
      It's even more common today. eg, Jupyter Lab or Jupyter Notebooks.
      • konjin 6y ago
        Notebooks are to Literate Programming what Word is to TeX.
• JoeAltmaier 6y ago
  Strangely, they were running (some of ) the code on old hardware. That's hardly a useful case, and much easier than 'resurrecting' the code for modern reuse.
  • therealx 6y ago
    Something with non-standard asm?
    • JoeAltmaier 6y ago
      That sounds like a big issue. And certainly part of getting 10-year-old code resurrected.
• noobermin 6y ago
  Does code from ten years ago run ever? Try running something on that runs on Python 2 on the current python interpreter today.
  • jnxx 6y ago
    Python is an extremely bad example. Try twenty years old Common Lisp code. Or Fortran.
    • noobermin 6y ago
      What are we doing then? We can choose bad scientific code but use "good examples" for other types of code? Seems convenient.
• roberto 6y ago
  For my first scientific article, in 2007, I created a Subversion repo with a Makefile. Running `make` would recreate the whole paper: downloading data, running analyses, creating pictures (color or BW, depending on an environment flag) and generating the PDF. I'm going to try to find the repo and see if it still works.
  • O_H_E 6y ago
    Wow, nice. I will be waiting :D
• closeparen 6y ago
  Almost certainly not, because it would have been written in Python 2.
• ris 6y ago
  A well-written Nix package should be buildable at any point in the future, producing near-identical results. This is why I sometimes publish Nix packages for obscure & hard to build pieces of software that I'm not likely to maintain - because it's like rescuing a snapshot of them from oblivion.
• garden_hermit 6y ago
  I favor open code, but like everything, there are issues. For example, the EPA years ago required that research can only inform policy when data is open; open data, however, takes a lot of effort to document and provide. Companies, however, with vested interest in EPA policy can easily produce open (and often very biased) data. Requirements for open code can lead to similar issues—what happens when a government agency rejects the outcome of a supercomputer simulation because the code wasn't documented well enough? What happens when those with vested interests are the ones best able to produce scientific code? Scientists already wear many hats. Any shift in policy and norms needs to consider that they have limited time, a fact that can have far-reaching consequences.
• wenc 6y ago
  The easiest way to preserve code for posterity is the wrap up the runtime environment in a VM. I can boot up a VM from 15 years ago (when I was in grad school) and it will run. When you're writing code for science, preserving code for posterity is rarely a priority. Your priority is to iterate quickly because the goal is scientific results, not code. (this is in fact, correct prioritization. Under most circumstances, though not all, most grad students who try to write pristine code find themselves progressing more slowly than those who don't.)
• CoffeeDregs 6y ago
  Would be a super-useful to have a sciencecode.com service which is a long-term CI system for scientific code and its required artifacts. Journals could include references to sciencecode.com/xyz and sciencecode.com/abc could be derived from sciencecode.com/xyz. Given Github Actions and Forks, the only thing holding this back is scientists doing it (and, possibly, the HN community helping). And I get that it's not fun to have your code publicly critiqued but it's also not fun to live lives based on (medical, epidemiological) unpublished, unaudited, unverified code... EDIT: hell, just post a "HELP HN: science code @ github.com/someone/project" and I'd be surprised if you weren't overwhelmed with offers of help.
• bobcostas55 6y ago
  How do you do reproducible builds in R? It seems like a huge PITA to specify versions of R and especially the packages used...
• deleted 6y ago
  [deleted]
• jpeloquin 6y ago
  The point of being able to run ten-year-old code is the ability to replay an analysis (exact replication). This allows an analysis to be verified after the fact, which increases trust and helps figure out what happened when contradictions appear between experiments. However, if the original work involved physical experimentation or any non-automated steps (as is the case for most science) the ability to run the original code provides only partial replication. Overall the ability to re-run old code is a fairly low priority. From the perspective of someone who primarily uses computers as a tool to facilitate research, the priority list is closer to: 1. Retain documentation of what was meant to happen. Objectives, experimental design, experimental & analysis protocols, relevant background, etc. 2. Retain documentation of what actually happened, usually in terms of noting deviations from the protocol. This is the purpose of a lab notebook. Pen & paper excels here. 3. Retain raw data files. 4. Retain files produced in the course of analysis. 5. Retain custom source code. 6. Version control all the above. 7. Make everything run in the correct order with one command (i.e, full automation). Only once all the above is achieved would it be worth ensuring that the software used in the analysis can be re-run in 10 years. Solving the "packaging problem" in a typical scientific context (multiple languages, multiple OSes, commercial software, mostly short scripts) is complex. When the outcome of an analysis is suspect, the easiest and most robust approach is to check the analysis by redoing it from scratch. This takes less time than trying to ensure every analysis will run on demand even as the computing ecosystem changes out from under it. Most of the time spent writing analysis code is deciding what the code should do, not actually writing the code. There is generally very little code because few people were involved, and they probably weren't programmers. So redoing the work from scratch is generally pretty easy, especially for anyone with the skill to routinely produce fully reproducible computational environments.
• Ericson2314 6y ago
  Glad to see many mentions of Nix in this thread! I wonder if Nix and Guix should standardize the derivation format both share to kick that off as the agreed-upon "thin waste" other projects and the the academy can standardize around.
  • rekado 6y ago
    The derivation format is little more than a compilation artifact (a low-level representation of a build), and I think standardizing on it would not be as useful as it may seem.
    • Ericson2314 6y ago
      Exactly, it's mostly boring, just like IP packets in isolation are boring. Great thing to standardize. Also, we're working on making them a bit less boring :). I've been working on adding hashing schemes compatible with IPFS's IPLD. Also, we can make derivations that produce derivations, making a sort of recursive Nix that encourages more up-front planning than crude "nix-build inside derivations. I hope Guix would want some of this stuff too.
• pruthvishetty 6y ago
  I read it as does your ten-year-old still run code, and was thinking if this was a challenge for scientists to have their kids do better things than coding.
• pvaldes 6y ago
  Yes, it still run, of course.
• fizzled 6y ago
  Yep. I wrote a netlist analyzer in Perl that provides statistics... in 1997. It is still part of a regression suite because it is very small, very fast and callable through the command line without loading hundreds of megabytes of libraries (unlike foundation tools). I reconnected with a peer on LinkedIn who still works at the company and joked that he still sees my sill script name in verification flows. The only changes I made to in 20+ years it was moving to PERL 5.61 so that I could parse files >1GB, but it has been maintained and kept to standard practices.
• scipute68 6y ago
  As the systems architect and infra programmer for a scientific startup I'll simply chime in on the production != scientific conversation. When you don't hold your modeling code to the minimal production standard where it counts (documentation, comments, debug) it _will_ cause your evolving team hardship. When that same code goes into production for a startup (as it could/should) you will be causing everyone long nights and 80 hour weeks.
• uberdru 6y ago
  Sure it does. On a 10-year old machine.
• alexeiz 6y ago
  Pfff. Does my 3 month old code still run? Uh, nope. And I don't remember what it was supposed to do!
• daly 6y ago
  Axiom is a computer algebra system written in the 1970s-80s. It still runs (and is open source).
• majewsky 6y ago
  CMake Error at /usr/share/cmake-3.18/Modules/FindQt4.cmake:1314 (message): Found unsuitable Qt version "5.15.0" from /usr/bin/qmake, this code requires Qt 4.x Well fuck.
• snowwrestler 6y ago
  The gold standard for a scientific finding is not whether an particular experiment can be repeated, it is whether a different experiment can confirm the finding. The idea is that you have learned something about how the universe works. Which means that the details of your experiment should not change what you find... assuming it's a true finding. Concerns about software quality in science are primarily about avoiding experimental error at the time of publication, not the durability of the results. If you did the experiment correctly, it doesn't matter if your code can run 10 years later. Someone else can run their own experiment, write their own code, and find the same thing you did. And if you did the experiment incorrectly, it also doesn't matter if you can run your code 10 years later; running wrong code a decade later does not tell you what the right answer is. Again--conducting new research to explore the same phenomenon would be better. When it comes to hardware, we get this. Could you pick up a PCR machine that's been sitting in a basement for 10 years and get it running to confirm a finding from a decade ago? The real question is, why would you bother? There are plenty of new PCR machines available today, that work even better. And it's the same for custom hardware. We use all sorts of different telescopes to look at Jupiter. Unless the telescope is broken, it looks the same in all of them. Software is also a tool for scientific observation and experimentation. Like a telescope, the thing that really matters is whether it gives a clear view of nature at the time we look through it.
  • nextaccountic 6y ago
    > running wrong code a decade later does not tell you what the right answer is. It can tell, however, exactly where the error lies (if the error is in software at all). Like a math teacher that can circle where the student made a mistake in an exam.
  • ISL 6y ago
    Reproducibility is about understanding the result. It is the modern version of "showing your work". One of the unsung and wonderful properties of reproducible workflows is the fact that it can allow science to be salvaged from an analysis that contains an error. If I had made an error in my thesis data analysis (and I did, pre-graduation), the error can be corrected and the analysis re-run. This works even if the authors are dead (which I am not :) ). Reproducibility abstracts the analysis from data in a rigorous (and hopefully in the future, sustainable) fashion.
• cabaalis 6y ago
  I visited my first employer recently (a local government) and found that the first MySQL/PHP database I created, an internal app, had been in continuous use for nearly 18 years.
• modeless 6y ago
  Let's not criticize people who release their code. Let's criticize the people who don't release their code instead. We don't need more barriers to releasing code. I'd much rather fix someone's broken build than reimplement a whole research paper from scratch without the relevant details that seem to always be accidentally omitted from the paper.
• tanilama 6y ago
  If you wrap you code into Docker, I would say...probably.
• mensetmanusman 6y ago
  Yes, it is all pasted into my thesis, comments and all, like all code should.
• brobdingnagians 6y ago
  The Fossil documentation has this gem: > "The global state of a fossil repository is kept simple so that it can endure in useful form for decades or centuries. A fossil repository is intended to be readable, searchable, and extensible by people not yet born." I always liked that they planned for the long-term. Keeping that in mind helps you build systems that will work in 10 years, or in 100, if it happens to last that long. When you are building a foundation, like a language or database, it is nice to plan for long term support since so much depends on it. C has stayed mostly recognizable over the years, much more so than C++ or other high level languages. When your design is simple, you can have a "feature complete" end.
• rkagerer 6y ago
  "Visual Basic," Maggi writes in his report, "is a dead language and long since has been replaced..." In fact I still have the VB6 IDE installed on my primary workstation and use it for quick and dirty projects from time to time.
• biophysboy 6y ago
  I'm a grad student in biophysics - even if I wrote perfect code, it would almost certainly go obsolete in 10 years because the hardware that it interfaces with would go obsolete.
• Fishysoup 6y ago
  As a scientist I've written massive amounts of shitty code that turned out to be reproducible by lucky accident. Part of the problem are the tools: depending on the field, scientists either use Matlab, C++, Fortran or some other framework that needs to die. They base their code on other ancient code that runs for unknown reasons, and use packages written by other scientists with the same problems. As someone who's transitioning into industry, I can tell you that scientists will never adopt software engineering principles to any significant extent. It takes too much time to do things like write tests and thorough documentation, learn Git, etc., and software engineering just isn't interesting to most of them. So the only alternative I see is changing the tools to stuff that's still easy to hack around with but where it's harder to mess up (or it's more obvious when you do so). That doesn't leave a ton of options (that I can see). Some I can think of are: - Make your code look more like math and less like mathlib.linalg.dot(x1, x2).reshape(a, b).mean().euclidean_distance((x3, x4)) + (other long expression) or whatever: Use a language like Julia - Your language/environment gets angry when you write massive hairballs, loads of nested for-loops and variables that keep getting changed: Use a language like Rust, and/or write more modular code with a functional-leaning language like Rust or Julia. - You're forced to make your code semi-understandable to you and others more than an hour after writing it: Forcing people to write documentation isn't gonna work (a lot). Forcing sensible variable names is slightly more realistic. More likely, you need some combination of the above two things that just make your code more legible. How do you make that happen? No idea.
  • lambdatronics 6y ago
    Julia could be a big win, not just b/c of the notation, but the dependency control is a first-class language feature. Also, the Lispy-ness of Julia allows to do things like Latexify expressions. To quote someone upthread: https://news.ycombinator.com/item?id=24260590 https://news.ycombinator.com/item?id=24260590 > Everyone just put together a few text files and Python or MATLAB scripts that output some numbers that went into Excel or gnuplot scripts that got copy-pasted into LaTeX documents with suffixes like "v2_final_modified.tex", shared over Dropbox. It would be amazing to have an environment that could handle the entire workflow. Not everybody has time to make an executable thesis like this person did: https://github.com/4kbt/PlateWash https://github.com/4kbt/PlateWash
• klyrs 6y ago
  Snark: No, I wrote it in Python 2.7, because Python 3 was still wearing very smelly diapers in 2010. Ask me in another 10 years and I'll tell you the same, because Python doesn't care about backwards compatibility. Hopefully I'll have moved on to a more stable language by then. Maybe fortran?
• tester89 6y ago
  I hear Common Lisp is quite good for this. > but for fun, he spent €100 (US$110) on a Raspberry Pi, a single-board hobbyist computer that runs Linux and has Mathematica 12 pre-installed. Did he get scammed?
• kkylin 6y ago
  To throw in one more data point: several years ago I wrote a simulator for a paper, it involves tracking large numbers of particles bouncing through a 2D space with different geometries. There's a front end, in PLT Scheme, and a backend in C. I'm pretty sure that today - the C code would still compile and run - the Scheme code would not without some effort, because PLT Scheme 372 is no longer supported, Racket made a number of breaking changes, so one would either have to rewrite the Scheme code for Racket or rewrite it for another Scheme implementation.
• eointierney 6y ago
  Code must be re-used or it rots. Fortran was the first language I learned, and naming things was a problem. Guix, mentioned in the fine article, or like it is the answer as long as we keep reading and writing our code. This is why all scientific code should be public, we need continuous review. Papers without code should be refused. I'm in the middle of a storm in the west of Ireland. It's windy and wet and hn is still a thing. Happy out
• renox 6y ago
  Bah, even the STEPS project (from Alan Kay) was a total failure from reproducibility POV and that was a software research project :-(
• debbiedowner 6y ago
  With deep learning, using conda environments and environment modules to switch between various tensorflow, pytorch, cuda, and cudnn versions for reproducing/building off of other's results is essential. It is sort of hilarious how many breaking changes there are even between minor versions of these. Thank gosh I was able to install a 2 year old version of pytorch (0.4.1) today to reproduce some code from within the last year.
• zhte415 6y ago
  My 20-30+ year old code still runs: TI-85 BASIC, Turbo Pascal, Turing, Haskell, Smalltalk, Fortran, roughly in that order. Bit of a skeleton cupboard.
• Heliosmaster 6y ago
  We are building Nextjournal[0] exactly for this purpose. It's a platform for interactive notebooks built on immutable and persistent storage (Datomic) and Docker: - Changes to the document are automatically versioned to an immutable database (Datomic). Previous versions can be accessed and restored any time. - Uploaded data or generated result files are automatically versioned in append-only content-addressed storage and can’t be accidentally overwritten. - Changes to the file system state can be committed as Docker images. Reproducibility is ensured by referencing these images only by their immutable hashes. [0]https://nextjournal.com https://nextjournal.com
• known 6y ago
  Depends on context and competition