6 ms·

Going to bang a small drum here and proclaim: When doing science, don't use a tech stack that will be obselete next year or in 3 years. Use something that wil

by pnathan 13y ago

Going to bang a small drum here and proclaim: When doing science, don't use a tech stack that will be obselete next year or in 3 years. Use something that will be AROUND; something that is well defined, something that other people can use down the road on a different system. Virtual machines are "okay". Running them requires having the hypervisor working correctly and - hopefully - the original architecture supported. Personally, I used Common Lisp for my Master's work. Other examples of tech stacks that are very stable are C, C++, and Fortran.

ics 13y ago
It's certainly ideal if everyone can manage to future proof their work, but I don't really know how realistic that is for many considering the demands. Learning Fortran is a formidable proposition to anyone who can do their work well in Python. At least we can be optimistic that as the tech gets better and better it will be easier to emulate dated environments with greater resources, but of course it's important that there be some interested party. If the research sphere embraces these sorts of action the pool of interested parties should only grow, and the methods for sustaining the practice should get better. Whether that means some sort of gentleman's agreement or standardization on how to create or maintain the machines I couldn't say, but we could probably do much better.
deleted 13y ago
[deleted]
derefr 13y ago
I fundamentally agree--yet, at the same time, it'd be so nice if a paper could just point to a URL for a Docker container, and you could pull it and immediately execute the program.
- limmeau 13y ago
  However, I fear that a Docker container in 2023 will be as useful as a Plex86 VM image today...
  - DannoHung 13y ago
    The issue isn't really software then, it's hardware obsolescence.
    - limmeau 13y ago
      That example still completely in the realm of software obsolescence. The LXC API may not be around in 2023's Linux kernels any more, just like Plex86 has been abandoned (and I doubt that Plex86 runs unchanged on a modern x86_64 machine with a 3.x kernel). But hardware obsolescence is a valid issue, too. That's why I suggested Bochs as a lowest common denominator in another comment.
      - DannoHung 13y ago
        Why are you limited to using the 2023 version of the kernel? Are they going to erase the 2013 version of the kernel in 2018?
        limmeau 13y ago
        No, but maybe your Apple GeForce 5500 graphics drivers and the drivers for the ansible network haven't been backported, so you'd have to run headless and offline. Or in another layer of VM, or two if the VM's virtual hardware is still too new for the old kernel. At some point, you'd just start to believe unseen that the paper's authors were probably right.
        vidarh 13y ago
        The point is not that everyone will recompute everything, but ensuring that it remains possible.
        vidarh 13y ago
        The point is not for everyone to recompute everything, but to ensure that it remains possible, and with known, well defined hardware platforms capable of running a specific bundle of software, it will likely reman possible for much longer than anyone cares about the specific papers.
- natejenkins 13y ago
  I am working on a startup addressing this exact problem. Docker sounds really exciting although I haven't played much with it yet. Limmeau's concern below is a valid one. With the popularity of cloud computing centered around VMs, it seems like a wise choice to at least offer a VM as a worst-case option for recomputing an experiment. As mentioned in the article, performance should be secondary when thinking about recomputation. A reasonable solution to the large downloads necessary is to spin up the VM on the server side and allow access to it either through the browser or through a terminal login.
  - nickstinemates 13y ago
    You should definitely give Docker a try. Drop by the IRC channel and you'll find a great community ready to give you some advice.
chalst 13y ago
ANSI C is standard, as are its standard libraries, but I guess little C code avoids the use of nonstandard libraries, and some popular libraries are tricky to build even today. And much C code uses compiler-specific extensions.
lifebeyondfife 13y ago
For certain research problems, different and often obscure tool sets are required. That, to me, is a great argument behind recreating everything your environment needs in a VM - unlike Chemists who can't bundle up their lab into a little transportable box, Computer Scientists can. In making the argument for recomputation to aid its adoption, it's important not to add too many constraints on the researcher e.g. "Your results should be reproduceable and written in C because it should still be around ten years from now." Even an obsolete language in a working VM sandbox can be edited and inspected to make changes, verify correctness etc. If you want to extend the results of that research you may have a porting project to update the code to more a current language, but that's leaps and bounds ahead of how things are now: read and re-read the paper until it makes sense and implement the algorithms from scratch. Agree with your hypervisor/architecture point but I suppose there has to be some broad architectural choice made. (The author of this article was my PhD supervisor. I've always found him to be an inspiring, forward thinking researcher so as much as I love the recomputation idea on merit, I'm also a little biased.)
- tetha 13y ago
  > read and re-read the paper until it makes sense and implement the algorithms from scratch. More often than not to realize that some nasty detail got omitted because it was nasty.
  - dasmoth 13y ago
    To me, this is one of the reasons why it's good for people to do independent re-implementations. Academia doesn't value these highly enough. Recomputation and reproduction are both valuable
    - lifebeyondfife 13y ago
      I agree completely. From my experience, because of things like the nasty omitted details mentioned in the parent comment, techniques are rarely reimplemented unless they claim pretty substantial improvements. I think having a starting point of reproduceability would lead to more reimplementations.
SkyMarshal 13y ago
>Other examples of tech stacks that are very stable are C, C++, and Fortran. You can probably add Python to that list now too, given how embedded it is in *nix ecosystem and its excellent scientific/numerical libs.
vidarh 13y ago
> Other examples of tech stacks that are very stable are C, C++, and Fortran. On the scale of a few years, yes. Assuming you don't depend on external libraries at all. But I've worked on recompiling more than enough C code in the 15-20 year range or older that contained sufficient assumptions - some valid then, some that just happened to work then - that are no longer valid on most common architectures. If you care about recomputability for <5 years, you can probably get away with "just" shipping a bundle with the code and any library dependencies. If you care about anything more than that, you're on shaky ground very quickly. On the other hand, we have javascript emulators for a bunch of 30+ year old architectures. As it happens, it is becoming increasingly clear that targeting hardware stacks is far more stable than trying to re-assemble a software stack.

by pnathan 13y ago

ics 13y ago

It's certainly ideal if everyone can manage to future proof their work, but I don't really know how realistic that is for many considering the demands. Learning Fortran is a formidable proposition to anyone who can do their work well in Python. At least we can be optimistic that as the tech gets better and better it will be easier to emulate dated environments with greater resources, but of course it's important that there be some interested party. If the research sphere embraces these sorts of action the pool of interested parties should only grow, and the methods for sustaining the practice should get better. Whether that means some sort of gentleman's agreement or standardization on how to create or maintain the machines I couldn't say, but we could probably do much better.

deleted 13y ago

[deleted]

derefr 13y ago

I fundamentally agree--yet, at the same time, it'd be so nice if a paper could just point to a URL for a Docker container, and you could pull it and immediately execute the program.

limmeau 13y ago
However, I fear that a Docker container in 2023 will be as useful as a Plex86 VM image today...
- DannoHung 13y ago
  The issue isn't really software then, it's hardware obsolescence.
  - limmeau 13y ago
    That example still completely in the realm of software obsolescence. The LXC API may not be around in 2023's Linux kernels any more, just like Plex86 has been abandoned (and I doubt that Plex86 runs unchanged on a modern x86_64 machine with a 3.x kernel). But hardware obsolescence is a valid issue, too. That's why I suggested Bochs as a lowest common denominator in another comment.
    - DannoHung 13y ago
      Why are you limited to using the 2023 version of the kernel? Are they going to erase the 2013 version of the kernel in 2018?
      - limmeau 13y ago
        No, but maybe your Apple GeForce 5500 graphics drivers and the drivers for the ansible network haven't been backported, so you'd have to run headless and offline. Or in another layer of VM, or two if the VM's virtual hardware is still too new for the old kernel. At some point, you'd just start to believe unseen that the paper's authors were probably right.
        vidarh 13y ago
        The point is not that everyone will recompute everything, but ensuring that it remains possible.
        vidarh 13y ago
        The point is not for everyone to recompute everything, but to ensure that it remains possible, and with known, well defined hardware platforms capable of running a specific bundle of software, it will likely reman possible for much longer than anyone cares about the specific papers.
natejenkins 13y ago
I am working on a startup addressing this exact problem. Docker sounds really exciting although I haven't played much with it yet. Limmeau's concern below is a valid one. With the popularity of cloud computing centered around VMs, it seems like a wise choice to at least offer a VM as a worst-case option for recomputing an experiment. As mentioned in the article, performance should be secondary when thinking about recomputation. A reasonable solution to the large downloads necessary is to spin up the VM on the server side and allow access to it either through the browser or through a terminal login.
- nickstinemates 13y ago
  You should definitely give Docker a try. Drop by the IRC channel and you'll find a great community ready to give you some advice.

chalst 13y ago

ANSI C is standard, as are its standard libraries, but I guess little C code avoids the use of nonstandard libraries, and some popular libraries are tricky to build even today. And much C code uses compiler-specific extensions.

lifebeyondfife 13y ago

For certain research problems, different and often obscure tool sets are required. That, to me, is a great argument behind recreating everything your environment needs in a VM - unlike Chemists who can't bundle up their lab into a little transportable box, Computer Scientists can. In making the argument for recomputation to aid its adoption, it's important not to add too many constraints on the researcher e.g. "Your results should be reproduceable and written in C because it should still be around ten years from now." Even an obsolete language in a working VM sandbox can be edited and inspected to make changes, verify correctness etc. If you want to extend the results of that research you may have a porting project to update the code to more a current language, but that's leaps and bounds ahead of how things are now: read and re-read the paper until it makes sense and implement the algorithms from scratch. Agree with your hypervisor/architecture point but I suppose there has to be some broad architectural choice made. (The author of this article was my PhD supervisor. I've always found him to be an inspiring, forward thinking researcher so as much as I love the recomputation idea on merit, I'm also a little biased.)

tetha 13y ago
> read and re-read the paper until it makes sense and implement the algorithms from scratch. More often than not to realize that some nasty detail got omitted because it was nasty.
- dasmoth 13y ago
  To me, this is one of the reasons why it's good for people to do independent re-implementations. Academia doesn't value these highly enough. Recomputation and reproduction are both valuable
  - lifebeyondfife 13y ago
    I agree completely. From my experience, because of things like the nasty omitted details mentioned in the parent comment, techniques are rarely reimplemented unless they claim pretty substantial improvements. I think having a starting point of reproduceability would lead to more reimplementations.

SkyMarshal 13y ago

>Other examples of tech stacks that are very stable are C, C++, and Fortran. You can probably add Python to that list now too, given how embedded it is in *nix ecosystem and its excellent scientific/numerical libs.

vidarh 13y ago

> Other examples of tech stacks that are very stable are C, C++, and Fortran. On the scale of a few years, yes. Assuming you don't depend on external libraries at all. But I've worked on recompiling more than enough C code in the 15-20 year range or older that contained sufficient assumptions - some valid then, some that just happened to work then - that are no longer valid on most common architectures. If you care about recomputability for <5 years, you can probably get away with "just" shipping a bundle with the code and any library dependencies. If you care about anything more than that, you're on shaky ground very quickly. On the other hand, we have javascript emulators for a bunch of 30+ year old architectures. As it happens, it is becoming increasingly clear that targeting hardware stacks is far more stable than trying to re-assemble a software stack.