3 ms·

This article misses the point for the Scientist in data science. HPC will not support docker, Singularity seems to be the winner: http://singularity.lbl.gov/ ht

by tbenst 9y ago

This article misses the point for the Scientist in data science. HPC will not support docker, Singularity seems to be the winner: http://singularity.lbl.gov/ http://singularity.lbl.gov/

codles 9y ago
This is a great point. People on HN often forget that data science is more than just machine learning, and that we have existing computing infrastructure (beyond AWS) to interface with.
- rbanffy 9y ago
  Well... Singularity can import Docker containers, so I assume it's easy to cross over to that space.
deleted 9y ago
[deleted]
batbomb 9y ago
More than that really: For those who don't understand why this is important: In NSF and DOE funded projects, you are increasingly being told to use supercomputing centers, which effectively all install global GPFS and Lustre file systems to the HPC computers, usually fronted by Slurm, but maybe Torque, LSF, GridEngine, or HTCondor. There are two Container runtimes that (mostly) work for HPC. Shifter, which is just Docker + image patching (and no layers) + some other unhelpful things like mounting in your home directory. Shifter came from NERSC who sort of gave it to Cray but still maintain it, and Singularity, which came from LBNL, which actually runs NERSC. NERSC thinks their Shifter is going to win, and they won't even, in fact, install Singularity on their clusters (they say it's for security reasons). Outside of NERSC, there is a tiny bit of traction of Shifter, but not much. Cray is pushing it hard. Singularity is inferior in many ways to Docker. The most glaring being you can't use it with Kubernetes easily (if at all). It exports a large image. In the grid, those aren't that big of a problem. When you pair it with CVMFS, you get a nice way to cache and distribute containers (with the CVMFS cache) and a bunch of other benefits that are relatively nice. The Kubernetes thing is still a big issue, because it means you can't provide a secure way of making something like JupyterHub+Jupyterlab available with your mounted GPFS file systems (though you can sort of get away with it by exporting GPFS through NFS with root squashing, but that's not a great or complete solution). The recommendations I give to projects are start with Docker, export to Singularity. Don't give your scientists Singularity containers for their Macs, lord knows how crappy that experience is on a Mac. The situation sucks. If Docker was willing to do something about it, Singularity would die tomorrow.
- stochastician 9y ago
  Can you explain by what you mean w.r.t. "NSF and DOE funded projects, you are increasingly being told to use supercomputing centers"?
  - batbomb 9y ago
    If you're a DOE project at a lab and you need compute, they aren't willing to give you money to purchase hardware or cloud unless you have an extremely compelling reason why you need to do that, and "Our software doesn't support GPUs/KNL/the Cray OS" isn't a valid excuse. They don't want you buying pizza boxes and they don't want you using cloud, they want you using the supercomputers they've already sunk money in. Small NSF projects are probably fine to do what they want, 50M+ range is probably where they kick you off to an XSEDE service provider.

by tbenst 9y ago

This article misses the point for the Scientist in data science. HPC will not support docker, Singularity seems to be the winner: http://singularity.lbl.gov/ http://singularity.lbl.gov/

codles 9y ago

This is a great point. People on HN often forget that data science is more than just machine learning, and that we have existing computing infrastructure (beyond AWS) to interface with.

rbanffy 9y ago
Well... Singularity can import Docker containers, so I assume it's easy to cross over to that space.

deleted 9y ago

[deleted]

batbomb 9y ago

More than that really: For those who don't understand why this is important: In NSF and DOE funded projects, you are increasingly being told to use supercomputing centers, which effectively all install global GPFS and Lustre file systems to the HPC computers, usually fronted by Slurm, but maybe Torque, LSF, GridEngine, or HTCondor. There are two Container runtimes that (mostly) work for HPC. Shifter, which is just Docker + image patching (and no layers) + some other unhelpful things like mounting in your home directory. Shifter came from NERSC who sort of gave it to Cray but still maintain it, and Singularity, which came from LBNL, which actually runs NERSC. NERSC thinks their Shifter is going to win, and they won't even, in fact, install Singularity on their clusters (they say it's for security reasons). Outside of NERSC, there is a tiny bit of traction of Shifter, but not much. Cray is pushing it hard. Singularity is inferior in many ways to Docker. The most glaring being you can't use it with Kubernetes easily (if at all). It exports a large image. In the grid, those aren't that big of a problem. When you pair it with CVMFS, you get a nice way to cache and distribute containers (with the CVMFS cache) and a bunch of other benefits that are relatively nice. The Kubernetes thing is still a big issue, because it means you can't provide a secure way of making something like JupyterHub+Jupyterlab available with your mounted GPFS file systems (though you can sort of get away with it by exporting GPFS through NFS with root squashing, but that's not a great or complete solution). The recommendations I give to projects are start with Docker, export to Singularity. Don't give your scientists Singularity containers for their Macs, lord knows how crappy that experience is on a Mac. The situation sucks. If Docker was willing to do something about it, Singularity would die tomorrow.

stochastician 9y ago
Can you explain by what you mean w.r.t. "NSF and DOE funded projects, you are increasingly being told to use supercomputing centers"?
- batbomb 9y ago
  If you're a DOE project at a lab and you need compute, they aren't willing to give you money to purchase hardware or cloud unless you have an extremely compelling reason why you need to do that, and "Our software doesn't support GPUs/KNL/the Cray OS" isn't a valid excuse. They don't want you buying pizza boxes and they don't want you using cloud, they want you using the supercomputers they've already sunk money in. Small NSF projects are probably fine to do what they want, 50M+ range is probably where they kick you off to an XSEDE service provider.