4 ms·

Has anyone outside of CERN used it on a large scale successfully?

tempay 4y ago
It’s very widely used outside of CERN in research/academic contexts. There are definitely commercial users (though I know less about their status). Next week there is a workshop with talks from some of the users: https://indico.cern.ch/event/1079490/timetable/#20220912.detailed https://indico.cern.ch/event/1079490/timetable/#20220912.det...
- CaliforniaKarl 4y ago
  Cool! Do you know, will there be a recording?
  - jblomer 4y ago
    Project owner here. Most likely yes, pending speakers' consent.
    - CaliforniaKarl 4y ago
      Thank you!
rohithkp 4y ago
The Neurodesk Project[1] uses CVMFS to fetch Singularity container images dynamically on-the-fly from a central repository. [1] https://neurodesk.org https://neurodesk.org
- carapace 4y ago
  Yeah I saw a (mind-blowing) talk on Neurodesk (and Brainhack Cloud) last night (NeuroTechX Hacknight at Noisebridge in SF https://www.meetup.com/neurotechx-sf/events/288358663/ https://www.meetup.com/neurotechx-sf/events/288358663/ ) and that's what led me to CVMFS. Neurodesk is nuts (in a good way!) It successfully abstracts away so much of the gnarly details. You need different versions of that obscure program? No problem, have them all, run them concurrently, whatever. Even if you're not a researcher it's worth checking out for the technical accomplishments.
mboisso 4y ago
Compute Canada (now the Digital Research Alliance of Canada, http://alliancecan.ca/ http://alliancecan.ca/) has been using CVMFS in production for nearly 7 years now. We use it to distribute all of the software environment used by our researchers on all of our national clusters, as well as to make it available for researchers themselves and university owned computers, and virtual machines or virtual clusters in the cloud. Documentation on using it on any Linux computer here: https://docs.alliancecan.ca/wiki/Accessing_CVMFS https://docs.alliancecan.ca/wiki/Accessing_CVMFS. CVMFS has been essential in making a uniform user-facing software environment available throughout our distributed infrastructure (we have 5 different clusters located across the country, totalizing over 250k cores, close to 2k GPUs, and something like 200PB of disk storage). It is a proven geo-distributed redundant and high performance (through local caching) filesystem which enables a install-once-available-everywhere strategy. Our software stack currently hosts over 1150 different scientific software packages in over 2800 versions, and built optimized for multiple CPU architectures, totalizing nearly 10k different builds. This is in addition to nearly 10k python packages which we make available in the form of python wheels. Our software stack now is close to 8TB in size, and contains tens of millions of files.

It’s very widely used outside of CERN in research/academic contexts. There are definitely commercial users (though I know less about their status). Next week there is a workshop with talks from some of the users: https://indico.cern.ch/event/1079490/timetable/#20220912.detailed https://indico.cern.ch/event/1079490/timetable/#20220912.det...

CaliforniaKarl 4y ago
Cool! Do you know, will there be a recording?
- jblomer 4y ago
  Project owner here. Most likely yes, pending speakers' consent.
  - CaliforniaKarl 4y ago
    Thank you!

rohithkp 4y ago

The Neurodesk Project[1] uses CVMFS to fetch Singularity container images dynamically on-the-fly from a central repository. [1] https://neurodesk.org https://neurodesk.org

carapace 4y ago
Yeah I saw a (mind-blowing) talk on Neurodesk (and Brainhack Cloud) last night (NeuroTechX Hacknight at Noisebridge in SF https://www.meetup.com/neurotechx-sf/events/288358663/ https://www.meetup.com/neurotechx-sf/events/288358663/ ) and that's what led me to CVMFS. Neurodesk is nuts (in a good way!) It successfully abstracts away so much of the gnarly details. You need different versions of that obscure program? No problem, have them all, run them concurrently, whatever. Even if you're not a researcher it's worth checking out for the technical accomplishments.

mboisso 4y ago

Compute Canada (now the Digital Research Alliance of Canada, http://alliancecan.ca/ http://alliancecan.ca/) has been using CVMFS in production for nearly 7 years now. We use it to distribute all of the software environment used by our researchers on all of our national clusters, as well as to make it available for researchers themselves and university owned computers, and virtual machines or virtual clusters in the cloud. Documentation on using it on any Linux computer here: https://docs.alliancecan.ca/wiki/Accessing_CVMFS https://docs.alliancecan.ca/wiki/Accessing_CVMFS. CVMFS has been essential in making a uniform user-facing software environment available throughout our distributed infrastructure (we have 5 different clusters located across the country, totalizing over 250k cores, close to 2k GPUs, and something like 200PB of disk storage). It is a proven geo-distributed redundant and high performance (through local caching) filesystem which enables a install-once-available-everywhere strategy. Our software stack currently hosts over 1150 different scientific software packages in over 2800 versions, and built optimized for multiple CPU architectures, totalizing nearly 10k different builds. This is in addition to nearly 10k python packages which we make available in the form of python wheels. Our software stack now is close to 8TB in size, and contains tens of millions of files.