3 ms·
Spack [1,2] is our attempt to do this for HPC, where we have two things going on: 1) Our applications aren't single-language. They include Fortran, Python, C,
by tgamblin 11y ago
Spack [1,2] is our attempt to do this for HPC, where we have two things going on:
1) Our applications aren't single-language. They include Fortran, Python, C, C++, Yorick (!), Lua, etc. etc.
2) We have a major combinatorial versioning problem, because HPC clusters (and scientific software in general) rely on ABI-incompatible libraries like MPI, and MANY different compilers.
Spack is build-from-source and takes a lot of inspiration from Nix and its descendants, but it adds support for things like swapping out compilers/MPI versions/etc., as well as a syntax for composing new builds of things on the fly That is, you can build something with a slightly different version of a dependency without ever editing a package file. Our users to be able to build a package with 6 different compilers and 3 different MPI library versions, and have their 18 versions coexist on the same system. This has been super helpful for testing production code at LLNL, because we have to run our stuff on other peoples' often very different clusters. For instance, at LANL, we prefer PGI compilers. At LLNL we tend to like Intel or Clang.
One reason languages tend to grow their own package managers is that there are all kinds of peculiarities about how different languages manage modules/extensions. Have you seen how many different ways there are to deploy Python modules? It's very hard to have multiple versions of things coexisting in the same Python env, and things like pkg_resources require editing client code [3]. Ew. Spack tries to address this by allowing different language packages to implement their own module activation/deactivation logic, so you can kind of have a language-specfic package manager for each install of the language. I'd rather have something more like Nix's profiles, though.
I wouldn't say that the project is huge yet, but we've had some interest from mostly other HPC sites. It would be cool if the project caught on in the "real" world, too.
[1] https://github.com/llnl/spack https://github.com/llnl/spack
[2] http://www.computer.org/csdl/proceedings/sc/2015/3723/00/2807623.pdf http://www.computer.org/csdl/proceedings/sc/2015/3723/00/280...
[3] http://stackoverflow.com/questions/5265731 http://stackoverflow.com/questions/5265731
- civodul 11y agoI find that Spack in fact takes very little inspiration from functional package managers, Nix and GNU Guix. Dependencies are specified in a loose fashion which, on one hand, gives a lot of flexibility, but on the other hand is detrimental to reproducibility. More importantly, part of the dependency graph, including compilers, is treated specially and essentialy considered outside of the scope of Spack. This seriously hinders reproducibility, as we tried to explain in https://hal.inria.fr/hal-01161771/en https://hal.inria.fr/hal-01161771/en .