3 ms·

How does it compares to other HPC software managers such as easybuild?

throw0101b 4y ago
From the 7th EasyBuild User Meeting (2022), 'A noob test: Spack "vs" EasyBuild' presentation is available on YouTube: * https://easybuild.io/eum22/#spack-vs-eb https://easybuild.io/eum22/#spack-vs-eb @tgamblin also presented 'The Spack 2022 Roadmap': * https://easybuild.io/eum22/#spack https://easybuild.io/eum22/#spack And will also present at 8th EUM (April): * https://easybuild.io/eum/#spack https://easybuild.io/eum/#spack
- boegel 4y ago
  Better link for Spack talk at EUM'23: (the 'eum' link is a moving target) There's also an (now a bit outdated) talk compared EasyBuild with Spack and other alternatives like Nix/Guix which I gave a FOSDEM'18: https://archive.fosdem.org/2018/schedule/event/installing_software_for_scientists https://archive.fosdem.org/2018/schedule/event/installing_so... The EasyBuild User Meetings have always been very open to having talks on "competing" tools. Todd (or someone else) giving an update on recent developments in Spack is becoming a tradition (see also https://easybuild.io/eum21/#spack https://easybuild.io/eum21/#spack, https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild-User-Meeting https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild..., etc.)
tgamblin 4y ago
If I have to pick one difference, the fundamental one is that Spack does dependency resolution with a solver, while EB does not. So, in Spack, you write a package.py file (like say the one for gromacs: https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/gromacs/package.py https://github.com/spack/spack/blob/develop/var/spack/repos/...), and you can build that package with any of the versions/options/dependencies/etc. expressed in the package. With Easybuild, someone (maybe you?) has to write a specific configuration (easyconfig) for every build you want to do. Want to tweak a version? Write a new easyconfig. The dependencies point at specific versions, too, so now write configs for all the dependencies, and so on. There is a lot of copying/tweaking of file trees in the EB workflow. Aside from that: - Spack has a package database and allows you to query what is installed. - EB does not have "uninstall", just `rm -rf` - Spack builds from source and also handles binary packages (no binaries in EB) - Spack supports environments via `spack.yaml` / `spack.lock`; EB doesn't have that. - Spack doesn't require Lmod to work -- you can just `spack load` or `spack env activate` to use packages. - Spack builds with RPATH by default (much like Nix does) -- dependency libraries are found automatically even if you run an executable straight from its directory. There's more but Spack is a package manager, while EB is really set up to automate a certain type of installation common in HPC.
- boegel 4y ago
  EasyBuild lead developer here ^_^ Not all of this is 100% correct, so let me pitch in: - EasyBuild currently doesn't have an uninstall option, that's true, but since every software installation sits in its own separate directory, it basically boils down to removing that directory + the environment module file that EasyBuild generated for it; - EasyBuild can install "binary packages" (see the 'Binary' easyblock). Examples are the Intel compilers, CUDA, etc. We don't provides pre-built binaries for software that EasyBuild installs from source though, that's true; - EasyBuild has no "environments" concept. The closest thing perhaps is the 'Bundle' easyblock, that can be used to "glue" different environment modules together. We mostly rely on the environment modules tool (Lmod) so this, see for example module collections: https://lmod.readthedocs.io/en/latest/010_user.html#user-collections https://lmod.readthedocs.io/en/latest/010_user.html#user-col...; - EasyBuild does indeed require an environment modules tool. Both Lmod (Lua-based) and Environment Modules (Tcl-based) are supported; - EasyBuild also supports RPATH linking (see https://docs.easybuild.io/rpath-support/ https://docs.easybuild.io/rpath-support/), but it's not the default. > "EB is really set up to automate a certain type of installation common in HPC" EasyBuild is definitely geared towards installing software on HPC systems, but there's no "certain type of installation common in HPC": we support software using a wide variety of installation procedures. But maybe you're referring to installing software in a shared filesystem (NFS, usually something like /apps).
  - tgamblin 4y ago
    Maybe I should clarify: > it basically boils down to removing that directory + the environment module file that EasyBuild generated for it Another important thing is knowing what depends on that package (so you don't remove something else's dependency). This is something that EB doesn't track after installation time. > EasyBuild can install "binary packages" What I meant here is that EB has no binary packaging system of its own. It has no binary package format, no signing, no way to take an installation and bundle it up into a file that can be installed (potentially in a different location) on another system. Spack, Nix, and Guix all have systems for creating and (re)installing binary substitutes for source builds (we call them build caches). EB can install someone else's binary package/distribution (we can too FWIW), but it can't create its own. > But maybe you're referring to installing software in a shared filesystem (NFS, usually something like /apps). Yes -- specifically that, with environment modules. Spack installations similarly all go in their own directories, but you can compose them in many different ways with environments and views. e.g., you can: - Create a symlink tree of many packages together in a common prefix (a view) - "project" package installations into different directory layouts with symlinks, hardlinks, or relocated copies See https://spack.readthedocs.io/en/latest/environments.html#view-projections https://spack.readthedocs.io/en/latest/environments.html#vie... And you don't have to load packages one by one with modules. You can avoid modules entirely with `spack load` or use `spack env activate` to put everything in an env into your `PATH` (and other env vars).
stabbles 4y ago
Another point that I don't think gets emphasized a lot is that research software developers use spack as a dev tool on their laptop/cluster. Setting up easybuild locally is too much effort? If developers maintain spack recipes because they need them, it's much easier for sysadmins to deploy this software with spack too
- boegel 4y ago
  I don't think it's about effort to set up EasyBuild (although we do have a hard requirement for a modules tool). Spack is perhaps more attractive to software developers because it has specific features for that use, like the flexible dependency handling mechanism and the concretiser. In my view, EasyBuild is better suited than Spack to maintain a central software stack, but I'm definitely biased. :)
Moissanite 4y ago
Spack has significantly higher adoption than EasyBuild overall, though Spack usage has a US bias while EasyBuild is more popular with EU public sector HPC sites.
stonogo 4y ago
Spack recipes are more python-like, it is less tied to LMod, and its concretization algorithm has no real rival. Easybuild supports recursive modules and features to generalize boilerplate.
- deleted 4y ago
  [deleted]

From the 7th EasyBuild User Meeting (2022), 'A noob test: Spack "vs" EasyBuild' presentation is available on YouTube: * https://easybuild.io/eum22/#spack-vs-eb https://easybuild.io/eum22/#spack-vs-eb @tgamblin also presented 'The Spack 2022 Roadmap': * https://easybuild.io/eum22/#spack https://easybuild.io/eum22/#spack And will also present at 8th EUM (April): * https://easybuild.io/eum/#spack https://easybuild.io/eum/#spack

boegel 4y ago
Better link for Spack talk at EUM'23: (the 'eum' link is a moving target) There's also an (now a bit outdated) talk compared EasyBuild with Spack and other alternatives like Nix/Guix which I gave a FOSDEM'18: https://archive.fosdem.org/2018/schedule/event/installing_software_for_scientists https://archive.fosdem.org/2018/schedule/event/installing_so... The EasyBuild User Meetings have always been very open to having talks on "competing" tools. Todd (or someone else) giving an update on recent developments in Spack is becoming a tradition (see also https://easybuild.io/eum21/#spack https://easybuild.io/eum21/#spack, https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild-User-Meeting https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild..., etc.)

tgamblin 4y ago

If I have to pick one difference, the fundamental one is that Spack does dependency resolution with a solver, while EB does not. So, in Spack, you write a package.py file (like say the one for gromacs: https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/gromacs/package.py https://github.com/spack/spack/blob/develop/var/spack/repos/...), and you can build that package with any of the versions/options/dependencies/etc. expressed in the package. With Easybuild, someone (maybe you?) has to write a specific configuration (easyconfig) for every build you want to do. Want to tweak a version? Write a new easyconfig. The dependencies point at specific versions, too, so now write configs for all the dependencies, and so on. There is a lot of copying/tweaking of file trees in the EB workflow. Aside from that: - Spack has a package database and allows you to query what is installed. - EB does not have "uninstall", just `rm -rf` - Spack builds from source and also handles binary packages (no binaries in EB) - Spack supports environments via `spack.yaml` / `spack.lock`; EB doesn't have that. - Spack doesn't require Lmod to work -- you can just `spack load` or `spack env activate` to use packages. - Spack builds with RPATH by default (much like Nix does) -- dependency libraries are found automatically even if you run an executable straight from its directory. There's more but Spack is a package manager, while EB is really set up to automate a certain type of installation common in HPC.

boegel 4y ago
EasyBuild lead developer here ^_^ Not all of this is 100% correct, so let me pitch in: - EasyBuild currently doesn't have an uninstall option, that's true, but since every software installation sits in its own separate directory, it basically boils down to removing that directory + the environment module file that EasyBuild generated for it; - EasyBuild can install "binary packages" (see the 'Binary' easyblock). Examples are the Intel compilers, CUDA, etc. We don't provides pre-built binaries for software that EasyBuild installs from source though, that's true; - EasyBuild has no "environments" concept. The closest thing perhaps is the 'Bundle' easyblock, that can be used to "glue" different environment modules together. We mostly rely on the environment modules tool (Lmod) so this, see for example module collections: https://lmod.readthedocs.io/en/latest/010_user.html#user-collections https://lmod.readthedocs.io/en/latest/010_user.html#user-col...; - EasyBuild does indeed require an environment modules tool. Both Lmod (Lua-based) and Environment Modules (Tcl-based) are supported; - EasyBuild also supports RPATH linking (see https://docs.easybuild.io/rpath-support/ https://docs.easybuild.io/rpath-support/), but it's not the default. > "EB is really set up to automate a certain type of installation common in HPC" EasyBuild is definitely geared towards installing software on HPC systems, but there's no "certain type of installation common in HPC": we support software using a wide variety of installation procedures. But maybe you're referring to installing software in a shared filesystem (NFS, usually something like /apps).
- tgamblin 4y ago
  Maybe I should clarify: > it basically boils down to removing that directory + the environment module file that EasyBuild generated for it Another important thing is knowing what depends on that package (so you don't remove something else's dependency). This is something that EB doesn't track after installation time. > EasyBuild can install "binary packages" What I meant here is that EB has no binary packaging system of its own. It has no binary package format, no signing, no way to take an installation and bundle it up into a file that can be installed (potentially in a different location) on another system. Spack, Nix, and Guix all have systems for creating and (re)installing binary substitutes for source builds (we call them build caches). EB can install someone else's binary package/distribution (we can too FWIW), but it can't create its own. > But maybe you're referring to installing software in a shared filesystem (NFS, usually something like /apps). Yes -- specifically that, with environment modules. Spack installations similarly all go in their own directories, but you can compose them in many different ways with environments and views. e.g., you can: - Create a symlink tree of many packages together in a common prefix (a view) - "project" package installations into different directory layouts with symlinks, hardlinks, or relocated copies See https://spack.readthedocs.io/en/latest/environments.html#view-projections https://spack.readthedocs.io/en/latest/environments.html#vie... And you don't have to load packages one by one with modules. You can avoid modules entirely with `spack load` or use `spack env activate` to put everything in an env into your `PATH` (and other env vars).

stabbles 4y ago

Another point that I don't think gets emphasized a lot is that research software developers use spack as a dev tool on their laptop/cluster. Setting up easybuild locally is too much effort? If developers maintain spack recipes because they need them, it's much easier for sysadmins to deploy this software with spack too

boegel 4y ago
I don't think it's about effort to set up EasyBuild (although we do have a hard requirement for a modules tool). Spack is perhaps more attractive to software developers because it has specific features for that use, like the flexible dependency handling mechanism and the concretiser. In my view, EasyBuild is better suited than Spack to maintain a central software stack, but I'm definitely biased. :)

Moissanite 4y ago

Spack has significantly higher adoption than EasyBuild overall, though Spack usage has a US bias while EasyBuild is more popular with EU public sector HPC sites.

stonogo 4y ago

Spack recipes are more python-like, it is less tied to LMod, and its concretization algorithm has no real rival. Easybuild supports recursive modules and features to generalize boilerplate.

deleted 4y ago
[deleted]