18 ms·

Original author here -- since I know that people click the comments to decide whether to read further, here's a (biased?) summary: Spack is aimed at end users,

Original author here -- since I know that people click the comments to decide whether to read further, here's a (biased?) summary: Spack is aimed at end users, developers, and admins. It's used primarily in HPC at the moment but it would be great to see use outside of the HPC community. Spack supports: 1. Basic installation (spack install foo, foo@3.1.2, foo@3.1.2 +option, etc.) 2. Reproducible environments, via `spack.yaml` and `spack.lock` files that you can version in a repo. You can use this to build a custom software stack, build some dependencies for a project, etc. 3. Building many versions of the same package, e.g. these are perfectly fine and can coexist: - spack install hdf5 - spack install hdf5 %clang - spack install hdf5 %oneapi@2023.2.0 - spack install hdf5@1.18.0 cxxflags="-O3 -fast" target=cascadelake - spack install hdf5 +mpi ^mpich - spack install hdf5 +mpi ^openmpi 4. Both source builds and installation from relocatable binary caches 5. Lots of conditional features in packages that other systems do not offer, e.g.: - injecting compiler flags - building for specific uarch targets - optional dependencies - very fine-grained options (e.g. +/- cuda, set cuda_arch, etc.) - building the same package with different compilers - building the same package with different potentially ABI-incompatible dependencies like MPI or different boost versions. 5. Autogenerating things like container recipes, CI pipelines, Lmod/environment modules, etc. 6. Good support for building against external packages that may already be on your system. At a very high level, Spack has: * Nix's installation model and configuration hashing * Homebrew-like packages, but in a more expressive Python DSL, and with more versions/options * A very powerful dependency resolver that doesn't just pick from a set of available configurations -- it configures your build according to possible configurations. You could think of it like Nix with dependency resolution, but with a nice Python DSL. There is more on the "concretizer" (resolver) and how we've used ASP for it here: * "Using Answer Set Programming for HPC Dependency Solving", https://arxiv.org/abs/2210.08404 https://arxiv.org/abs/2210.08404

throw0101b 4y ago
> 6. Good support for building against external packages that may already be on your system. Able to use 'internal' packages (spack install openssl) or external packages (link against /usr/lib/libssl.so) on a per item basis: sometimes you want the newest code, but other times you want the OS package so that you can get security updates 'for free'. Most other system do one or the other: Spack allows you to choose.
gdevenyi 4y ago
lmod integration?
- throw0101b 4y ago
  Yes: * https://spack.readthedocs.io/en/latest/module_file_support.html https://spack.readthedocs.io/en/latest/module_file_support.h... Both Lmod and (TCL) Modules: can generate either/both.
yjftsjthsd-h 4y ago
Very neat:) Given the similarity in features, could you comment on how it compares to nix?
- 331c8c71 4y ago
  +100 From what I saw the package definitions are in python. Given python is really an imperative language I don't see how spack could be comparably powerful for definitions and reuse.
  - stabbles 4y ago
    The package definitions are quite declarative even though the language is imperative, see an example here [1]. For hashes, Spack uses the normalized (comments, whitespace etc removed) abstract syntax tree. With Spack it's easier to contribute, you don't have to learn a new language as you need for Nix / Guix. [1] https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/zstd/package.py#L9-L57 https://github.com/spack/spack/blob/develop/var/spack/repos/...
  - yjftsjthsd-h 4y ago
    I could be persuaded to trade a certain amount of power for ease of use
  - tgamblin 4y ago
    The Spack DSL for defining versions, options, etc. is declarative. That part of a package exposes options to the solver and very much allows reuse, in the sense that the same package can be built many different ways, and even with different dependencies. The whole system is parameterized in ways that Nix is not. The imperative part of a Spack package is the build recipe, which AFAICT is not so different from bundling a bash script in a Nix derivation. Just like Nix, it's included in our configuration hash, so if you change the recipe you get a different installation of the software.
    - 331c8c71 4y ago
      Thanks for the answer. > That part of a package exposes options ... in the sense that the same package can be built many different ways, and even with different dependencies. It is quite a common pattern in nix to have package build flags and the dependencies as inputs in a derivation. This definitely makes it possible to build in different ways and with different dependencies.
deleted 4y ago
[deleted]
eviks 4y ago
Compared to homebrew, do prebuilt binaries have to be installed in a predefined path they were built in or is it also flexible? Also, is there an indication/warning whether the package is prebuilt before you install?
- tgamblin 4y ago
  > do prebuilt binaries rerouted to be installed in a predefined path they were built in or is it also flexible? Prebuilt binaries don't have to go in a fixed prefix -- we'll relocate RPATHs, shebangs, strings in text files, and (as long as you build in a longer path than you install to -- see `padded_length` at https://spack.readthedocs.io/en/latest/config_yaml.html https://spack.readthedocs.io/en/latest/config_yaml.html) strings in binaries. There is not currently a way to see before you install if an install will be from binary or from source, but that is something we are working on.
  - eviks 4y ago
    So how would I know whether the prebuilt binaries were built in a longer path than I install to? Is there at least a way to block source builds if the app is not in the binary cache to avoid any surprises (basically have manual override/confirmation instead of automatic fallbacks to source builds)?
    - tgamblin 4y ago
      > So how would I know whether the prebuilt binaries were built in a longer path than I install to? All the public ones are built with very long padding; if you make your own you're currently on your own. > Is there at least a way to block source builds if the app is not in the binary cache to avoid any surprises (basically have manual override/confirmation instead of automatic fallbacks to source builds)? Yep: spack install --cache-only ...
nerdponx 4y ago
What do you think about Spack vs. Conda/Mamba? I've been a contented (maybe not "happy") Conda user for years. Is there any reason to switch? I'd be concerned about package availability.
- throw0101b 4y ago
  Spack has Anaconda as a package: * https://spack.readthedocs.io/en/latest/package_list.html#anaconda3 https://spack.readthedocs.io/en/latest/package_list.html#ana... * https://spack.readthedocs.io/en/latest/replace_conda_homebrew.html https://spack.readthedocs.io/en/latest/replace_conda_homebre... So you can continue using it, but you also have a super-set of features for other things as well.
- tgamblin 4y ago
  We have a guide for this written by Adam Stewart, probably our most prolific (package) contributor: * https://spack.readthedocs.io/en/latest/replace_conda_homebrew.html https://spack.readthedocs.io/en/latest/replace_conda_homebre... The thing you'll likely notice most about Spack vs. conda/mamba/rattler(?) is that it lets you pick your versions and options much more fluidly, and it'll build you a package from source if there's no binary for it. Spack is much more seamless for integrating binary installations with source builds. `conda-build` is a whole other tool outside the normal user workflow. Spack has over 7,000 packages at this point, so I don't think the lack of packages is going to hurt too much. We don't offer nearly as many public binaries yet, though, so the source builds may make the UX more painful, depending on what you're trying to do.

throw0101b 4y ago

> 6. Good support for building against external packages that may already be on your system. Able to use 'internal' packages (spack install openssl) or external packages (link against /usr/lib/libssl.so) on a per item basis: sometimes you want the newest code, but other times you want the OS package so that you can get security updates 'for free'. Most other system do one or the other: Spack allows you to choose.

gdevenyi 4y ago

lmod integration?

throw0101b 4y ago
Yes: * https://spack.readthedocs.io/en/latest/module_file_support.html https://spack.readthedocs.io/en/latest/module_file_support.h... Both Lmod and (TCL) Modules: can generate either/both.

yjftsjthsd-h 4y ago

Very neat:) Given the similarity in features, could you comment on how it compares to nix?

331c8c71 4y ago
+100 From what I saw the package definitions are in python. Given python is really an imperative language I don't see how spack could be comparably powerful for definitions and reuse.
- stabbles 4y ago
  The package definitions are quite declarative even though the language is imperative, see an example here [1]. For hashes, Spack uses the normalized (comments, whitespace etc removed) abstract syntax tree. With Spack it's easier to contribute, you don't have to learn a new language as you need for Nix / Guix. [1] https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/zstd/package.py#L9-L57 https://github.com/spack/spack/blob/develop/var/spack/repos/...
- yjftsjthsd-h 4y ago
  I could be persuaded to trade a certain amount of power for ease of use
- tgamblin 4y ago
  The Spack DSL for defining versions, options, etc. is declarative. That part of a package exposes options to the solver and very much allows reuse, in the sense that the same package can be built many different ways, and even with different dependencies. The whole system is parameterized in ways that Nix is not. The imperative part of a Spack package is the build recipe, which AFAICT is not so different from bundling a bash script in a Nix derivation. Just like Nix, it's included in our configuration hash, so if you change the recipe you get a different installation of the software.
  - 331c8c71 4y ago
    Thanks for the answer. > That part of a package exposes options ... in the sense that the same package can be built many different ways, and even with different dependencies. It is quite a common pattern in nix to have package build flags and the dependencies as inputs in a derivation. This definitely makes it possible to build in different ways and with different dependencies.

deleted 4y ago

[deleted]

eviks 4y ago

Compared to homebrew, do prebuilt binaries have to be installed in a predefined path they were built in or is it also flexible? Also, is there an indication/warning whether the package is prebuilt before you install?

tgamblin 4y ago
> do prebuilt binaries rerouted to be installed in a predefined path they were built in or is it also flexible? Prebuilt binaries don't have to go in a fixed prefix -- we'll relocate RPATHs, shebangs, strings in text files, and (as long as you build in a longer path than you install to -- see `padded_length` at https://spack.readthedocs.io/en/latest/config_yaml.html https://spack.readthedocs.io/en/latest/config_yaml.html) strings in binaries. There is not currently a way to see before you install if an install will be from binary or from source, but that is something we are working on.
- eviks 4y ago
  So how would I know whether the prebuilt binaries were built in a longer path than I install to? Is there at least a way to block source builds if the app is not in the binary cache to avoid any surprises (basically have manual override/confirmation instead of automatic fallbacks to source builds)?
  - tgamblin 4y ago
    > So how would I know whether the prebuilt binaries were built in a longer path than I install to? All the public ones are built with very long padding; if you make your own you're currently on your own. > Is there at least a way to block source builds if the app is not in the binary cache to avoid any surprises (basically have manual override/confirmation instead of automatic fallbacks to source builds)? Yep: spack install --cache-only ...

nerdponx 4y ago

What do you think about Spack vs. Conda/Mamba? I've been a contented (maybe not "happy") Conda user for years. Is there any reason to switch? I'd be concerned about package availability.

throw0101b 4y ago
Spack has Anaconda as a package: * https://spack.readthedocs.io/en/latest/package_list.html#anaconda3 https://spack.readthedocs.io/en/latest/package_list.html#ana... * https://spack.readthedocs.io/en/latest/replace_conda_homebrew.html https://spack.readthedocs.io/en/latest/replace_conda_homebre... So you can continue using it, but you also have a super-set of features for other things as well.
tgamblin 4y ago
We have a guide for this written by Adam Stewart, probably our most prolific (package) contributor: * https://spack.readthedocs.io/en/latest/replace_conda_homebrew.html https://spack.readthedocs.io/en/latest/replace_conda_homebre... The thing you'll likely notice most about Spack vs. conda/mamba/rattler(?) is that it lets you pick your versions and options much more fluidly, and it'll build you a package from source if there's no binary for it. Spack is much more seamless for integrating binary installations with source builds. `conda-build` is a whole other tool outside the normal user workflow. Spack has over 7,000 packages at this point, so I don't think the lack of packages is going to hurt too much. We don't offer nearly as many public binaries yet, though, so the source builds may make the UX more painful, depending on what you're trying to do.