4 ms·

I'm surprised that no one here has talked about the build times yet. As a new user, the first thing you notice out of the box is that even on a relatively beefy

I'm surprised that no one here has talked about the build times yet. As a new user, the first thing you notice out of the box is that even on a relatively beefy system, you spend ~2 hours getting to a working state, even when you're building a relatively modest set of packages. This is particularly surprising since one of the points of Spack is to integrate with the existing system software, and so much of what you're building already exists and works fine on the system. In the past when I've talked to people about this, the answer I got was "well, set it up once for your team and don't reinstall it all the time". Which is kind of awful from a "cattle, not pets" perspective. Several other design decisions in Spack (like the lack of certain kinds of configuration isolation by default, and the general difficulty of configuring certain kinds of things) make it feel like a "pets" system. I've gotten instructions from people on using Spack to install things that have been impossible to follow because they fiddled with something in their environment and forgot to tell me. The best thing I can say about Spack is that all the other systems are worse. I just wish they could get around to fixing some of this stuff already.

EntrePrescott 4y ago
> like the lack of certain kinds of configuration isolation by default could you please explain a little what kind of "configuration isolation" you mean? > and the general difficulty of configuring certain kinds of things like what for example?
- eslaught 4y ago
  > could you please explain a little what kind of "configuration isolation" you mean? It's been a while so I don't have the whole list off the top of my head, but there are several configuration files that live in $HOME by default unless you set SPACK_DISABLE_LOCAL_CONFIG=1. Those files can e.g., list compilers and software packages, if I recall, among various other settings. > like what for example? I was trying to follow someone's recipe for building something with Spack. I hit some sort of compiler error (sorry, it's been too long, don't remember). And the solution they suggested was to modify compilers.yml by hand to set various things. (Something about setting the right paths for using Cray's compiler wrappers with rocmcc...) That and I had to fuss with packages.yml to configure the system MPI. I remember trying to figure out how use "spack config set" to configure these files automatically, but the syntax was (a) completely undocumented and (b) not actually powerful enough in practice to configure either of those files. So I gave up on that, set SPACK_DISABLE_LOCAL_CONFIG and just saved the modified yaml files into my repository so that I can at least recover my settings if I need them in the future. For advanced use cases, it seems like you basically need to modify these yaml files to get anything done, but the amount of documentation and support for doing this automatically is really lacking. (And the idea that you're supposed to require every user to do this manually is even more insane.)
rekado 4y ago
Guix has a different take on this problem: it provides package transformations to modify the build graph on the command line (or in a Scheme file), which don't preclude the use of binary substitutes and reproducible builds. All stuff is cached in /gnu/store so people won't have to rebuild packages, and with `guix publish` people can share built binaries with others (discovery via Avahi optional).
trws 4y ago
> This is particularly surprising since one of the points of Spack is to integrate with the existing system software, and so much of what you're building already exists and works fine on the system. Try spack external find, and much of this problem goes away. It will register the existing packages as externals. As to the rest, spack actually has public binary mirrors with builds of many packages for common distributions now, in addition to the E4S mirror. It may not be as easy to ensure use of these as would be ideal, but this situation has gotten much, much better over the last few years.
- eslaught 4y ago
  > Try spack external find, and much of this problem goes away. It will register the existing packages as externals. To be clear, this goes through a hard-coded list of packages, and adds them to Spack. It cuts maybe about 20-30 minutes off your 2+ hour build time. And it doesn't find any libraries (MPI being the most obvious one you'd want). So, better than nothing, but far from what it needs to be. > public binary mirrors Is that on by default? I went through this only a couple of months ago and I don't think any of my packages came from mirrors. One practical challenge in using this in practice is that you definitely want some of your packages built with the local compilers, with maximally native settings (so native MPI, march=native, etc.). Other packages you may not care about at all (autoconf, etc.). So while I believe that Spack allows you to do that, it doesn't necessarily make it easy to do. I suspect when you tell Spack to build something with a certain compiler, it just builds everything from scratch because it can't tell where you actually care about getting the most tuned code.
  - slavik81 4y ago
    > And it doesn't find any libraries There is some support for libraries with spack external find. For example, most of the ROCm libraries can be used that way.

EntrePrescott 4y ago

> like the lack of certain kinds of configuration isolation by default could you please explain a little what kind of "configuration isolation" you mean? > and the general difficulty of configuring certain kinds of things like what for example?

eslaught 4y ago
> could you please explain a little what kind of "configuration isolation" you mean? It's been a while so I don't have the whole list off the top of my head, but there are several configuration files that live in $HOME by default unless you set SPACK_DISABLE_LOCAL_CONFIG=1. Those files can e.g., list compilers and software packages, if I recall, among various other settings. > like what for example? I was trying to follow someone's recipe for building something with Spack. I hit some sort of compiler error (sorry, it's been too long, don't remember). And the solution they suggested was to modify compilers.yml by hand to set various things. (Something about setting the right paths for using Cray's compiler wrappers with rocmcc...) That and I had to fuss with packages.yml to configure the system MPI. I remember trying to figure out how use "spack config set" to configure these files automatically, but the syntax was (a) completely undocumented and (b) not actually powerful enough in practice to configure either of those files. So I gave up on that, set SPACK_DISABLE_LOCAL_CONFIG and just saved the modified yaml files into my repository so that I can at least recover my settings if I need them in the future. For advanced use cases, it seems like you basically need to modify these yaml files to get anything done, but the amount of documentation and support for doing this automatically is really lacking. (And the idea that you're supposed to require every user to do this manually is even more insane.)

rekado 4y ago

Guix has a different take on this problem: it provides package transformations to modify the build graph on the command line (or in a Scheme file), which don't preclude the use of binary substitutes and reproducible builds. All stuff is cached in /gnu/store so people won't have to rebuild packages, and with `guix publish` people can share built binaries with others (discovery via Avahi optional).

trws 4y ago

> This is particularly surprising since one of the points of Spack is to integrate with the existing system software, and so much of what you're building already exists and works fine on the system. Try spack external find, and much of this problem goes away. It will register the existing packages as externals. As to the rest, spack actually has public binary mirrors with builds of many packages for common distributions now, in addition to the E4S mirror. It may not be as easy to ensure use of these as would be ideal, but this situation has gotten much, much better over the last few years.

eslaught 4y ago
> Try spack external find, and much of this problem goes away. It will register the existing packages as externals. To be clear, this goes through a hard-coded list of packages, and adds them to Spack. It cuts maybe about 20-30 minutes off your 2+ hour build time. And it doesn't find any libraries (MPI being the most obvious one you'd want). So, better than nothing, but far from what it needs to be. > public binary mirrors Is that on by default? I went through this only a couple of months ago and I don't think any of my packages came from mirrors. One practical challenge in using this in practice is that you definitely want some of your packages built with the local compilers, with maximally native settings (so native MPI, march=native, etc.). Other packages you may not care about at all (autoconf, etc.). So while I believe that Spack allows you to do that, it doesn't necessarily make it easy to do. I suspect when you tell Spack to build something with a certain compiler, it just builds everything from scratch because it can't tell where you actually care about getting the most tuned code.
- slavik81 4y ago
  > And it doesn't find any libraries There is some support for libraries with spack external find. For example, most of the ROCm libraries can be used that way.