4 ms·

I’m naive about HPC, but it’s always surprised me a bit that HPC software seems to be a fairly distinct ecosystem. I’m not doubting it’s for good reasons, but w

by klysm 4y ago

I’m naive about HPC, but it’s always surprised me a bit that HPC software seems to be a fairly distinct ecosystem. I’m not doubting it’s for good reasons, but why is a special package manager needed for this domain?

tgamblin 4y ago
HPC isn't exactly special, and neither to some extent is Spack. Spack's gotten some attention in talks at, e.g., CppCon. The big differentiator is really the degree to which people want to tune and customize their builds in HPC vs. other communities, and the diversity of hardware that needs to be supported. Things that stand out to me: - different applications' needs to customize the builds of their dependencies - e.g., one app might need HDF5 with MPI support, another might want it built without. Those are two different, incompatible HDF5 builds. - need for specific microarchitecture builds to take advantage of vectorization - need for GPU support (for NVIDIA, AMD, *and* Intel GPUs) - need to use sometimes vendor-specific tuned system libraries like MPI, cray-libsci, mol - need for specific *versions* of dependencies (solvers, mesh libs, etc.) for numerical reproducibility - need to integrate across languages, e.g. C, C++, Fortran, Python, Lua, perl, R, and dare I even say Yorick. Most of these requirements are not so dissimilar from peoples' dev environments in, say, the AI community, where people really want their special version of PyTorch. Where monorepos are common in industry, they really haven't taken off in the distributed, worldwide scientific community, so you get things like Spack that let you keep rebuilding the world -- and all the microcosms in it. So I'd say Spack is not so much a special package manager as a much more general one. You can use it for combinatorial deployments at HPC centers, dev workflows for people dealing with multi-physics and other complex codes, and as sort of a distributed poly repo with lock files. The intent was never to be specific to HPC, and I would love to see broader adoption outside this community.
- klysm 4y ago
  Thanks for the thoughtful response. That makes a lot of sense and these capabilities are not something you'll find in debian packages. It's not easy to ask for different compilation flags there.
eloop 4y ago
First paragraph of the docs - "Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments. It was designed for large supercomputing centers, where many users and application teams share common installations of software on clusters with exotic architectures, using libraries that do not have a standard ABI. Spack is non-destructive: installing a new version does not break existing installations, so many configurations can coexist on the same system."
gnufx 4y ago
The need for HPC specifics is definitely oversold, if not by Spack. One thing is the need for proper package management for unprivileged users if they don't use container messes, and the fact that there's so much stuff to share recipes for. It would help if there was more attention to performance engineering, even engineering generally, like having the sort of dynamic micro-architecture selection that BLAS libraries typically do, or attention to library substitution via dynamic linking (e.g. BLAS again).

by klysm 4y ago

tgamblin 4y ago

HPC isn't exactly special, and neither to some extent is Spack. Spack's gotten some attention in talks at, e.g., CppCon. The big differentiator is really the degree to which people want to tune and customize their builds in HPC vs. other communities, and the diversity of hardware that needs to be supported. Things that stand out to me: - different applications' needs to customize the builds of their dependencies - e.g., one app might need HDF5 with MPI support, another might want it built without. Those are two different, incompatible HDF5 builds. - need for specific microarchitecture builds to take advantage of vectorization - need for GPU support (for NVIDIA, AMD, *and* Intel GPUs) - need to use sometimes vendor-specific tuned system libraries like MPI, cray-libsci, mol - need for specific *versions* of dependencies (solvers, mesh libs, etc.) for numerical reproducibility - need to integrate across languages, e.g. C, C++, Fortran, Python, Lua, perl, R, and dare I even say Yorick. Most of these requirements are not so dissimilar from peoples' dev environments in, say, the AI community, where people really want their special version of PyTorch. Where monorepos are common in industry, they really haven't taken off in the distributed, worldwide scientific community, so you get things like Spack that let you keep rebuilding the world -- and all the microcosms in it. So I'd say Spack is not so much a special package manager as a much more general one. You can use it for combinatorial deployments at HPC centers, dev workflows for people dealing with multi-physics and other complex codes, and as sort of a distributed poly repo with lock files. The intent was never to be specific to HPC, and I would love to see broader adoption outside this community.

klysm 4y ago
Thanks for the thoughtful response. That makes a lot of sense and these capabilities are not something you'll find in debian packages. It's not easy to ask for different compilation flags there.

eloop 4y ago

First paragraph of the docs - "Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments. It was designed for large supercomputing centers, where many users and application teams share common installations of software on clusters with exotic architectures, using libraries that do not have a standard ABI. Spack is non-destructive: installing a new version does not break existing installations, so many configurations can coexist on the same system."

gnufx 4y ago

The need for HPC specifics is definitely oversold, if not by Spack. One thing is the need for proper package management for unprivileged users if they don't use container messes, and the fact that there's so much stuff to share recipes for. It would help if there was more attention to performance engineering, even engineering generally, like having the sort of dynamic micro-architecture selection that BLAS libraries typically do, or attention to library substitution via dynamic linking (e.g. BLAS again).