6 ms·

Mapping Out the HPC Dependency Chaos

49 points by setheron 4y ago

linksnapzz 4y ago
This is a nice paper, and I do appreciate the authors' efforts. I fight with this stuff daily. But...this caught my eye: "While it’s a pleasant thought experiment to imagine a world where we do not require backwards compatibility with estab- lished loaders, the state of the practice is that we must work within the limitations of ELF and the System V ABI model." In light of the current HPC orthopraxis, I wonder if there might not be real value in seeing how far towards reality one could drag the thought experiment of not having to depend on that backwards compatibility. E.G. what sort of work abandoning the ELF/SysV ABI model might require. Dijkstra said something once about the problems of the real world being the ones that you're left with when you refuse to apply their effective solutions.
- convolvatron 4y ago
  I tried to remove the expectation that supercomputer applications would load their libraries dynamically and that became one of the single most hated aspects of that product line people really hate having their expectations violated - even if its an expectation of a fork in the eye
  - linksnapzz 4y ago
    Do tell; which product was this?
    - convolvatron 4y ago
      cray xt3. my understanding that in later versions Cray shipped compute node linux with some solution for maintaining the .so file on the nodes
      - linksnapzz 4y ago
        Interesting! I'd never touched a Catamount/SUNMOS system; didn't know that static everything was an option there, although some of my friends did work on those. Do you suppose the friction was due to it not being how their local workstation/previous system worked?
        convolvatron 4y ago
        yeah, mostly just 'what the hell'. I pretty sure it really didn't get in anyone's way - all the libraries were installed on the front end. it wasn't an option - we didn't support dynamic linking there
        trws 4y ago
        I never worked with the XT3, but I’ve worked with others with similar limitations. Many of our projects prefer static linking, but at any given time they tend to have to run on several platforms. As of today, they have to run on at least two where one won’t load gpu components in libraries correctly if they are statically linked, and one where it won’t if they’re dynamically linked. Forcing projects into one model or the other makes the smaller requirements on other platforms much harder to deal with, even if the requirement aligns with the project’s preference.
- trws 4y ago
  Another author here (Tom Scogland). We seriously talked about it, and are exploring some other loading models as part of subsequent work. The real trick is that this has all been neglected and effectively set for so many decades that if we do manage to find something better, it will take convincing some of the most conservative open source maintainers and code owners in existence to update the loader and c standard library to roll it out in a way most users would be able to leverage. We’re trying, and think we’ve found some really preferable alternatives and options, so here’s hoping.
  - linksnapzz 4y ago
    Neat! I look forward to hearing about yr. work!
37ef_ced3 4y ago
There are completely stand-alone systems like: https://NN-512.com https://NN-512.com The stand-alone code generator (a statically linked executable written in Go with no dependencies outside the Go standard library) generates stand-alone POSIX C code for the neural net, requiring only gcc to compile. Also see Fabrice Bellard's LibNC: https://bellard.org/libnc/ https://bellard.org/libnc/ C API. Small library, no external dependencies, available for Linux and Windows.
throw0101c 4y ago
This is why Spack was created: > Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments. It was designed for large supercomputing centers, where many users and application teams share common installations of software on clusters with exotic architectures, using libraries that do not have a standard ABI. Spack is non-destructive: installing a new version does not break existing installations, so many configurations can coexist on the same system. * https://spack.readthedocs.io/en/latest/ https://spack.readthedocs.io/en/latest/ > Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. Spack isn’t tied to a particular language; you can build a software stack in Python or R, link to libraries written in C, C++, or Fortran, and easily swap compilers or target specific microarchitectures. * https://spack.io https://spack.io Intro presentation from some HPC conferences: * https://www.youtube.com/watch?v=edpgwyOD79E https://www.youtube.com/watch?v=edpgwyOD79E * https://www.youtube.com/watch?v=DhUVbroMLJY https://www.youtube.com/watch?v=DhUVbroMLJY Very similar to (home)brew, MacPorts, etc, with more of a focus on HPC so you can have multiple versions of a particular piece of software, made with different compiles and linked against different library version.
- ajdecon 4y ago
  It’s worth noting that one of the paper’s authors is Todd Gamblin, the Spack project lead.
  - whinvik 4y ago
    Spack is really awesome. Made my development work during my PhD much easier.
  - throw0101c 4y ago
    Todd Gamblin seems to be on HN and is around in this thread: * https://news.ycombinator.com/item?id=33566807 https://news.ycombinator.com/item?id=33566807
- 0cf8612b2e1e 4y ago
  I have not personally used Spack, but a similar product in HPC, and I have often wondered why something analogous did not take off for desktop computing. We are finally seeing the rise of Nix, but the experience is currently so bad, I will give it a decade before it starts gaining serious traction. Sooner only if some big platform latches onto it.

49 points by setheron 4y ago

This is a nice paper, and I do appreciate the authors' efforts. I fight with this stuff daily. But...this caught my eye: "While it’s a pleasant thought experiment to imagine a world where we do not require backwards compatibility with estab- lished loaders, the state of the practice is that we must work within the limitations of ELF and the System V ABI model." In light of the current HPC orthopraxis, I wonder if there might not be real value in seeing how far towards reality one could drag the thought experiment of not having to depend on that backwards compatibility. E.G. what sort of work abandoning the ELF/SysV ABI model might require. Dijkstra said something once about the problems of the real world being the ones that you're left with when you refuse to apply their effective solutions.

convolvatron 4y ago
I tried to remove the expectation that supercomputer applications would load their libraries dynamically and that became one of the single most hated aspects of that product line people really hate having their expectations violated - even if its an expectation of a fork in the eye
- linksnapzz 4y ago
  Do tell; which product was this?
  - convolvatron 4y ago
    cray xt3. my understanding that in later versions Cray shipped compute node linux with some solution for maintaining the .so file on the nodes
    - linksnapzz 4y ago
      Interesting! I'd never touched a Catamount/SUNMOS system; didn't know that static everything was an option there, although some of my friends did work on those. Do you suppose the friction was due to it not being how their local workstation/previous system worked?
      - convolvatron 4y ago
        yeah, mostly just 'what the hell'. I pretty sure it really didn't get in anyone's way - all the libraries were installed on the front end. it wasn't an option - we didn't support dynamic linking there
      - trws 4y ago
        I never worked with the XT3, but I’ve worked with others with similar limitations. Many of our projects prefer static linking, but at any given time they tend to have to run on several platforms. As of today, they have to run on at least two where one won’t load gpu components in libraries correctly if they are statically linked, and one where it won’t if they’re dynamically linked. Forcing projects into one model or the other makes the smaller requirements on other platforms much harder to deal with, even if the requirement aligns with the project’s preference.
trws 4y ago
Another author here (Tom Scogland). We seriously talked about it, and are exploring some other loading models as part of subsequent work. The real trick is that this has all been neglected and effectively set for so many decades that if we do manage to find something better, it will take convincing some of the most conservative open source maintainers and code owners in existence to update the loader and c standard library to roll it out in a way most users would be able to leverage. We’re trying, and think we’ve found some really preferable alternatives and options, so here’s hoping.
- linksnapzz 4y ago
  Neat! I look forward to hearing about yr. work!

37ef_ced3 4y ago

There are completely stand-alone systems like: https://NN-512.com https://NN-512.com The stand-alone code generator (a statically linked executable written in Go with no dependencies outside the Go standard library) generates stand-alone POSIX C code for the neural net, requiring only gcc to compile. Also see Fabrice Bellard's LibNC: https://bellard.org/libnc/ https://bellard.org/libnc/ C API. Small library, no external dependencies, available for Linux and Windows.

throw0101c 4y ago

This is why Spack was created: > Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments. It was designed for large supercomputing centers, where many users and application teams share common installations of software on clusters with exotic architectures, using libraries that do not have a standard ABI. Spack is non-destructive: installing a new version does not break existing installations, so many configurations can coexist on the same system. * https://spack.readthedocs.io/en/latest/ https://spack.readthedocs.io/en/latest/ > Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. Spack isn’t tied to a particular language; you can build a software stack in Python or R, link to libraries written in C, C++, or Fortran, and easily swap compilers or target specific microarchitectures. * https://spack.io https://spack.io Intro presentation from some HPC conferences: * https://www.youtube.com/watch?v=edpgwyOD79E https://www.youtube.com/watch?v=edpgwyOD79E * https://www.youtube.com/watch?v=DhUVbroMLJY https://www.youtube.com/watch?v=DhUVbroMLJY Very similar to (home)brew, MacPorts, etc, with more of a focus on HPC so you can have multiple versions of a particular piece of software, made with different compiles and linked against different library version.

ajdecon 4y ago
It’s worth noting that one of the paper’s authors is Todd Gamblin, the Spack project lead.
- whinvik 4y ago
  Spack is really awesome. Made my development work during my PhD much easier.
- throw0101c 4y ago
  Todd Gamblin seems to be on HN and is around in this thread: * https://news.ycombinator.com/item?id=33566807 https://news.ycombinator.com/item?id=33566807
0cf8612b2e1e 4y ago
I have not personally used Spack, but a similar product in HPC, and I have often wondered why something analogous did not take off for desktop computing. We are finally seeing the rise of Nix, but the experience is currently so bad, I will give it a decade before it starts gaining serious traction. Sooner only if some big platform latches onto it.