4 ms·

I'd never buy a POWER11 system given that IBM barely supported our POWER9 cluster, and we ended up compiling everything ourselves for it, with many, many errors

I'd never buy a POWER11 system given that IBM barely supported our POWER9 cluster, and we ended up compiling everything ourselves for it, with many, many errors and problems. Any difference in cost is more than made up for with the amount of staff time spend trying to fix bizarre compiler bugs.

pabs3 5y ago
What did you have to compile? Most Linux distros have a fully supported ppc64el port.
- physicsguy 5y ago
  Pretty much every Python + scientific package (and these days, why else would you buy a P9 cluster but for that kind of work?)
  - pabs3 5y ago
    Some of the Linux distros (like Debian) already include that sort of stuff. For eg the Debian Science Team have packaged a lot of difference types of scientific software. The Debian Python modules team packages a lot of different Python modules too. https://blends.debian.org/science/tasks/ https://blends.debian.org/science/tasks/ https://wiki.debian.org/Teams/DebianScience https://wiki.debian.org/Teams/DebianScience Edit: of course a lot of more modern science and ML stuff is notoriously difficult to package, with pre-generated files, vendored code, lock files and so on.
    - physicsguy 5y ago
      Yeah, I am aware of that, but Linux distros package very old versions, and we need to be able to support multiple for users. Realistically, academic researchers like to use 'bleeding edge' type things. Generally a big HPC cluster comes with RHEL too and a support contract, because you need the drivers for the hardware - things like Infiniband networking, etc. so switching distro isn't an option really.
      - pabs3 5y ago
        Linux does seem to have some open source Infiniband drivers, but maybe not for the ones you own, or maybe the proprietary ones are faster. https://wiki.gentoo.org/wiki/InfiniBand https://wiki.gentoo.org/wiki/InfiniBand https://wiki.archlinux.org/title/InfiniBand https://wiki.archlinux.org/title/InfiniBand https://pve.proxmox.com/wiki/Infiniband https://pve.proxmox.com/wiki/Infiniband

pabs3 5y ago

What did you have to compile? Most Linux distros have a fully supported ppc64el port.

physicsguy 5y ago
Pretty much every Python + scientific package (and these days, why else would you buy a P9 cluster but for that kind of work?)
- pabs3 5y ago
  Some of the Linux distros (like Debian) already include that sort of stuff. For eg the Debian Science Team have packaged a lot of difference types of scientific software. The Debian Python modules team packages a lot of different Python modules too. https://blends.debian.org/science/tasks/ https://blends.debian.org/science/tasks/ https://wiki.debian.org/Teams/DebianScience https://wiki.debian.org/Teams/DebianScience Edit: of course a lot of more modern science and ML stuff is notoriously difficult to package, with pre-generated files, vendored code, lock files and so on.
  - physicsguy 5y ago
    Yeah, I am aware of that, but Linux distros package very old versions, and we need to be able to support multiple for users. Realistically, academic researchers like to use 'bleeding edge' type things. Generally a big HPC cluster comes with RHEL too and a support contract, because you need the drivers for the hardware - things like Infiniband networking, etc. so switching distro isn't an option really.
    - pabs3 5y ago
      Linux does seem to have some open source Infiniband drivers, but maybe not for the ones you own, or maybe the proprietary ones are faster. https://wiki.gentoo.org/wiki/InfiniBand https://wiki.gentoo.org/wiki/InfiniBand https://wiki.archlinux.org/title/InfiniBand https://wiki.archlinux.org/title/InfiniBand https://pve.proxmox.com/wiki/Infiniband https://pve.proxmox.com/wiki/Infiniband