3 ms·

Several years ago, when I was CTO of an ML startup, we got slapped on the wrist by NVIDIA -- NVIDIA refused to even service our [very expensive GPU cluster] unl

Several years ago, when I was CTO of an ML startup, we got slapped on the wrist by NVIDIA -- NVIDIA refused to even service our [very expensive GPU cluster] unless we used Red Hat Linux. Apparently none of the Ubuntu releases were supported distros. So we purchased Red Hat Linux, the exact version NVIDIA mandated for support, and used that, just so we could get a warranty supported machine. I did learn that a whole lot of "scientific computing" happens on Red Hat or CentOS and not debian distros, though I'm not sure why. This was 4yrs ago, perhaps things have changed now.

izacus 7y ago
Red Hat (and by extension Centos) is pretty much the only thing I see in larger installations like CERN and other big industry. It's mostly connected to the fact that the distribution has a very long paid support for security patches and other bug fixes without demanding whole distribution upgrades. You don't want your proprietary medical/science software to break on your whole cluster just because you had to upgrade whole Debian to fix a security patch. Red Hat is prepared to maintain their distribution for far longer than pretty much anyone else outside Microsoft.
- TuringNYC 7y ago
  Yep, I ended up speaking with HPC folks running big NVIDIA GPU rigs at Oak Ridge and they also confirmed, there is no realistic option aside from Red Hat.
shaklee3 7y ago
Ubuntu has been officially supported on nvidia GPUs (cuda, really) for over 5 years. What did you need serviced? https://developer.nvidia.com/cuda-toolkit-60 https://developer.nvidia.com/cuda-toolkit-60
- TuringNYC 7y ago
  So yes, CUDA may have been supported for Ubuntu, but we had a 16x K-80 cluster which apparently needs more hand-holding. There were issues with cards dropping and becoming unavailable during use (turned out it was a slight power-imbalance from the power supply to the cards.) We encountered this within a month of purchase from an NVIDIA-listed Preferred/Authorized Enterprise Reseller. They couldnt fix the issue and escalated it to NVIDIA who promptly yelled at us for even trying to run Ubuntu. They also noted that if there were overheat-related damage they would refuse to honor the warranty. On a side note, apparently they were very upset at the Authorized Reseller and soon that reseller was no longer on the NVIDIA list of Authorized Resellers. On a side note, features like GPU-Direct were not even available on Ubuntu.
  - pkaye 7y ago
    Did switching to Red Hat Linux resolve your problems?
    - TuringNYC 7y ago
      >> Did switching to Red Hat Linux resolve your problems? The core issue for random k80s dropping out of our cluster at random times was a power imbalance from the power supply to the cards, so technically no. However -- for NVIDIA to even diagnose the machine and figure out root causes, they required RHEL (Red Hat Enterprise Linux) before they would help. Secondly, the NVIDIA diagnostic utilities such as "GPU-Burn" were not supported on Ubuntu, so NVIDIA would not even take results from those if we were still running on Ubuntu. Also, consider the cost. This cluster was a $100k+ CapEx for a fledgling startup. We were not going to risk losing support on that for several hundred dollars of RHEL license costs. Finally, consider uptime. We purchased this complicated cluster because we were a healthcare ML startup and had locality restrictions associated with underlying medical data and had no Cloud Service Provider local nodes available in said country. If the server was down or -- heavens forbid, needed to be sent back for servicing -- we weren't training models and we were no longer making progress and burning thru valuable runway. The RHEL license was the a drop in the bucket.
  - jijji 7y ago
    Exactly... The big reasons I've seen people using Red Hat is because most of the core Linux developers either currently work at Red Hat or have worked at Red Hat. So when serious complex issues arise (like in kernel drivers), the support is there and these issues get fixed. These same customers might have been using HP-UX, AIX or Solaris previously and have migrated to Red Hat because of the great support.
vermontdevil 7y ago
Indiana University uses Red Hat for Carbonate large memory cluster. I’m sure many universities still use Red Hat for various projects like this. https://kb.iu.edu/d/aolp https://kb.iu.edu/d/aolp
- jlgaddis 7y ago
  IU uses RHEL quite extensively, for a variety of projects. Many years ago, Gentoo was actually very widely used (much to my surprise). Once IU got a site license for RHEL -- one of the first .edu's to do so, IIRC -- they pretty much standardized on that. In addition to all "official" University systems, their RHEL license also covers personal systems for faculty, staff, and even students (my first "real" RHEL boxes were registered to their newfangled Satellite server).
  - vermontdevil 7y ago
    Ironically they are using some sort of Clay Linux and SUSE for the new Big Red III

izacus 7y ago

Red Hat (and by extension Centos) is pretty much the only thing I see in larger installations like CERN and other big industry. It's mostly connected to the fact that the distribution has a very long paid support for security patches and other bug fixes without demanding whole distribution upgrades. You don't want your proprietary medical/science software to break on your whole cluster just because you had to upgrade whole Debian to fix a security patch. Red Hat is prepared to maintain their distribution for far longer than pretty much anyone else outside Microsoft.

TuringNYC 7y ago
Yep, I ended up speaking with HPC folks running big NVIDIA GPU rigs at Oak Ridge and they also confirmed, there is no realistic option aside from Red Hat.

shaklee3 7y ago

Ubuntu has been officially supported on nvidia GPUs (cuda, really) for over 5 years. What did you need serviced? https://developer.nvidia.com/cuda-toolkit-60 https://developer.nvidia.com/cuda-toolkit-60

TuringNYC 7y ago
So yes, CUDA may have been supported for Ubuntu, but we had a 16x K-80 cluster which apparently needs more hand-holding. There were issues with cards dropping and becoming unavailable during use (turned out it was a slight power-imbalance from the power supply to the cards.) We encountered this within a month of purchase from an NVIDIA-listed Preferred/Authorized Enterprise Reseller. They couldnt fix the issue and escalated it to NVIDIA who promptly yelled at us for even trying to run Ubuntu. They also noted that if there were overheat-related damage they would refuse to honor the warranty. On a side note, apparently they were very upset at the Authorized Reseller and soon that reseller was no longer on the NVIDIA list of Authorized Resellers. On a side note, features like GPU-Direct were not even available on Ubuntu.
- pkaye 7y ago
  Did switching to Red Hat Linux resolve your problems?
  - TuringNYC 7y ago
    >> Did switching to Red Hat Linux resolve your problems? The core issue for random k80s dropping out of our cluster at random times was a power imbalance from the power supply to the cards, so technically no. However -- for NVIDIA to even diagnose the machine and figure out root causes, they required RHEL (Red Hat Enterprise Linux) before they would help. Secondly, the NVIDIA diagnostic utilities such as "GPU-Burn" were not supported on Ubuntu, so NVIDIA would not even take results from those if we were still running on Ubuntu. Also, consider the cost. This cluster was a $100k+ CapEx for a fledgling startup. We were not going to risk losing support on that for several hundred dollars of RHEL license costs. Finally, consider uptime. We purchased this complicated cluster because we were a healthcare ML startup and had locality restrictions associated with underlying medical data and had no Cloud Service Provider local nodes available in said country. If the server was down or -- heavens forbid, needed to be sent back for servicing -- we weren't training models and we were no longer making progress and burning thru valuable runway. The RHEL license was the a drop in the bucket.
- jijji 7y ago
  Exactly... The big reasons I've seen people using Red Hat is because most of the core Linux developers either currently work at Red Hat or have worked at Red Hat. So when serious complex issues arise (like in kernel drivers), the support is there and these issues get fixed. These same customers might have been using HP-UX, AIX or Solaris previously and have migrated to Red Hat because of the great support.

vermontdevil 7y ago

Indiana University uses Red Hat for Carbonate large memory cluster. I’m sure many universities still use Red Hat for various projects like this. https://kb.iu.edu/d/aolp https://kb.iu.edu/d/aolp

jlgaddis 7y ago
IU uses RHEL quite extensively, for a variety of projects. Many years ago, Gentoo was actually very widely used (much to my surprise). Once IU got a site license for RHEL -- one of the first .edu's to do so, IIRC -- they pretty much standardized on that. In addition to all "official" University systems, their RHEL license also covers personal systems for faculty, staff, and even students (my first "real" RHEL boxes were registered to their newfangled Satellite server).
- vermontdevil 7y ago
  Ironically they are using some sort of Clay Linux and SUSE for the new Big Red III