3 ms·

Ubuntu has been officially supported on nvidia GPUs (cuda, really) for over 5 years. What did you need serviced? https://developer.nvidia.com/cuda-toolkit-60 h

Ubuntu has been officially supported on nvidia GPUs (cuda, really) for over 5 years. What did you need serviced? https://developer.nvidia.com/cuda-toolkit-60 https://developer.nvidia.com/cuda-toolkit-60

TuringNYC 7y ago
So yes, CUDA may have been supported for Ubuntu, but we had a 16x K-80 cluster which apparently needs more hand-holding. There were issues with cards dropping and becoming unavailable during use (turned out it was a slight power-imbalance from the power supply to the cards.) We encountered this within a month of purchase from an NVIDIA-listed Preferred/Authorized Enterprise Reseller. They couldnt fix the issue and escalated it to NVIDIA who promptly yelled at us for even trying to run Ubuntu. They also noted that if there were overheat-related damage they would refuse to honor the warranty. On a side note, apparently they were very upset at the Authorized Reseller and soon that reseller was no longer on the NVIDIA list of Authorized Resellers. On a side note, features like GPU-Direct were not even available on Ubuntu.
- pkaye 7y ago
  Did switching to Red Hat Linux resolve your problems?
  - TuringNYC 7y ago
    >> Did switching to Red Hat Linux resolve your problems? The core issue for random k80s dropping out of our cluster at random times was a power imbalance from the power supply to the cards, so technically no. However -- for NVIDIA to even diagnose the machine and figure out root causes, they required RHEL (Red Hat Enterprise Linux) before they would help. Secondly, the NVIDIA diagnostic utilities such as "GPU-Burn" were not supported on Ubuntu, so NVIDIA would not even take results from those if we were still running on Ubuntu. Also, consider the cost. This cluster was a $100k+ CapEx for a fledgling startup. We were not going to risk losing support on that for several hundred dollars of RHEL license costs. Finally, consider uptime. We purchased this complicated cluster because we were a healthcare ML startup and had locality restrictions associated with underlying medical data and had no Cloud Service Provider local nodes available in said country. If the server was down or -- heavens forbid, needed to be sent back for servicing -- we weren't training models and we were no longer making progress and burning thru valuable runway. The RHEL license was the a drop in the bucket.
- jijji 7y ago
  Exactly... The big reasons I've seen people using Red Hat is because most of the core Linux developers either currently work at Red Hat or have worked at Red Hat. So when serious complex issues arise (like in kernel drivers), the support is there and these issues get fixed. These same customers might have been using HP-UX, AIX or Solaris previously and have migrated to Red Hat because of the great support.

TuringNYC 7y ago

So yes, CUDA may have been supported for Ubuntu, but we had a 16x K-80 cluster which apparently needs more hand-holding. There were issues with cards dropping and becoming unavailable during use (turned out it was a slight power-imbalance from the power supply to the cards.) We encountered this within a month of purchase from an NVIDIA-listed Preferred/Authorized Enterprise Reseller. They couldnt fix the issue and escalated it to NVIDIA who promptly yelled at us for even trying to run Ubuntu. They also noted that if there were overheat-related damage they would refuse to honor the warranty. On a side note, apparently they were very upset at the Authorized Reseller and soon that reseller was no longer on the NVIDIA list of Authorized Resellers. On a side note, features like GPU-Direct were not even available on Ubuntu.

pkaye 7y ago
Did switching to Red Hat Linux resolve your problems?
- TuringNYC 7y ago
  >> Did switching to Red Hat Linux resolve your problems? The core issue for random k80s dropping out of our cluster at random times was a power imbalance from the power supply to the cards, so technically no. However -- for NVIDIA to even diagnose the machine and figure out root causes, they required RHEL (Red Hat Enterprise Linux) before they would help. Secondly, the NVIDIA diagnostic utilities such as "GPU-Burn" were not supported on Ubuntu, so NVIDIA would not even take results from those if we were still running on Ubuntu. Also, consider the cost. This cluster was a $100k+ CapEx for a fledgling startup. We were not going to risk losing support on that for several hundred dollars of RHEL license costs. Finally, consider uptime. We purchased this complicated cluster because we were a healthcare ML startup and had locality restrictions associated with underlying medical data and had no Cloud Service Provider local nodes available in said country. If the server was down or -- heavens forbid, needed to be sent back for servicing -- we weren't training models and we were no longer making progress and burning thru valuable runway. The RHEL license was the a drop in the bucket.
jijji 7y ago
Exactly... The big reasons I've seen people using Red Hat is because most of the core Linux developers either currently work at Red Hat or have worked at Red Hat. So when serious complex issues arise (like in kernel drivers), the support is there and these issues get fixed. These same customers might have been using HP-UX, AIX or Solaris previously and have migrated to Red Hat because of the great support.