2 ms·

Did switching to Red Hat Linux resolve your problems?

• TuringNYC 7y ago
  >> Did switching to Red Hat Linux resolve your problems? The core issue for random k80s dropping out of our cluster at random times was a power imbalance from the power supply to the cards, so technically no. However -- for NVIDIA to even diagnose the machine and figure out root causes, they required RHEL (Red Hat Enterprise Linux) before they would help. Secondly, the NVIDIA diagnostic utilities such as "GPU-Burn" were not supported on Ubuntu, so NVIDIA would not even take results from those if we were still running on Ubuntu. Also, consider the cost. This cluster was a $100k+ CapEx for a fledgling startup. We were not going to risk losing support on that for several hundred dollars of RHEL license costs. Finally, consider uptime. We purchased this complicated cluster because we were a healthcare ML startup and had locality restrictions associated with underlying medical data and had no Cloud Service Provider local nodes available in said country. If the server was down or -- heavens forbid, needed to be sent back for servicing -- we weren't training models and we were no longer making progress and burning thru valuable runway. The RHEL license was the a drop in the bucket.