4 ms·

Poor GPU utilization even when available is the rule. Truly amazing. Staging of data is probably a huge part of it.

• mirker 4y ago
  Is it failures or is this some backfill/budget scheduling while everyone is sleeping?
  • foobiekr 4y ago
    A lot of it appears to be non-streaming approaches to data distribution resulting in actual job behavior that looks a lot more like stage-process-clear batch jobs than what you'd want to hide the latency of data moves.
• pavelstoev 4y ago
  At CentML, we profiled GPU utilization on a larger AI/ML research institute cluster. 10% to 45% range, mostly in 10% utilization range. We then offered them software optimizers (which do not affect model accuracy) to get to the 90% utilization for GPUs
  • foobiekr 4y ago
    90% sustained utilization is quite amazing, and 10% is shockingly typical. I am a quite skeptical that this holds for training and very large data sets, of the sort where data placement comes into play, but if so, congratulations, and I hope things go well for you.