I was recently in a similar situation having to first investigate how (various) GPU utilization metrics are defined, and then I also had to explain this informa

I was recently in a similar situation having to first investigate how (various) GPU utilization metrics are defined, and then I also had to explain this information to someone who had the task of representing GPU-related metrics to users. Both turned out to be quite difficult... First had to do with NVML (the library behind nvidia-smi), which isn't very well documented... but I had some help there. But, even after sort of making a mental image of available metrics, the task of presenting users with actionable statistics turned out to be also very hard. On top of what OP writes, imagine now the added modality of having a hierarchical structure of PU workers, some of which, are not even hardware... and, subsequently, the stalls resulting from bad data alignment rather than waiting on physical processes to complete. Add to this yet another modality of vRAM occupancy and its bandwidth utilization... And if you want to show a single number that tells users how loaded their system is... what do you do? Average? Sum? Weighted sum? What if you wanted some absolute units rather than percents? When the data doesn't properly align (to fill up all the SIMD lanes) should this be counted towards "bad" GPU performance or just assumed to be the nature of the workload? Interestingly, the "utilization" nvidia-smi reports is about time, not the number of instructions executed. And it's about time that at least one SP wasn't marked as idle. This stuff becomes even more fuzzy, when eg. NVLink is involved... it's really hard to give concise information to not extremely technical users that'd be actionable from their perspective.