6 ms·
Yes, I understand this argument. %CPU, as we know it, tells you OS CPU utilization (what I called non-idle time), and you can look at the kernel code and how it
by brendangregg 9y ago
Yes, I understand this argument. %CPU, as we know it, tells you OS CPU utilization (what I called non-idle time), and you can look at the kernel code and how it is calculated and conclude that it's not wrong. It is what it is. It's non-idle time.
Your point that we can't utilize those cycles, so therefore they are utilized... Well, I have a production instance running with an IPC of 0.11 (yes, seriously), and system-wide %CPU of 60, and idle hyperthreads. If I added a high IPC workload to those idle hyperthreads, do you think I could recover those stalled cycles? Probably. So were those cycles utilized or not? :-)
As for using CPU counters for free: sure, they basically are. Just instructions to read & configure (WRMSR, RDMSR, RDPMC), and the OS can do it on kernel context switch, so that they can be associated per-process. It'd cost very very little: adding a few instructions to each context switch, and some members to task_struct. I'd be a bit annoyed if the kernel started doing this everywhere as it'd tie up some PMCs (physical resource), so I'd rather that behavior be configurable.
- faragon 9y agoIt is not cheap enough, yet. Rationale: although it could be easy to account user processes, it would not be that easy for the OS kernel and kernel components (e.g. figure how to account cache pollution because of hardware IRQ dispatching and devices doing DMA...). So, except for, may be, micro-kernel OS's, and adding performance counters to every single device capable of bus-mastering, it is far from being a reality, in my opinion.
- jsmthrowaway 9y agoIt'd cost a little more than that, wouldn't it? I thought once the counters were configured there was a (potentially tiny?) overhead to any type of forward progress of RIP. You know way more about this than me, so glad to learn something if I'm wrong, and if you say it barely exists I'll be using far more of perf and other things like it... Maybe it's negligible enough that the data is worth the trade, anyway. Great and timely article, by the way, though it disappointingly lacks a video of you yelling at CPUs.
- nkurz 9y agoif you say it barely exists I'll be using far more of perf and other things like it... I'll back Brendan and claim that using hardware CPU counters on modern server processors is basically free. They are internal to the processor, and essentially don't utilize any resources shared with program execution. Maybe you could contrive a case where they cause faster thermal throttling, but I haven't seen it. For a simple count of events that is started and stopped at context switches (and when you are counting fewer events than available hardware counters) I'd be surprised if you even saw 1% overhead. If you are trying to multiplex many more events than you have physical counters, the overhead of "reprogramming" the counters might cost you single digit overhead. If you are trying to save all stack traces, record all branches, or monitor all memory accesses you can manage to create a more significant slowdown, but even then you can usually lower this something acceptable by sampling. So yes --- you probably should be using perf far more! Or VTune, or likwid (https://github.com/RRZE-HPC/likwid https://github.com/RRZE-HPC/likwid).
- amluto 9y ago> It'd cost very very little: adding a few instructions to each context switch, and some members to task_struct. RDPMC is fast. WRMSR is amazingly slow. Put another way, when you're using WRMSR, your IPC (CPU usage?) might be around .002.
- lamontcg 9y ago> it's not wrong. It is what it is. It's non-idle time. exactly what i was going to tell you. stick with that next time and stay away from 'wrong'.
- brendangregg 9y agoAnd %CPU reports busy when an _external_ component, DRAM, is actually busy, and CPU is waiting. So it's not "CPU utilized". That's wrong.