5 ms·

I respect Brendan, and although it is an interesting article, I have to disagree with him: The OS tells you about OS CPU utilization, not CPU micro-architecture

I respect Brendan, and although it is an interesting article, I have to disagree with him: The OS tells you about OS CPU utilization, not CPU micro-architecture functional unit utilization. So if the OS uses a CPU for running code until a physical interrupt or a software trap happens, in that period the CPU has been doing work. Unless the CPU could be able to do a "free" context switch to a cached area not having to wait for e.g. a cache miss (hint: SMT/"hyperthreading" was invented exactly for that use case), the CPU would be actually busy. If in the future (TM) using CPU performance counters for every process becomes really "free" (as in "gratis" or "cheap"), the OS could report bad performing processes because the reasons exposed in the article (low IPC indicating poor memory access patterns, unoptimized code, code using too small buffers for I/O -causing system performance degradation because excessive kernel processing time because-, etc.), showing the user that despite having high CPU usage, the CPU is not getting enough work done (in that sense I could agree with the article).

CogitoCogito 9y ago
Maybe the point from a user's perspective would be yours and from a developer's perspective would be his. Of course once the binary has been shipped to the user much can't be done there (except for things like hyperthreading as you say). However, a developer should not see 100% cpu as an indication that it can't be done faster and should realize that better algorithms and/or better memory access patterns can cause more work to be done even if both show 100% cpu. Of course I would argue that this should be obvious to all but novice programmers who worry about performance, but his article does paint the picture in a pretty clear way and will maybe help more realize what's really going on here.
- faragon 9y ago
  I'm a programmer. And I use the CPU performance counters for optimizing code, so go figure :-)
- deleted 9y ago
  [deleted]
brendangregg 9y ago
Yes, I understand this argument. %CPU, as we know it, tells you OS CPU utilization (what I called non-idle time), and you can look at the kernel code and how it is calculated and conclude that it's not wrong. It is what it is. It's non-idle time. Your point that we can't utilize those cycles, so therefore they are utilized... Well, I have a production instance running with an IPC of 0.11 (yes, seriously), and system-wide %CPU of 60, and idle hyperthreads. If I added a high IPC workload to those idle hyperthreads, do you think I could recover those stalled cycles? Probably. So were those cycles utilized or not? :-) As for using CPU counters for free: sure, they basically are. Just instructions to read & configure (WRMSR, RDMSR, RDPMC), and the OS can do it on kernel context switch, so that they can be associated per-process. It'd cost very very little: adding a few instructions to each context switch, and some members to task_struct. I'd be a bit annoyed if the kernel started doing this everywhere as it'd tie up some PMCs (physical resource), so I'd rather that behavior be configurable.
- faragon 9y ago
  It is not cheap enough, yet. Rationale: although it could be easy to account user processes, it would not be that easy for the OS kernel and kernel components (e.g. figure how to account cache pollution because of hardware IRQ dispatching and devices doing DMA...). So, except for, may be, micro-kernel OS's, and adding performance counters to every single device capable of bus-mastering, it is far from being a reality, in my opinion.
- jsmthrowaway 9y ago
  It'd cost a little more than that, wouldn't it? I thought once the counters were configured there was a (potentially tiny?) overhead to any type of forward progress of RIP. You know way more about this than me, so glad to learn something if I'm wrong, and if you say it barely exists I'll be using far more of perf and other things like it... Maybe it's negligible enough that the data is worth the trade, anyway. Great and timely article, by the way, though it disappointingly lacks a video of you yelling at CPUs.
  - nkurz 9y ago
    if you say it barely exists I'll be using far more of perf and other things like it... I'll back Brendan and claim that using hardware CPU counters on modern server processors is basically free. They are internal to the processor, and essentially don't utilize any resources shared with program execution. Maybe you could contrive a case where they cause faster thermal throttling, but I haven't seen it. For a simple count of events that is started and stopped at context switches (and when you are counting fewer events than available hardware counters) I'd be surprised if you even saw 1% overhead. If you are trying to multiplex many more events than you have physical counters, the overhead of "reprogramming" the counters might cost you single digit overhead. If you are trying to save all stack traces, record all branches, or monitor all memory accesses you can manage to create a more significant slowdown, but even then you can usually lower this something acceptable by sampling. So yes --- you probably should be using perf far more! Or VTune, or likwid (https://github.com/RRZE-HPC/likwid https://github.com/RRZE-HPC/likwid).
bjackman 9y ago
I came here to say this but since you've already said it, I'll also provide an on-the-other-hand! On the other hand, CPU utilisation is used to scale CPU frequency on some systems. This generally works OK but if a workload is spending most of its time blocking on the memory subsystem then scaling up the CPU frequency might (depending on the HW architecture) not actually speed it up at all, but just waste power. In that case the perf counters mentioned in the article might be a better metric for freq scaling. I think Intel CPUs usually do this already but not exactly sure. So yeah basically the utilisation metric isn't "wrong", you just have to understand what it actually means before using it.

CogitoCogito 9y ago

Maybe the point from a user's perspective would be yours and from a developer's perspective would be his. Of course once the binary has been shipped to the user much can't be done there (except for things like hyperthreading as you say). However, a developer should not see 100% cpu as an indication that it can't be done faster and should realize that better algorithms and/or better memory access patterns can cause more work to be done even if both show 100% cpu. Of course I would argue that this should be obvious to all but novice programmers who worry about performance, but his article does paint the picture in a pretty clear way and will maybe help more realize what's really going on here.

faragon 9y ago
I'm a programmer. And I use the CPU performance counters for optimizing code, so go figure :-)
deleted 9y ago
[deleted]

brendangregg 9y ago

Yes, I understand this argument. %CPU, as we know it, tells you OS CPU utilization (what I called non-idle time), and you can look at the kernel code and how it is calculated and conclude that it's not wrong. It is what it is. It's non-idle time. Your point that we can't utilize those cycles, so therefore they are utilized... Well, I have a production instance running with an IPC of 0.11 (yes, seriously), and system-wide %CPU of 60, and idle hyperthreads. If I added a high IPC workload to those idle hyperthreads, do you think I could recover those stalled cycles? Probably. So were those cycles utilized or not? :-) As for using CPU counters for free: sure, they basically are. Just instructions to read & configure (WRMSR, RDMSR, RDPMC), and the OS can do it on kernel context switch, so that they can be associated per-process. It'd cost very very little: adding a few instructions to each context switch, and some members to task_struct. I'd be a bit annoyed if the kernel started doing this everywhere as it'd tie up some PMCs (physical resource), so I'd rather that behavior be configurable.

faragon 9y ago
It is not cheap enough, yet. Rationale: although it could be easy to account user processes, it would not be that easy for the OS kernel and kernel components (e.g. figure how to account cache pollution because of hardware IRQ dispatching and devices doing DMA...). So, except for, may be, micro-kernel OS's, and adding performance counters to every single device capable of bus-mastering, it is far from being a reality, in my opinion.
jsmthrowaway 9y ago
It'd cost a little more than that, wouldn't it? I thought once the counters were configured there was a (potentially tiny?) overhead to any type of forward progress of RIP. You know way more about this than me, so glad to learn something if I'm wrong, and if you say it barely exists I'll be using far more of perf and other things like it... Maybe it's negligible enough that the data is worth the trade, anyway. Great and timely article, by the way, though it disappointingly lacks a video of you yelling at CPUs.
- nkurz 9y ago
  if you say it barely exists I'll be using far more of perf and other things like it... I'll back Brendan and claim that using hardware CPU counters on modern server processors is basically free. They are internal to the processor, and essentially don't utilize any resources shared with program execution. Maybe you could contrive a case where they cause faster thermal throttling, but I haven't seen it. For a simple count of events that is started and stopped at context switches (and when you are counting fewer events than available hardware counters) I'd be surprised if you even saw 1% overhead. If you are trying to multiplex many more events than you have physical counters, the overhead of "reprogramming" the counters might cost you single digit overhead. If you are trying to save all stack traces, record all branches, or monitor all memory accesses you can manage to create a more significant slowdown, but even then you can usually lower this something acceptable by sampling. So yes --- you probably should be using perf far more! Or VTune, or likwid (https://github.com/RRZE-HPC/likwid https://github.com/RRZE-HPC/likwid).

bjackman 9y ago

I came here to say this but since you've already said it, I'll also provide an on-the-other-hand! On the other hand, CPU utilisation is used to scale CPU frequency on some systems. This generally works OK but if a workload is spending most of its time blocking on the memory subsystem then scaling up the CPU frequency might (depending on the HW architecture) not actually speed it up at all, but just waste power. In that case the perf counters mentioned in the article might be a better metric for freq scaling. I think Intel CPUs usually do this already but not exactly sure. So yeah basically the utilisation metric isn't "wrong", you just have to understand what it actually means before using it.