8 ms·

CPU Utilization Is Wrong (2017)

94 points by xk3 3y ago

brucethemoose2 3y ago
SMT is pitched as a good way to "hide" low IPC loads by giving the core something else to do while waiting for memory. Hence IBM and (very briefly) Marvell both shipped 4-way SMT cores for database-heavy workloads. AFAIK this further complicates the IPC metric. What looks like a stalled core might actually be working on another thread. And at the other extreme, two very high IPC threads on one core would lower the observed IPC of each thread.
- hinkley 3y ago
  I was always curious why they didn't just make pairs of CPUs that shared a few logic units in common. Like I don't know what the instruction mix is these days and just how long the pipeline is, but say most code can do 2-3 ALU instructions per clock, why not 2 cores with 1-2 ALUs each and 2 shared? As things are, there is very little guarantee that the hyperthread will make any substantial progress.
  - csherratt 3y ago
    AMD Bulldozer did something like this. Ultraspac T1 had a shared FPU too. There is are a savings there, but neither of these solutions ended up being practically fast. The amount of die space spent on an ALU is relatively low compared to the rest of the CPU, hence why they shared FPUs instread. We might see it again now that the server world is having an ARM CPU renaissance. But I doubt AMD or Intel will make anything that exotic.
    - sweetjuly 3y ago
      I sort of hope we don't see SMT ARM cores. SMT was/is a huge pain for side channels and all the ARM chip houses dodged that one by nature of just never implementing it. I would hope that the concern of cloud vendors over isolation would be enough to discourage bringing more SMT uarchs in to this world.
      - ilyt 3y ago
        Doesn't just giving a VM both threads of the core mitigate that ?
        d_tr 3y ago
        Or just disabling SMT? SMT can give a good utilization boost and it feels like a shame for it not to be an option, especially since CPUs are not getting faster at the same rate they used to anymore. You can disable it on AMD and Intel x86 cores, as well as on the IBM Power ones.
        ilyt 3y ago
        but you're throwing performance away. If you don't want cross-customer leaks just give each customer both SMT threads of the core
        hinkley 3y ago
        Citation needed. Generation after generation, hyperthreads on almost every generation of Intel core have turned in mediocre to terrible benchmarks. Neither thread makes full progress , and on some versions you would see more IPC with them turned off. AMD might have managed better, though I see benchmarks where theirs is also a wash. It doesn’t mean Arm or RISC-V will see any benefit.
        ilyt 3y ago
        I've seen around 20% increase in even dumb "a web app returning some text" benchmarks. It certainly shouldt be charged as a second real core would but it isn't nothing (like in P4 era you apparently time travelled from) But that's entirely beside the point and I'm not sure why you try to derail the conversation into different direciton.
        brucethemoose2 3y ago
        Y'all are talking about seperate things, I think. SMT often does boost database heavy "cloud" workloads, but not client workloads like media processing, games, work apps and so on. We have it on desktops/laptops because they use hand-me-down server cores. Its not worth redesigning the cores to bifurcate the architecture.
  - gpderetta 3y ago
    ALUs[1] do not really consume that large of an area of a CPU so just sharing those is a significant complication with little benefits. Sharing register files, caches, store buffers, ROBs, TLBs, prediction tables, and the logic to drive them is a better trade-off. [1] outside of large SIMD FPUs, which indeed bulldozer tried to share with not great results.
- BackBlast 3y ago
  Except that you've just split your available cache region between multiple processes. This can actually decrease performance, I remember cache thrashing back in the P4 days that could cause an application like MS Word to take minutes to load. One process evicting data the other one needs and vice versa. It circumstantially makes the situation worse instead of better. Turn off hyper-threading and boom, everything started to function normally. The solution to not having enough food for one mouth often isn't to add another mouth. I still see this in modern benchmarks on modern systems where multicore throughput on many benchmarks is just better when you just turn off SMT. AMD's 3D chips go faster because of the huge cache. Every time we add another process to a core we just split the cache in half instead. Clearly, this is workload dependent. But I immediately turn off SMT on new systems I get. I'm not core limited, I'm usually single thread limited. Like, always. No reason to keep SMT on for this case.
hinkley 3y ago
How long has it been the case that a CPU would basically catch on fire if it could run without pipeline stalls slowing down the thermal dissipation?
- sweetjuly 3y ago
  Stalls are not necessary for thermal regulation, DVFS [1] is the tool we actually use for handling thermal limits (generally referred to as "thermal throttling"). In a modern CPU, you really don't expect (or, rather, want) to be fully stalled ever, that's the point of very deep reorder buffers. You can try this yourself, make a long loop with a ton of unrelated int and fp operations (unrelated so as not to cause dependency chains) and you'll get 100% occupancy without the core catching on fire. [1] https://en.wikipedia.org/wiki/Dynamic_frequency_scaling https://en.wikipedia.org/wiki/Dynamic_frequency_scaling
kaelinl 3y ago
(2017)
nubinetwork 3y ago
Why use top when you can use htop? It breaks down CPU usage into various parts like IO wait and vm guest time... they even integrated functionality from iotop so you eventually won't need that program either.
- pavon 3y ago
  The "Detailed CPU time" setting in htop doesn't show the time the CPU is waiting on memory, it is just traditional I/O (disk, network, etc).
  - oriettaxx 3y ago
    he is suggesting the use of `tiptop`
sweetjuly 3y ago
Hmm, I feel like this is a bit too big of a generalization to be useful. It's a cool observation but I don't think you can interpet IPC so directly as to conclude that IPC < 1 indicates "you are likely memory stalled". There are a million reasons why you'd have an IPC < 1, many of them which are not memory. Maybe you just have a lot of hard to predict branches which take a long time to resolve (all that mispeculated work is wasted cycles!), maybe you use a lot of long latency FP operations, maybe you just have really long dependency chains. If your core doesn't have full speculative wakeups for variable latency instructions (meaning things like loads but often things like division and multiplication have early exits), you can very easily end up with plenty of cycles where you don't complete an instruction. And, yes, of course, it could be that you're having a lot of cache misses. Memory is a good guess but it's not really actionable advice for performance analysis, you really need to pull out some proper tools to understand why you're getting that number. I think this is a cool article in that, yes, Task Manager/top/Activity Monitor aren't telling you the full story when it comes to what "CPU %" means. In the end though, there really isn't an easy way to come up with a better metric for utilization that can be summarized in one number, so realistically "how much time did the scheduler put this process on a core for" is plenty good enough for most purposes.
- failuser 3y ago
  Memory is just the most common. Most software does not involve complicated calculations to get to the other reasons.
- cthalupa 3y ago
  >Memory is a good guess but it's not really actionable advice for performance analysis, you really need to pull out some proper tools to understand why you're getting that number. I don't really understand and by extension agree with this point. If you're doing performance analysis on a regular basis, having more readily available tools that give you a solid guess at where you want to look next is useful. It's a lot easier for me to get pcm up and running on any random system than it is to dig in with vtune, for example, and if I can get a fairly accurate determination that pcm and pcm-memory are all I will need to run to conclusively know that it is in fact memory, then I've saved time out of my day. While the absolute number of possible reasons to have a <1 IPC might be large, the handful of most common ones make up the vast majority of situations in my experience.
YZF 3y ago
I love Brendan's article about load averages but I feel like this is missing the mark. A core, or an execution unit stalling within a core, still count as busy. I.e. the CPU can't process something else. The utilization metric is correct. The question of whether a CPU be used more efficiently is in the domain of optimization. It might be executing NOPs and not stalling. It could be using an O(N^3) algorithm instead of O(N).
- Leszek 3y ago
  With out of order execution, speculation, and SMT, it's hard to say if an instruction stalling means that the CPU can't process something else; CPUs are complex parallel streams of processing and trying to think of them in linear terms necessarily misses some complexity.
  - M95D 3y ago
    If it could process something else, it would. If it doesn't, it means it can't. So, what are you trying to say?
    - FartyMcFarter 3y ago
      In the case of SMT (aka hyperthreading) you'd only know if the CPU can process something else by profiling a different thread that's running on the same core.
  - bpye 3y ago
    Another factor (also mentioned in the article) is dynamic frequency scaling. Is a core at 100% when it's running at it's running 'flat-out' at its nominal frequency or when it has boosted? The boost clock generally depends on thermals, silicon quality and maybe time - so in that case what do you make 100%? If you go for nominal being 100% then sometimes you're going to be at say 120%.
    - mjevans 3y ago
      The most obvious answer: 100% is what the manufacturer claims it capable of handling continuously under the worst supported conditions. The more difficult answer/question is how to communicate that 'full use' value as well as the current use (possibly greater than full) to software which calculates a usage estimate based on various already existing interfaces. Or if yet another interface (standard, if thinking about that XKCD comic) is needed.
failuser 3y ago
Did anyone had the impression that utilization reflects the actual computations the CPU does? It only reflects the fraction the of wall clock time spent in the context, nothing more.
- JBiserkov 3y ago
  Reminds me of the old saying "A computer is a device that moves data around and occasionally does computations with it." think network/disk -> RAM -> L3 -> L2 -> L1 -> registers
  - moonchild 3y ago
    CPUs can broadly be considered to have three major components: - Memory subsystem - Frontend - Backend That is: - Get the data to work on - Figure out what work to do on it - Actually do the work. These are in order of decreasing importance. The von neumann bottleneck is real.
M95D 3y ago
I'm not familiar with modern CPUs, but I remember that most instructions take more than one cycle to execute, without counting memory or cache delays. So, expecting to see 1:1 IPC is ... a fantasy?
- kryptiskt 3y ago
  Yes, that's true, instructions generally take more than one cycle to execute, although the most common ones are fast. But processors these days can execute more than one instruction at a time (this has nothing to do with multiple cores, this is on a single core) and aren't bound to the order the instructions appear in the code, it will make sure that instructions that are dependent on each other will be executed in the right order, otherwise it can opportunistically execute whatever it has room for. Modern processors even speculatively executes code from branches it doesn't yet know will be taken, which caused the whole Meltdown/Spectre security headache.
- anurag6892 3y ago
  yes, most instructions are multicycle. but multiple instructions can execute in parallel in the CPU pipeline. Moreover, while latency is multiple cycles, most ALU units in CPU can start a new operation every cycle i.e. throughput is 1 operation per cycle.
- somat 3y ago
  Many processing units can offset this via superscaler methods, that is tricks to have more than one instruction processing at the same time. pipelining, speculative execution, smt, etc. The article does not go into great depth about it, but does say that the 1 ipc ratio number is based off more gut feel than anything else. I assume the idea is that the superscaler bits(greater than 1 ipc ratio) help compensate for the slow bits(less than 1 ipc ratio) normalizing out at around a 1 ipc ratio when your code is good.
- gpderetta 3y ago
  They do take more than cycle (how many depend on how you count), but they are fully pipelined so they can start executing an ( independent) instruction before a previous one has finished. They are also superscalar so they have multiple (pipelined) units that can start executing instructions at the same time.
kazinator 3y ago
Umm, no. While the memory stalled information could be valuable in optimizing a program, it is correct to count it as part of the CPU busy. Busy means the CPU was occupied with a task (other than the operating system's idle task), and thus not available for another task, regardless of how well or poorly it is making any sort of useful progress. Just like you're busy at work even when you're wiping your monitor instead of coding. Since "waiting for memory" isn't something handled by the scheduler (is not a scheduling wait), it just looks like any other non-scheduler-related busy. A program can keep a CPU busy in multiple of ways such that there is no "utility" even in the absence of memory stalls. It might have a bug which causes it to loop infinitely. Or it could be a service application which is wrongly blowing through what should be a wait at the top of its loop. In the days of multi-user systems at universities, administrations looked upon the running of game programs as a no-utility activity.
- blueflow 3y ago
  Memory stall means re-fetching an evicted or not-yet-faulted-in page from disk, so its disk I/O, which is done asynchronously and leaves the CPU available for another task.
  - anonymoushn 3y ago
    This is not what it means. Maybe you're thinking of page faults, which happen for this reason among other reasons.
  - gpderetta 3y ago
    I would say that the article is really talking about cache misses, not page faults.
  - kazinator 3y ago
    A page fault stall is visible to tools like top. A task waiting for a page to be faulted in so that it can continue is descheduled, and so 0% CPU. So I don't think that's what the article is talking about.
- scrubs 3y ago
bullen 3y ago
So is IPC another measurement of cache-misses? Tried running tiptop on Raspberry 4, it fails with no workaround on the internet. shrugs and goes back to write poor software. Last week I tried to get RDMA bypassing the kernel. Also did not work. Nothing works. Like NOTHING!
belter 3y ago
Discussed before with comments by Gregg: https://news.ycombinator.com/item?id=14301739 https://news.ycombinator.com/item?id=14301739
crabbone 3y ago
I was recently in a similar situation having to first investigate how (various) GPU utilization metrics are defined, and then I also had to explain this information to someone who had the task of representing GPU-related metrics to users. Both turned out to be quite difficult... First had to do with NVML (the library behind nvidia-smi), which isn't very well documented... but I had some help there. But, even after sort of making a mental image of available metrics, the task of presenting users with actionable statistics turned out to be also very hard. On top of what OP writes, imagine now the added modality of having a hierarchical structure of PU workers, some of which, are not even hardware... and, subsequently, the stalls resulting from bad data alignment rather than waiting on physical processes to complete. Add to this yet another modality of vRAM occupancy and its bandwidth utilization... And if you want to show a single number that tells users how loaded their system is... what do you do? Average? Sum? Weighted sum? What if you wanted some absolute units rather than percents? When the data doesn't properly align (to fill up all the SIMD lanes) should this be counted towards "bad" GPU performance or just assumed to be the nature of the workload? Interestingly, the "utilization" nvidia-smi reports is about time, not the number of instructions executed. And it's about time that at least one SP wasn't marked as idle. This stuff becomes even more fuzzy, when eg. NVLink is involved... it's really hard to give concise information to not extremely technical users that'd be actionable from their perspective.
docfort 3y ago
The article mixes a few things together. It essentially describes the von Neumann bottleneck, a consequence of the literal way many traditional computer architectures interpreted Turing’s vision. The thrust of the article is to warn performance engineers to not confuse the OS scheduler view (of whether a CPU is available for more work) and the micro-architectural view (of whether you expect the CPU to retire more instructions for a given number of clock cycles). In my experience on large ARM cores, the max IPC can be high, but programs that do useful work rarely achieve it. Scientific code intended for HPC makes good use of vector units or just superscalar processing, along with (manual) interleaving of compute and memory I/O. Other code, like most web browsers, can hit IPC=1, only after a ton of tuning. Both categories are important, but usually the pot of money is larger for the HPC code, or at least the optimization path is clearer. In other words: the article is intended primarily for someone to understand when they might want a performance engineer and not just call it a day when they see full CPU scheduling utilization.
tetha 3y ago
Hmm, could the new Pressure Stall Information in /proc/pressure give further insights into this at a broad level?
dehrmann 3y ago
I didn't realize how much pressure there can be on memory. This explains that part of Apple's M1/2 performance is just from SoC benefits, and that even in non-IO applications, hyperthreading is more than a marketing gimmick.

94 points by xk3 3y ago

SMT is pitched as a good way to "hide" low IPC loads by giving the core something else to do while waiting for memory. Hence IBM and (very briefly) Marvell both shipped 4-way SMT cores for database-heavy workloads. AFAIK this further complicates the IPC metric. What looks like a stalled core might actually be working on another thread. And at the other extreme, two very high IPC threads on one core would lower the observed IPC of each thread.

hinkley 3y ago
I was always curious why they didn't just make pairs of CPUs that shared a few logic units in common. Like I don't know what the instruction mix is these days and just how long the pipeline is, but say most code can do 2-3 ALU instructions per clock, why not 2 cores with 1-2 ALUs each and 2 shared? As things are, there is very little guarantee that the hyperthread will make any substantial progress.
- csherratt 3y ago
  AMD Bulldozer did something like this. Ultraspac T1 had a shared FPU too. There is are a savings there, but neither of these solutions ended up being practically fast. The amount of die space spent on an ALU is relatively low compared to the rest of the CPU, hence why they shared FPUs instread. We might see it again now that the server world is having an ARM CPU renaissance. But I doubt AMD or Intel will make anything that exotic.
  - sweetjuly 3y ago
    I sort of hope we don't see SMT ARM cores. SMT was/is a huge pain for side channels and all the ARM chip houses dodged that one by nature of just never implementing it. I would hope that the concern of cloud vendors over isolation would be enough to discourage bringing more SMT uarchs in to this world.
    - ilyt 3y ago
      Doesn't just giving a VM both threads of the core mitigate that ?
      - d_tr 3y ago
        Or just disabling SMT? SMT can give a good utilization boost and it feels like a shame for it not to be an option, especially since CPUs are not getting faster at the same rate they used to anymore. You can disable it on AMD and Intel x86 cores, as well as on the IBM Power ones.
        ilyt 3y ago
        but you're throwing performance away. If you don't want cross-customer leaks just give each customer both SMT threads of the core
        hinkley 3y ago
        Citation needed. Generation after generation, hyperthreads on almost every generation of Intel core have turned in mediocre to terrible benchmarks. Neither thread makes full progress , and on some versions you would see more IPC with them turned off. AMD might have managed better, though I see benchmarks where theirs is also a wash. It doesn’t mean Arm or RISC-V will see any benefit.
        ilyt 3y ago
        I've seen around 20% increase in even dumb "a web app returning some text" benchmarks. It certainly shouldt be charged as a second real core would but it isn't nothing (like in P4 era you apparently time travelled from) But that's entirely beside the point and I'm not sure why you try to derail the conversation into different direciton.
        brucethemoose2 3y ago
        Y'all are talking about seperate things, I think. SMT often does boost database heavy "cloud" workloads, but not client workloads like media processing, games, work apps and so on. We have it on desktops/laptops because they use hand-me-down server cores. Its not worth redesigning the cores to bifurcate the architecture.
- gpderetta 3y ago
  ALUs[1] do not really consume that large of an area of a CPU so just sharing those is a significant complication with little benefits. Sharing register files, caches, store buffers, ROBs, TLBs, prediction tables, and the logic to drive them is a better trade-off. [1] outside of large SIMD FPUs, which indeed bulldozer tried to share with not great results.
BackBlast 3y ago
Except that you've just split your available cache region between multiple processes. This can actually decrease performance, I remember cache thrashing back in the P4 days that could cause an application like MS Word to take minutes to load. One process evicting data the other one needs and vice versa. It circumstantially makes the situation worse instead of better. Turn off hyper-threading and boom, everything started to function normally. The solution to not having enough food for one mouth often isn't to add another mouth. I still see this in modern benchmarks on modern systems where multicore throughput on many benchmarks is just better when you just turn off SMT. AMD's 3D chips go faster because of the huge cache. Every time we add another process to a core we just split the cache in half instead. Clearly, this is workload dependent. But I immediately turn off SMT on new systems I get. I'm not core limited, I'm usually single thread limited. Like, always. No reason to keep SMT on for this case.

hinkley 3y ago

How long has it been the case that a CPU would basically catch on fire if it could run without pipeline stalls slowing down the thermal dissipation?

sweetjuly 3y ago
Stalls are not necessary for thermal regulation, DVFS [1] is the tool we actually use for handling thermal limits (generally referred to as "thermal throttling"). In a modern CPU, you really don't expect (or, rather, want) to be fully stalled ever, that's the point of very deep reorder buffers. You can try this yourself, make a long loop with a ton of unrelated int and fp operations (unrelated so as not to cause dependency chains) and you'll get 100% occupancy without the core catching on fire. [1] https://en.wikipedia.org/wiki/Dynamic_frequency_scaling https://en.wikipedia.org/wiki/Dynamic_frequency_scaling

kaelinl 3y ago

(2017)

nubinetwork 3y ago

Why use top when you can use htop? It breaks down CPU usage into various parts like IO wait and vm guest time... they even integrated functionality from iotop so you eventually won't need that program either.

pavon 3y ago
The "Detailed CPU time" setting in htop doesn't show the time the CPU is waiting on memory, it is just traditional I/O (disk, network, etc).
- oriettaxx 3y ago
  he is suggesting the use of `tiptop`

sweetjuly 3y ago

Hmm, I feel like this is a bit too big of a generalization to be useful. It's a cool observation but I don't think you can interpet IPC so directly as to conclude that IPC < 1 indicates "you are likely memory stalled". There are a million reasons why you'd have an IPC < 1, many of them which are not memory. Maybe you just have a lot of hard to predict branches which take a long time to resolve (all that mispeculated work is wasted cycles!), maybe you use a lot of long latency FP operations, maybe you just have really long dependency chains. If your core doesn't have full speculative wakeups for variable latency instructions (meaning things like loads but often things like division and multiplication have early exits), you can very easily end up with plenty of cycles where you don't complete an instruction. And, yes, of course, it could be that you're having a lot of cache misses. Memory is a good guess but it's not really actionable advice for performance analysis, you really need to pull out some proper tools to understand why you're getting that number. I think this is a cool article in that, yes, Task Manager/top/Activity Monitor aren't telling you the full story when it comes to what "CPU %" means. In the end though, there really isn't an easy way to come up with a better metric for utilization that can be summarized in one number, so realistically "how much time did the scheduler put this process on a core for" is plenty good enough for most purposes.

failuser 3y ago
Memory is just the most common. Most software does not involve complicated calculations to get to the other reasons.
cthalupa 3y ago
>Memory is a good guess but it's not really actionable advice for performance analysis, you really need to pull out some proper tools to understand why you're getting that number. I don't really understand and by extension agree with this point. If you're doing performance analysis on a regular basis, having more readily available tools that give you a solid guess at where you want to look next is useful. It's a lot easier for me to get pcm up and running on any random system than it is to dig in with vtune, for example, and if I can get a fairly accurate determination that pcm and pcm-memory are all I will need to run to conclusively know that it is in fact memory, then I've saved time out of my day. While the absolute number of possible reasons to have a <1 IPC might be large, the handful of most common ones make up the vast majority of situations in my experience.

YZF 3y ago

I love Brendan's article about load averages but I feel like this is missing the mark. A core, or an execution unit stalling within a core, still count as busy. I.e. the CPU can't process something else. The utilization metric is correct. The question of whether a CPU be used more efficiently is in the domain of optimization. It might be executing NOPs and not stalling. It could be using an O(N^3) algorithm instead of O(N).

Leszek 3y ago
With out of order execution, speculation, and SMT, it's hard to say if an instruction stalling means that the CPU can't process something else; CPUs are complex parallel streams of processing and trying to think of them in linear terms necessarily misses some complexity.
- M95D 3y ago
  If it could process something else, it would. If it doesn't, it means it can't. So, what are you trying to say?
  - FartyMcFarter 3y ago
    In the case of SMT (aka hyperthreading) you'd only know if the CPU can process something else by profiling a different thread that's running on the same core.
- bpye 3y ago
  Another factor (also mentioned in the article) is dynamic frequency scaling. Is a core at 100% when it's running at it's running 'flat-out' at its nominal frequency or when it has boosted? The boost clock generally depends on thermals, silicon quality and maybe time - so in that case what do you make 100%? If you go for nominal being 100% then sometimes you're going to be at say 120%.
  - mjevans 3y ago
    The most obvious answer: 100% is what the manufacturer claims it capable of handling continuously under the worst supported conditions. The more difficult answer/question is how to communicate that 'full use' value as well as the current use (possibly greater than full) to software which calculates a usage estimate based on various already existing interfaces. Or if yet another interface (standard, if thinking about that XKCD comic) is needed.

failuser 3y ago

Did anyone had the impression that utilization reflects the actual computations the CPU does? It only reflects the fraction the of wall clock time spent in the context, nothing more.

JBiserkov 3y ago
Reminds me of the old saying "A computer is a device that moves data around and occasionally does computations with it." think network/disk -> RAM -> L3 -> L2 -> L1 -> registers
- moonchild 3y ago
  CPUs can broadly be considered to have three major components: - Memory subsystem - Frontend - Backend That is: - Get the data to work on - Figure out what work to do on it - Actually do the work. These are in order of decreasing importance. The von neumann bottleneck is real.

M95D 3y ago

I'm not familiar with modern CPUs, but I remember that most instructions take more than one cycle to execute, without counting memory or cache delays. So, expecting to see 1:1 IPC is ... a fantasy?

kryptiskt 3y ago
Yes, that's true, instructions generally take more than one cycle to execute, although the most common ones are fast. But processors these days can execute more than one instruction at a time (this has nothing to do with multiple cores, this is on a single core) and aren't bound to the order the instructions appear in the code, it will make sure that instructions that are dependent on each other will be executed in the right order, otherwise it can opportunistically execute whatever it has room for. Modern processors even speculatively executes code from branches it doesn't yet know will be taken, which caused the whole Meltdown/Spectre security headache.
anurag6892 3y ago
yes, most instructions are multicycle. but multiple instructions can execute in parallel in the CPU pipeline. Moreover, while latency is multiple cycles, most ALU units in CPU can start a new operation every cycle i.e. throughput is 1 operation per cycle.
somat 3y ago
Many processing units can offset this via superscaler methods, that is tricks to have more than one instruction processing at the same time. pipelining, speculative execution, smt, etc. The article does not go into great depth about it, but does say that the 1 ipc ratio number is based off more gut feel than anything else. I assume the idea is that the superscaler bits(greater than 1 ipc ratio) help compensate for the slow bits(less than 1 ipc ratio) normalizing out at around a 1 ipc ratio when your code is good.
gpderetta 3y ago
They do take more than cycle (how many depend on how you count), but they are fully pipelined so they can start executing an ( independent) instruction before a previous one has finished. They are also superscalar so they have multiple (pipelined) units that can start executing instructions at the same time.

kazinator 3y ago

Umm, no. While the memory stalled information could be valuable in optimizing a program, it is correct to count it as part of the CPU busy. Busy means the CPU was occupied with a task (other than the operating system's idle task), and thus not available for another task, regardless of how well or poorly it is making any sort of useful progress. Just like you're busy at work even when you're wiping your monitor instead of coding. Since "waiting for memory" isn't something handled by the scheduler (is not a scheduling wait), it just looks like any other non-scheduler-related busy. A program can keep a CPU busy in multiple of ways such that there is no "utility" even in the absence of memory stalls. It might have a bug which causes it to loop infinitely. Or it could be a service application which is wrongly blowing through what should be a wait at the top of its loop. In the days of multi-user systems at universities, administrations looked upon the running of game programs as a no-utility activity.

blueflow 3y ago
Memory stall means re-fetching an evicted or not-yet-faulted-in page from disk, so its disk I/O, which is done asynchronously and leaves the CPU available for another task.
- anonymoushn 3y ago
  This is not what it means. Maybe you're thinking of page faults, which happen for this reason among other reasons.
- gpderetta 3y ago
  I would say that the article is really talking about cache misses, not page faults.
- kazinator 3y ago
  A page fault stall is visible to tools like top. A task waiting for a page to be faulted in so that it can continue is descheduled, and so 0% CPU. So I don't think that's what the article is talking about.
scrubs 3y ago

bullen 3y ago

So is IPC another measurement of cache-misses? Tried running tiptop on Raspberry 4, it fails with no workaround on the internet. shrugs and goes back to write poor software. Last week I tried to get RDMA bypassing the kernel. Also did not work. Nothing works. Like NOTHING!

belter 3y ago

Discussed before with comments by Gregg: https://news.ycombinator.com/item?id=14301739 https://news.ycombinator.com/item?id=14301739

crabbone 3y ago

I was recently in a similar situation having to first investigate how (various) GPU utilization metrics are defined, and then I also had to explain this information to someone who had the task of representing GPU-related metrics to users. Both turned out to be quite difficult... First had to do with NVML (the library behind nvidia-smi), which isn't very well documented... but I had some help there. But, even after sort of making a mental image of available metrics, the task of presenting users with actionable statistics turned out to be also very hard. On top of what OP writes, imagine now the added modality of having a hierarchical structure of PU workers, some of which, are not even hardware... and, subsequently, the stalls resulting from bad data alignment rather than waiting on physical processes to complete. Add to this yet another modality of vRAM occupancy and its bandwidth utilization... And if you want to show a single number that tells users how loaded their system is... what do you do? Average? Sum? Weighted sum? What if you wanted some absolute units rather than percents? When the data doesn't properly align (to fill up all the SIMD lanes) should this be counted towards "bad" GPU performance or just assumed to be the nature of the workload? Interestingly, the "utilization" nvidia-smi reports is about time, not the number of instructions executed. And it's about time that at least one SP wasn't marked as idle. This stuff becomes even more fuzzy, when eg. NVLink is involved... it's really hard to give concise information to not extremely technical users that'd be actionable from their perspective.

docfort 3y ago

The article mixes a few things together. It essentially describes the von Neumann bottleneck, a consequence of the literal way many traditional computer architectures interpreted Turing’s vision. The thrust of the article is to warn performance engineers to not confuse the OS scheduler view (of whether a CPU is available for more work) and the micro-architectural view (of whether you expect the CPU to retire more instructions for a given number of clock cycles). In my experience on large ARM cores, the max IPC can be high, but programs that do useful work rarely achieve it. Scientific code intended for HPC makes good use of vector units or just superscalar processing, along with (manual) interleaving of compute and memory I/O. Other code, like most web browsers, can hit IPC=1, only after a ton of tuning. Both categories are important, but usually the pot of money is larger for the HPC code, or at least the optimization path is clearer. In other words: the article is intended primarily for someone to understand when they might want a performance engineer and not just call it a day when they see full CPU scheduling utilization.

tetha 3y ago

Hmm, could the new Pressure Stall Information in /proc/pressure give further insights into this at a broad level?

dehrmann 3y ago

I didn't realize how much pressure there can be on memory. This explains that part of Apple's M1/2 performance is just from SoC benefits, and that even in non-IO applications, hyperthreading is more than a marketing gimmick.