4 ms·

CPU utilization is wrong [video]

45 points by startupflix 8y ago

billysielu 8y ago
So is the CPU available to do something else or not?
- Tuna-Fish 8y ago
  The particular thread that is waiting for data is not, but in a SMT system the other thread has access to the full execution resources of the CPU.
DonbunEf7 8y ago
This nigh-useless blurb links to a Youtube video: https://youtu.be/QkcBASKLyeU https://youtu.be/QkcBASKLyeU
- ModernMech 8y ago
  Yeah, can someone quick summarize this? Can't watch where I am. edit: I found his blog post on this very topic. This should be the source: http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-... Also, now that we have the blog link, we have some previous HN threads: https://news.ycombinator.com/item?id=14301739 https://news.ycombinator.com/item?id=14301739
  - decebalus1 8y ago
    http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-...
    - ModernMech 8y ago
      Beat you to it but thanks :P
  - dragontamer 8y ago
    Brendan Gregg diagnoses a performance problem on Netflix servers. He starts with CPU Utilization, but CPU Utilization metrics from top or ps don't give enough detail. He checks out more detailed performance views (including flame graphs, and some custom tools he wrote himself) to eventually narrow the problem down to the TLB: Translation-Lookaside Buffer. The mechanism the CPU uses to implement virtual memory. Finally, he realizes that TLB-Misses were caused by Meltdown / Spectre Patches. Which is a difference between the two servers he was diagnosing the performance problem for.
dragontamer 8y ago
So first of all: the lightning talk is certainly "worth it". Brendan Gregg is very knowledgeable and I can certainly recommend watching the talk. Its a good overview into the mindset of low-level optimization. However, the core premise, the "headline" if you will of this talk, is utterly wrong. I get it: the headline needs to be a bit provocative to draw eyes, but its closer to "clickbait" than a good title of the talk. The TL;DR of the talk is: Spectre/Meltdown caused a slowdown. But said slowdown is coming up as a ~+20% "CPU Utilization" under Linux performance measurement (ie: top or ps). This slowdown is because of a severe increase in TLB-misses. Which... doesn't seem "wrong" to me. If the CPU is stalling because of TLB-buffer misses (kinda similar to an L2 cache miss or whatnot), then that's certainly CPU-time that is being wasted on something. Perhaps it isn't using power since the CPU is fully stalled, but the CPU is certainly "utilized" during that timeframe. Aside from hyperthreading: you can't really do something during the stall. ---------- Anyway, my criticism aside, the talk is certainly worthwhile to watch. And that's the important bit. So go ahead and watch the talk, even if I have a few pedantic qualms about the phrasing of the headline. The real benefit of the talk is how Brendan Gregg diagnosed this performance problem, which eventually led to his conclusion about TLB-misses + Spectre. Demonstrating this skill and walking through a system to be able to draw this kind of conclusion is incredibly important. I just don't... really like the title of the talk I guess. A rather minor concern all else considered.
- ModernMech 8y ago
  I think the problem Brendan is pointing out is that if we look at a task that appears CPU-bound, we might think that one solution to make things go faster is just make the CPU go faster. But beefing up the CPU won't help, because it'll just spend the same amount of time waiting for e.g. the TLB to make the round trip lookup. I think this is really a discussion about the Von Neumann Bottleneck (VNBN). The slowdown here is caused by the VNBN, but %CPU utilization is "disguising" slowdowns caused by this bottleneck as a CPU bottleneck. The VNBN is the primary reason Intel spends so much of their die space on caches -- it's really holding us back. Intel is basically a glorified SRAM vendor, selling vast blocks of memory attached as closely to a processor as possible, because it's just so damn expensive to fetch it from anywhere else, thanks to the VNBN. > Aside from hyperthreading: you can't really do something during the stall. This is true, but hyperthreading doesn't truly help us because the unit of concurrency -- the thread here -- is too coarse, and everyone is competing over the same resource: memory bandwidth (again, thanks VNBN), so any time you spend on another task is going to increase return time of the original task you were waiting for. But we can do something here --- use a different architecture. Imagine a dataflow machine, which sidesteps the VNBN entirely. Now if your processor is stalled waiting for a TLB return, there’s a vast pile of work that can be freely executed. Sure we have other resources at play now (like whatever message passing bandwidth we have), but we no longer have to worry about the VNBN.
arseraptor 8y ago
Buttfuck Gyro is nothing if not an utter and total glory hound. I often wonder if his parents ignored him?
- dang 8y ago
  Yikes, posting like this will get you banned here, so please read https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and don't post like this!
gfnord 8y ago
That's great and I agree with him. We have to pay attention to the stale.
gnode 8y ago
To save you the time of watching the video: Spectre / Meltdown mitigations (page table isolation) cause execution inefficiency because of TLB cache misses.

45 points by startupflix 8y ago

billysielu 8y ago

So is the CPU available to do something else or not?

Tuna-Fish 8y ago
The particular thread that is waiting for data is not, but in a SMT system the other thread has access to the full execution resources of the CPU.

DonbunEf7 8y ago

This nigh-useless blurb links to a Youtube video: https://youtu.be/QkcBASKLyeU https://youtu.be/QkcBASKLyeU

ModernMech 8y ago
Yeah, can someone quick summarize this? Can't watch where I am. edit: I found his blog post on this very topic. This should be the source: http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-... Also, now that we have the blog link, we have some previous HN threads: https://news.ycombinator.com/item?id=14301739 https://news.ycombinator.com/item?id=14301739
- decebalus1 8y ago
  http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-...
  - ModernMech 8y ago
    Beat you to it but thanks :P
- dragontamer 8y ago
  Brendan Gregg diagnoses a performance problem on Netflix servers. He starts with CPU Utilization, but CPU Utilization metrics from top or ps don't give enough detail. He checks out more detailed performance views (including flame graphs, and some custom tools he wrote himself) to eventually narrow the problem down to the TLB: Translation-Lookaside Buffer. The mechanism the CPU uses to implement virtual memory. Finally, he realizes that TLB-Misses were caused by Meltdown / Spectre Patches. Which is a difference between the two servers he was diagnosing the performance problem for.

dragontamer 8y ago

So first of all: the lightning talk is certainly "worth it". Brendan Gregg is very knowledgeable and I can certainly recommend watching the talk. Its a good overview into the mindset of low-level optimization. However, the core premise, the "headline" if you will of this talk, is utterly wrong. I get it: the headline needs to be a bit provocative to draw eyes, but its closer to "clickbait" than a good title of the talk. The TL;DR of the talk is: Spectre/Meltdown caused a slowdown. But said slowdown is coming up as a ~+20% "CPU Utilization" under Linux performance measurement (ie: top or ps). This slowdown is because of a severe increase in TLB-misses. Which... doesn't seem "wrong" to me. If the CPU is stalling because of TLB-buffer misses (kinda similar to an L2 cache miss or whatnot), then that's certainly CPU-time that is being wasted on something. Perhaps it isn't using power since the CPU is fully stalled, but the CPU is certainly "utilized" during that timeframe. Aside from hyperthreading: you can't really do something during the stall. ---------- Anyway, my criticism aside, the talk is certainly worthwhile to watch. And that's the important bit. So go ahead and watch the talk, even if I have a few pedantic qualms about the phrasing of the headline. The real benefit of the talk is how Brendan Gregg diagnosed this performance problem, which eventually led to his conclusion about TLB-misses + Spectre. Demonstrating this skill and walking through a system to be able to draw this kind of conclusion is incredibly important. I just don't... really like the title of the talk I guess. A rather minor concern all else considered.

ModernMech 8y ago
I think the problem Brendan is pointing out is that if we look at a task that appears CPU-bound, we might think that one solution to make things go faster is just make the CPU go faster. But beefing up the CPU won't help, because it'll just spend the same amount of time waiting for e.g. the TLB to make the round trip lookup. I think this is really a discussion about the Von Neumann Bottleneck (VNBN). The slowdown here is caused by the VNBN, but %CPU utilization is "disguising" slowdowns caused by this bottleneck as a CPU bottleneck. The VNBN is the primary reason Intel spends so much of their die space on caches -- it's really holding us back. Intel is basically a glorified SRAM vendor, selling vast blocks of memory attached as closely to a processor as possible, because it's just so damn expensive to fetch it from anywhere else, thanks to the VNBN. > Aside from hyperthreading: you can't really do something during the stall. This is true, but hyperthreading doesn't truly help us because the unit of concurrency -- the thread here -- is too coarse, and everyone is competing over the same resource: memory bandwidth (again, thanks VNBN), so any time you spend on another task is going to increase return time of the original task you were waiting for. But we can do something here --- use a different architecture. Imagine a dataflow machine, which sidesteps the VNBN entirely. Now if your processor is stalled waiting for a TLB return, there’s a vast pile of work that can be freely executed. Sure we have other resources at play now (like whatever message passing bandwidth we have), but we no longer have to worry about the VNBN.

arseraptor 8y ago

Buttfuck Gyro is nothing if not an utter and total glory hound. I often wonder if his parents ignored him?

dang 8y ago
Yikes, posting like this will get you banned here, so please read https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and don't post like this!

gfnord 8y ago

That's great and I agree with him. We have to pay attention to the stale.

gnode 8y ago

To save you the time of watching the video: Spectre / Meltdown mitigations (page table isolation) cause execution inefficiency because of TLB cache misses.