4 ms·

Advance profilers will access hardware performance counters that give insights. For example, if your instructions are (on average), waiting 10ns per load/store

Advance profilers will access hardware performance counters that give insights. For example, if your instructions are (on average), waiting 10ns per load/store, you know that you're operating at the L2 cache level or so. Although it might be easier to get the count of L1 cache misses, and L2 cache hits instead. :-) Pretty much every event at the lowest level can be counted (although its sampling: only every 1/1000th instruction or so gets profiled when these profilers get turned on). When you see that some branch is mostly unpredicted, you know to focus on that branch/if statement to figure out how to get the branch predictor to work. If you see L1 misses or L2 misses, maybe you know to shrink the data. Etc. etc.

gpderetta 3y ago
perf will give you exactly the number of misses in L1, L2 and L3 if asked (and the total number of accesses). It can also annotate the asm so you can spot the hot loads.
vient 3y ago
Yes but this level of info is too coarse. It is not sufficient to know your level of L1/L2/L3 misses if your app has megabytes of code and gigabytes in heap. As I said in adjacent response, I tried to record cache misses with `perf` but the info produced by `perf report` is confusing at best - I did not manage to find any good tutorial on debugging cache misses in that way.
- dragontamer 3y ago
  You know about the --control=fifo option of perf, right? You can turn perf on, and off, using that fifo. You can use this to programmatically enable the profiler for individual sections of code, and then disable it for the other sections. ----------- Are you coarse-profiling your code first? Have you run gprof first and figured out which section of code you're focusing on? EDIT: Using perf alone (though IMO a bad idea), you can also get instruction counts. You should be focusing on the instructions that are run the most. But because instructions take a variable amount of time, what you really want to be doing is profiling using a timer-based method (ex: gprof) and narrowing your search through that instead. But sometimes, the instruction-based approach is also useful.
  - vient 3y ago
    Manually enabling/disabling perf is too slow when you want to profile a part that takes <1ms to execute, but it is an option if nothing better exists. It would be great if you could just isolated interesting code in microbenchmark but we all know that this will skew results. About code execution profiling, I usually use Intel PT so yes, I know where I want to look.
- soulbadguy 3y ago
  > I did not manage to find any good tutorial on debugging cache misses in that way. Saddly this is very true. There is a lack of good performance investigation information out there. And pref while very useful is not really a beginner tools. In your case, you want an "annotated" profile. Where perf will annotate the source code with the even, and basically pin-point which part of your code is producing the offending events (in this case the cache misses). If you really want to dig deep, i can't recommend intel Vtune. Only works with intel cpu (i think ?), but is the best tool in you want to understand performance at a deeper level.
  - vient 3y ago
    The problem with cache misses specifically is that perf can't point an event to its exact location so you may get a high rate of cache misses on instruction that does not even work with memory, like jump, so you need to look what gets executed before that instruction and what looks like probable source there. I am interested if there are any more efficient approaches than this.
    - soulbadguy 3y ago
      Perf can. But you need to massage it a bit. Out of the box , perf uses a bunch of generic performance counter which maybe not be the best. Intel cpu's have a a collection of "precisise" events which record the exact originating/offending instructions. If you use those it should fix your problem Pmu-tool by default uses precise events when available
      - vient 3y ago
        Oh, I did not know that perf by default may use some uncore events that are less precise than some Intel-specific events. Thanks for the info!
        soulbadguy 3y ago
        You might want to look at https://icl.utk.edu/papi/ https://icl.utk.edu/papi/ . It's a generic library which abstract the details of specific cpu arch and aims to provide a single unify view of performance counters. I think somewhere in there you can see how perf translate counters to intel (or AMD) specific one.
soulbadguy 3y ago
Hmmm a fellow lover of CPU performance counter :). Couple of points : > although its sampling: only every 1/1000th instruction or so gets profiled when these profilers get turned on Not always, modern CPU ( intel's at least, not sure about arm and AMD), you can control the recorded event and the sampling event (and sampling rate) seperatly. So you can actually both record load/store misses and sample on cache misses to avoid loosing any events. You still get some sampling artifacts like the fact that the triggering of event is biais in favor or start of Basic block or that some events can be shadowed by others ... > if your instructions are (on average), waiting 10ns per load/store, you know that you're operating at the L2 cache level or so. Not always, sometime if you have a chain of depends loads the resolution time of the load has more to do with data dependencies. Intel's CPU has a collection of "precise" events, around caches events, so it's easy to just deduce precisely where a given loads resolves most of the time by doing the ration of L-cache misses/L cache hit dirrectly) > When you see that some branch is mostly unpredicted, you know to focus on that branch/if statement to figure out how to get the branch predictor to work. The hardest part here is to understand if the branch is unpredicted because it's unpredictable or because the branch predictor is not performing as expected : Branch predictors are limited resources, they can't predict all branch all the time.

gpderetta 3y ago

perf will give you exactly the number of misses in L1, L2 and L3 if asked (and the total number of accesses). It can also annotate the asm so you can spot the hot loads.

vient 3y ago

Yes but this level of info is too coarse. It is not sufficient to know your level of L1/L2/L3 misses if your app has megabytes of code and gigabytes in heap. As I said in adjacent response, I tried to record cache misses with `perf` but the info produced by `perf report` is confusing at best - I did not manage to find any good tutorial on debugging cache misses in that way.

dragontamer 3y ago
You know about the --control=fifo option of perf, right? You can turn perf on, and off, using that fifo. You can use this to programmatically enable the profiler for individual sections of code, and then disable it for the other sections. ----------- Are you coarse-profiling your code first? Have you run gprof first and figured out which section of code you're focusing on? EDIT: Using perf alone (though IMO a bad idea), you can also get instruction counts. You should be focusing on the instructions that are run the most. But because instructions take a variable amount of time, what you really want to be doing is profiling using a timer-based method (ex: gprof) and narrowing your search through that instead. But sometimes, the instruction-based approach is also useful.
- vient 3y ago
  Manually enabling/disabling perf is too slow when you want to profile a part that takes <1ms to execute, but it is an option if nothing better exists. It would be great if you could just isolated interesting code in microbenchmark but we all know that this will skew results. About code execution profiling, I usually use Intel PT so yes, I know where I want to look.
soulbadguy 3y ago
> I did not manage to find any good tutorial on debugging cache misses in that way. Saddly this is very true. There is a lack of good performance investigation information out there. And pref while very useful is not really a beginner tools. In your case, you want an "annotated" profile. Where perf will annotate the source code with the even, and basically pin-point which part of your code is producing the offending events (in this case the cache misses). If you really want to dig deep, i can't recommend intel Vtune. Only works with intel cpu (i think ?), but is the best tool in you want to understand performance at a deeper level.
- vient 3y ago
  The problem with cache misses specifically is that perf can't point an event to its exact location so you may get a high rate of cache misses on instruction that does not even work with memory, like jump, so you need to look what gets executed before that instruction and what looks like probable source there. I am interested if there are any more efficient approaches than this.
  - soulbadguy 3y ago
    Perf can. But you need to massage it a bit. Out of the box , perf uses a bunch of generic performance counter which maybe not be the best. Intel cpu's have a a collection of "precisise" events which record the exact originating/offending instructions. If you use those it should fix your problem Pmu-tool by default uses precise events when available
    - vient 3y ago
      Oh, I did not know that perf by default may use some uncore events that are less precise than some Intel-specific events. Thanks for the info!
      - soulbadguy 3y ago
        You might want to look at https://icl.utk.edu/papi/ https://icl.utk.edu/papi/ . It's a generic library which abstract the details of specific cpu arch and aims to provide a single unify view of performance counters. I think somewhere in there you can see how perf translate counters to intel (or AMD) specific one.

soulbadguy 3y ago

Hmmm a fellow lover of CPU performance counter :). Couple of points : > although its sampling: only every 1/1000th instruction or so gets profiled when these profilers get turned on Not always, modern CPU ( intel's at least, not sure about arm and AMD), you can control the recorded event and the sampling event (and sampling rate) seperatly. So you can actually both record load/store misses and sample on cache misses to avoid loosing any events. You still get some sampling artifacts like the fact that the triggering of event is biais in favor or start of Basic block or that some events can be shadowed by others ... > if your instructions are (on average), waiting 10ns per load/store, you know that you're operating at the L2 cache level or so. Not always, sometime if you have a chain of depends loads the resolution time of the load has more to do with data dependencies. Intel's CPU has a collection of "precise" events, around caches events, so it's easy to just deduce precisely where a given loads resolves most of the time by doing the ration of L-cache misses/L cache hit dirrectly) > When you see that some branch is mostly unpredicted, you know to focus on that branch/if statement to figure out how to get the branch predictor to work. The hardest part here is to understand if the branch is unpredicted because it's unpredictable or because the branch predictor is not performing as expected : Branch predictors are limited resources, they can't predict all branch all the time.