30 ms·

> our faster function could take advantage of no more than 8 cores; beyond that it started slowing down. Perhaps it started hitting some bottleneck other than c

by dahart 3y ago

> our faster function could take advantage of no more than 8 cores; beyond that it started slowing down. Perhaps it started hitting some bottleneck other than computation, like memory bandwidth. @itamarst Yes, this in interesting, you should profile it and get to the bottom of the issue! It seems like in my experience that being limited by hyperthreading or instruction-level parallism is relatively rare, and much more often it’s cache or memory access patterns or implicit synchronization or contention for a hardware resource. There’s a good chance you’ll learn something useful by figuring it out. Maybe it’s memory bus contention, maybe it’s cache, maybe numba compiled in something you aren’t expecting. Worth nothing that using 20 on the fast test isn’t that much slower than using 8. A good first guess/proxy for number of threads to use is the number of cores, and that pays off in this case compared to using too few cores. Out of curiosity, do you know if your images are stored row-major or column major? I see the outer loop over shape[0] and inner loop over shape[1]. Is the compiled code stepping in memory by 1 pixel at time, or by a whole column? If your stride is a column, you may be thrashing the cache. I’d also be curious to hear how the speed of this compiled code compares to a numpy or PIL image threshold operation, if you happen to know.

itamarst 3y ago
NumPy default is that you iterate over the earlier dimensions first. The slow code is likely at least partially slow due to branch misprediction (this is specific to my CPU, not true on CPUs with AVX-512), see https://pythonspeed.com/articles/speeding-up-numba/ https://pythonspeed.com/articles/speeding-up-numba/ where I use `perf stat` to get branch misprediction numbers on similar code. With SIMD disabled there's also a clear difference in IPC, I believe. The bigger picture though is that the goal of this article is not to demonstrate speeding up code, it's to ask about level of parallelism given unchanging code. Obviously all things being equal you'll do better if you can make your code faster, but code does get deployed, and when it's deployed you need to choose parallelism levels, regardless of how good the code is.
- dahart 3y ago
  > when it's deployed you need to choose parallelism levels, regardless of how good the code is. Yes, absolutely, exactly. That’s why it can be really helpful to pinpoint that cause of slowdown, right? It might not matter at deployment time if you have an automated shmoo that calculates the optimal thread load, but knowing the actual causes might be critical to the process, and/or really help if you don’t do it automatically. (For one, it’s possible the conditions for optimal thread load could change over the course of a run.)
deleted 3y ago
[deleted]
lumost 3y ago
Is there a good method to split out memory and CPU bottlenecks on a CPU? most folks simply optimize until they hit 100% and assume all CPU usage is created equal. Seems like there needs to be a tool similar to htop with utiization stats for different instructions to indicate how much of the CPU is idle/busy.
- grogers 3y ago
  I think most people would measure instructions per cycle as an estimate of how busy the CPU execution units actually are. Lots of performance tools do measure it.
  - dist1ll 3y ago
    IPC is not a good performance metric - especially for an instruction set like x86, but particularly for SIMD ISAs. Two seemingly similar instructions sequences may get decoded into a different number of uops, which in turn occupy different execution ports. The most extreme example of this being scalar code without stalls. It tends to have high IPC, even though it makes no use of FP/SIMD ports. Keeping the execution pipeline busy is not a useful goal in isolation. Really, IPC is better left as a sanity check.
- gnufx 3y ago
  Consider appropriate hardware counters. (Note that multiplexing to measure more then three(?) at once can be misleading.) At least for serial code on x86, there's "top-down performance analysis". Some tools (I know Maqao) suggest how to address bottlenecks.

by dahart 3y ago

itamarst 3y ago

NumPy default is that you iterate over the earlier dimensions first. The slow code is likely at least partially slow due to branch misprediction (this is specific to my CPU, not true on CPUs with AVX-512), see https://pythonspeed.com/articles/speeding-up-numba/ https://pythonspeed.com/articles/speeding-up-numba/ where I use `perf stat` to get branch misprediction numbers on similar code. With SIMD disabled there's also a clear difference in IPC, I believe. The bigger picture though is that the goal of this article is not to demonstrate speeding up code, it's to ask about level of parallelism given unchanging code. Obviously all things being equal you'll do better if you can make your code faster, but code does get deployed, and when it's deployed you need to choose parallelism levels, regardless of how good the code is.

dahart 3y ago
> when it's deployed you need to choose parallelism levels, regardless of how good the code is. Yes, absolutely, exactly. That’s why it can be really helpful to pinpoint that cause of slowdown, right? It might not matter at deployment time if you have an automated shmoo that calculates the optimal thread load, but knowing the actual causes might be critical to the process, and/or really help if you don’t do it automatically. (For one, it’s possible the conditions for optimal thread load could change over the course of a run.)

deleted 3y ago

[deleted]

lumost 3y ago

Is there a good method to split out memory and CPU bottlenecks on a CPU? most folks simply optimize until they hit 100% and assume all CPU usage is created equal. Seems like there needs to be a tool similar to htop with utiization stats for different instructions to indicate how much of the CPU is idle/busy.

grogers 3y ago
I think most people would measure instructions per cycle as an estimate of how busy the CPU execution units actually are. Lots of performance tools do measure it.
- dist1ll 3y ago
  IPC is not a good performance metric - especially for an instruction set like x86, but particularly for SIMD ISAs. Two seemingly similar instructions sequences may get decoded into a different number of uops, which in turn occupy different execution ports. The most extreme example of this being scalar code without stalls. It tends to have high IPC, even though it makes no use of FP/SIMD ports. Keeping the execution pipeline busy is not a useful goal in isolation. Really, IPC is better left as a sanity check.
gnufx 3y ago
Consider appropriate hardware counters. (Note that multiplexing to measure more then three(?) at once can be misleading.) At least for serial code on x86, there's "top-down performance analysis". Some tools (I know Maqao) suggest how to address bottlenecks.