6 ms·

Ask HN: Tell CPU as coding C/C++ certain variable must put in cache L1,L2

24 points by mardiyah 4y ago

How do we instruct CPU when we're writing or coding C/C++ certain (usually) global variable must be put in cache L1, L2, or L3 so on ?

colanderman 4y ago
Generally, caches don't hold fixed memory locations. They hold whatever has been most recently used (to a first approximation). Some architectures do support "cache pinning" -- wherein you can instruct the CPU to reserve a portion of cache for a specific memory location (actually cache line or page). To the best of my knowledge, neither Intel nor AMD processors implement such a feature. You can however instruct the CPU to load something into cache prior to using it, using a prefetch instruction. (It is still subject to be evicted from cache at a later time as usual.) In GCC, this is done using __builtin_prefetch() [1]. But -- if you add prefetches blindly, it will almost certainly slow down whatever it is you're trying to fix. You need to analyze your memory access patterns and the assembly code being executed, using tools like perf and llvm-mca, and recognize the cache line usage, cache access latency, pipeline stalls, cross-process and -core contention for the cache line, register pressure, etc. of the code in question, to understand whether a prefetch is appropriate, and where to place it if so. Notably, it's a challenge to get a compiler to emit a prefetch at a useful line of assembly. What evidence are you working from, that access latency to a single specific global variable is a performance bottleneck for your application? To share an anecdote -- the only time I've been in a similar situation, was working on a 10 Gbps line-rate network processor. Many functions had CPU budgets of fewer than 100 cycles. We found (using perf) that often, these cycles were eaten up stalling for various globals to be fetched from DRAM, despite that the globals were accessed with every packet. Notably, prefetching didn't help -- we could not prefetch early enough to avoid the entire stall, and issuing the prefetch itself ate precious CPU cycles. However, the true culprit was that the small handful of globals in question were each allocated on the start of a hugepage boundary -- and therefore assigned to the same cache line. This particular CPU had low cache associativity (L1 was 2-way IIRC), and thus the globals kept bumping each other out of cache. The solution in our case was to manually align the globals to different cache lines. [1] https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html
- mardiyah 4y ago
  I'm so lucky indeed to be replied and given such the answer. Thanks billions.. more importantly must God bless and reward you, ameen !
  - ungamedplayer 4y ago
    Not OP: One thing to remember when doing these kinds of optimizations is that different families of CPU's have different sized caches. If you're optimising one pipeline it may not work correctly on systems that have different sized L1/L2 cache or differing cache handling behavior.
- bXVsbGVy 4y ago
  Intel also implements the "cache pinning" (only for L3). The keyword is "Cache Allocation Technology (CAT)".
pid-1 4y ago
You can't.
- Gibbon1 4y ago
  Even worse anything that monkey's with the CPU's cache logic on a multiprocessor machine will corrupt memory. From a post by Bruce Dawson where he was working with a machine that had an instruction to bypass the cache. It was bad news, doubly so because the machine also did speculative execution.
kadoban 4y ago
Last I remember (which, it has been a while), on x86 and friends anyway, ~all reads go to cache. So the request is somewhat nonsensical I believe? You might then ask how to keep it there, but I doubt you can. Your best bet is to keep the working set of used memory small during the steps you care about. Also could be worth keeping in mind that caches work on "lines", small blocks of data. So you can give yourself a small edge if you keep all the crap you're going to use in one contiguous block of memory.
_moof 4y ago
You're in CPU reference manual territory - and if you got here by accident, you've taken at least seven wrong turns, and are about to take an eighth.
- drpixie 4y ago
  You are standing in a dark cavern. On a pedestal in front of you are several items: a crystal goblet containing a foaming potion an IBM System 360 Basic Assembler Language manual a teletype with a blinking cursor a blunt and rusty knife You hear distant noises, perhaps Orcs or Managers. >
  - cwillu 4y ago
    …managers!? > plunge knife into chest
ceeplusplus 4y ago
A better solution, if you don't need to synchronize writes to said variable across cores, is just to let the compiler figure it out. Dumping the contents of the variable into a register is much faster than accessing L1. If you label the variable as const the compiler should be able to figure out that it's safe to concurrently read it into a register from multiple cores. Unless you are writing highly optimized assembly it's a good bet the compiler will do a better job than whatever you do.
- ncmncm 4y ago
  True, even L1 cache is slow nowadays -- you can retire a dozen instructions in the time it takes to get a word of L1 cache into a register, if you're clever and lucky.
daniel-thompson 4y ago
Not exactly what you're asking, since C/C++ doesn't expose this functionality, but ISPC (a C-derived language that exposes a CUDA-like SPMD model over the vector lanes in your CPU) has some standard library functions to prefetch data into L1 cache; see https://ispc.github.io/ispc.html#prefetches https://ispc.github.io/ispc.html#prefetches for more details. If you're worried about this in your application, you may find it useful to investigate writing your compute kernel in ISPC (it uses C linkage and can be called from C code with no overhead).
josephcsible 4y ago
Why do you think you need/want to do that?
rramadass 4y ago
Oldie but Goodie: https://www.usenix.org/legacy/publications/library/proceedings/als00/2000papers/papers/full_papers/sears/sears_html/index.html https://www.usenix.org/legacy/publications/library/proceedin... See also the book The Art of Multiprocessor Programming.

24 points by mardiyah 4y ago

How do we instruct CPU when we're writing or coding C/C++ certain (usually) global variable must be put in cache L1, L2, or L3 so on ?

colanderman 4y ago

Generally, caches don't hold fixed memory locations. They hold whatever has been most recently used (to a first approximation). Some architectures do support "cache pinning" -- wherein you can instruct the CPU to reserve a portion of cache for a specific memory location (actually cache line or page). To the best of my knowledge, neither Intel nor AMD processors implement such a feature. You can however instruct the CPU to load something into cache prior to using it, using a prefetch instruction. (It is still subject to be evicted from cache at a later time as usual.) In GCC, this is done using __builtin_prefetch() [1]. But -- if you add prefetches blindly, it will almost certainly slow down whatever it is you're trying to fix. You need to analyze your memory access patterns and the assembly code being executed, using tools like perf and llvm-mca, and recognize the cache line usage, cache access latency, pipeline stalls, cross-process and -core contention for the cache line, register pressure, etc. of the code in question, to understand whether a prefetch is appropriate, and where to place it if so. Notably, it's a challenge to get a compiler to emit a prefetch at a useful line of assembly. What evidence are you working from, that access latency to a single specific global variable is a performance bottleneck for your application? To share an anecdote -- the only time I've been in a similar situation, was working on a 10 Gbps line-rate network processor. Many functions had CPU budgets of fewer than 100 cycles. We found (using perf) that often, these cycles were eaten up stalling for various globals to be fetched from DRAM, despite that the globals were accessed with every packet. Notably, prefetching didn't help -- we could not prefetch early enough to avoid the entire stall, and issuing the prefetch itself ate precious CPU cycles. However, the true culprit was that the small handful of globals in question were each allocated on the start of a hugepage boundary -- and therefore assigned to the same cache line. This particular CPU had low cache associativity (L1 was 2-way IIRC), and thus the globals kept bumping each other out of cache. The solution in our case was to manually align the globals to different cache lines. [1] https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html

mardiyah 4y ago
I'm so lucky indeed to be replied and given such the answer. Thanks billions.. more importantly must God bless and reward you, ameen !
- ungamedplayer 4y ago
  Not OP: One thing to remember when doing these kinds of optimizations is that different families of CPU's have different sized caches. If you're optimising one pipeline it may not work correctly on systems that have different sized L1/L2 cache or differing cache handling behavior.
bXVsbGVy 4y ago
Intel also implements the "cache pinning" (only for L3). The keyword is "Cache Allocation Technology (CAT)".

pid-1 4y ago

You can't.

Gibbon1 4y ago
Even worse anything that monkey's with the CPU's cache logic on a multiprocessor machine will corrupt memory. From a post by Bruce Dawson where he was working with a machine that had an instruction to bypass the cache. It was bad news, doubly so because the machine also did speculative execution.

kadoban 4y ago

Last I remember (which, it has been a while), on x86 and friends anyway, ~all reads go to cache. So the request is somewhat nonsensical I believe? You might then ask how to keep it there, but I doubt you can. Your best bet is to keep the working set of used memory small during the steps you care about. Also could be worth keeping in mind that caches work on "lines", small blocks of data. So you can give yourself a small edge if you keep all the crap you're going to use in one contiguous block of memory.

_moof 4y ago

You're in CPU reference manual territory - and if you got here by accident, you've taken at least seven wrong turns, and are about to take an eighth.

drpixie 4y ago
You are standing in a dark cavern. On a pedestal in front of you are several items: a crystal goblet containing a foaming potion an IBM System 360 Basic Assembler Language manual a teletype with a blinking cursor a blunt and rusty knife You hear distant noises, perhaps Orcs or Managers. >
- cwillu 4y ago
  …managers!? > plunge knife into chest

ceeplusplus 4y ago

A better solution, if you don't need to synchronize writes to said variable across cores, is just to let the compiler figure it out. Dumping the contents of the variable into a register is much faster than accessing L1. If you label the variable as const the compiler should be able to figure out that it's safe to concurrently read it into a register from multiple cores. Unless you are writing highly optimized assembly it's a good bet the compiler will do a better job than whatever you do.

ncmncm 4y ago
True, even L1 cache is slow nowadays -- you can retire a dozen instructions in the time it takes to get a word of L1 cache into a register, if you're clever and lucky.

daniel-thompson 4y ago

Not exactly what you're asking, since C/C++ doesn't expose this functionality, but ISPC (a C-derived language that exposes a CUDA-like SPMD model over the vector lanes in your CPU) has some standard library functions to prefetch data into L1 cache; see https://ispc.github.io/ispc.html#prefetches https://ispc.github.io/ispc.html#prefetches for more details. If you're worried about this in your application, you may find it useful to investigate writing your compute kernel in ISPC (it uses C linkage and can be called from C code with no overhead).

josephcsible 4y ago

Why do you think you need/want to do that?

rramadass 4y ago

Oldie but Goodie: https://www.usenix.org/legacy/publications/library/proceedings/als00/2000papers/papers/full_papers/sears/sears_html/index.html https://www.usenix.org/legacy/publications/library/proceedin... See also the book The Art of Multiprocessor Programming.