3 ms·

You're right in terms of fitting your program to memory, so that it can run in the first place. But in performance work, the relative speed of RAM relative to

by fulafel 8mo ago

You're right in terms of fitting your program to memory, so that it can run in the first place. But in performance work, the relative speed of RAM relative to computation has dropped such that it's a common wisdom to treat today's cache as RAM of old (and today's RAM as disk of old, etc). In software performance work it's been all about hitting the cache for a long time. LLMs aren't too amenable to caching though.

seanmcdirmid 8mo ago
LLMs need memory bandwidth to stream lots of data through quickly, not so much caching. Well, this is basically the same way that a GPU uses memory.
- zozbot234 8mo ago
  OTOH, LLM inference tends to have very predictable memory access patterns. So well-placed prefetch instructions that can execute predictable memory fetches in parallel with expensive compute might help CPU performance quite a bit. I assume that this is done already as part of optimized numerical primitives such as GEMM, since that's where most of the gain would be.
makapuf 8mo ago
AFAIK, you can't explicitly allocate cache like you allocate RAM however. A bit like if you could only work on files and ram was used for cache. Maybe I am mistaken ? (Edit: typo)
- lou1306 8mo ago
  You can't explicitly allocate cache, but you can lay things out in memory to minimize cache misses.
  - crakenzak 8mo ago
    Out of curiosity, why has there not been a slight paradigm shift in modern system programming languages to expose more control over the caches?
    - greybcg 8mo ago
      There are cache control instructions already. The reason why it goes no further than prefetch/invalidate hints is probably because exposing a fuller api on the chip level to control the cache would overcomplicate designs, not be backwards compatible/stable api. Treating the cache as ram would also require a controller, which then also needs to receive instructions, or the cpu has to suddenly manage the cache itself. I can understand why they just decide to bake the cache algorithms into hardware, validate it and be done with it. Id love if a hardware engineer or more well-read fellow could chime in.
      - Someone 8mo ago
        Another reason for doing cache algorithms in hardware is that cache access (especially for level 1 caches) has to be low latency to be useful.
    - rwmj 8mo ago
      There has! Intel has Cache Acceleration Technology, and I was very peripherally involved in reviewing research projects at Boston University into this. One that I remember was allowing the operating system to divide up cache and memory bandwidth for better prioritization. https://www.intel.com/content/www/us/en/developer/articles/technical/introduction-to-cache-allocation-technology.html https://www.intel.com/content/www/us/en/developer/articles/t...
    - Tuna-Fish 8mo ago
      Because programmers are in general worse at managing them than the basic LRU algorithm. And because the abstraction is simple and easy enough to understand that when you do need close control, it's easy to achieve by just writing to the abstraction. Careful control of data layout and nontemporal instructions are almost always all you need.
    - 8mo ago
  - tux3 8mo ago
    A fun fact for the people who like to go on rabbit holes. There is an x86 technique called cache-as-RAM (CAR) that allows you to explicitly allocate a range of memory to be stored directly in cache, avoiding the DRAM entirely. CAR is often used in early boot before the DRAM is initialized. It works because the x86 disable cache bit actually only decouples the cache from the memory, but the CPU will still use the cache if you primed it with valid cache lines before setting the cache disable bit. So the technique is to mark a particular range of memory as write-back cacheable, prime the cache with valid cache lines for the entire region, and then set the bit to decouple the cache from memory. Now every access to this memory region is a cache hit that doesn't write back to DRAM. The one downside is that when CAR is on, any cache you don't allocate as memory is wasted. You could allocate only half the cache as RAM to a particular memory region, but the disable bit is global, so the other half would just sit idle.
    - antonkochubey 8mo ago
      Thanks – I was wondering how code that initializes DRAM actually runs
- KeplerBoy 8mo ago
  You can in CUDA. You can have shared memory which is basically L1 cache you have full control over. It's called shared memory because all threads within a block (which reside on a common SM) have fast access to it. The downside: you now have less regular L1 cache.
KellyCriterion 8mo ago
Reminds me somehow on: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu... ;-)

by fulafel 8mo ago

seanmcdirmid 8mo ago

LLMs need memory bandwidth to stream lots of data through quickly, not so much caching. Well, this is basically the same way that a GPU uses memory.

zozbot234 8mo ago
OTOH, LLM inference tends to have very predictable memory access patterns. So well-placed prefetch instructions that can execute predictable memory fetches in parallel with expensive compute might help CPU performance quite a bit. I assume that this is done already as part of optimized numerical primitives such as GEMM, since that's where most of the gain would be.

makapuf 8mo ago

AFAIK, you can't explicitly allocate cache like you allocate RAM however. A bit like if you could only work on files and ram was used for cache. Maybe I am mistaken ? (Edit: typo)

lou1306 8mo ago
You can't explicitly allocate cache, but you can lay things out in memory to minimize cache misses.
- crakenzak 8mo ago
  Out of curiosity, why has there not been a slight paradigm shift in modern system programming languages to expose more control over the caches?
  - greybcg 8mo ago
    There are cache control instructions already. The reason why it goes no further than prefetch/invalidate hints is probably because exposing a fuller api on the chip level to control the cache would overcomplicate designs, not be backwards compatible/stable api. Treating the cache as ram would also require a controller, which then also needs to receive instructions, or the cpu has to suddenly manage the cache itself. I can understand why they just decide to bake the cache algorithms into hardware, validate it and be done with it. Id love if a hardware engineer or more well-read fellow could chime in.
    - Someone 8mo ago
      Another reason for doing cache algorithms in hardware is that cache access (especially for level 1 caches) has to be low latency to be useful.
  - rwmj 8mo ago
    There has! Intel has Cache Acceleration Technology, and I was very peripherally involved in reviewing research projects at Boston University into this. One that I remember was allowing the operating system to divide up cache and memory bandwidth for better prioritization. https://www.intel.com/content/www/us/en/developer/articles/technical/introduction-to-cache-allocation-technology.html https://www.intel.com/content/www/us/en/developer/articles/t...
  - Tuna-Fish 8mo ago
    Because programmers are in general worse at managing them than the basic LRU algorithm. And because the abstraction is simple and easy enough to understand that when you do need close control, it's easy to achieve by just writing to the abstraction. Careful control of data layout and nontemporal instructions are almost always all you need.
  - 8mo ago
- tux3 8mo ago
  A fun fact for the people who like to go on rabbit holes. There is an x86 technique called cache-as-RAM (CAR) that allows you to explicitly allocate a range of memory to be stored directly in cache, avoiding the DRAM entirely. CAR is often used in early boot before the DRAM is initialized. It works because the x86 disable cache bit actually only decouples the cache from the memory, but the CPU will still use the cache if you primed it with valid cache lines before setting the cache disable bit. So the technique is to mark a particular range of memory as write-back cacheable, prime the cache with valid cache lines for the entire region, and then set the bit to decouple the cache from memory. Now every access to this memory region is a cache hit that doesn't write back to DRAM. The one downside is that when CAR is on, any cache you don't allocate as memory is wasted. You could allocate only half the cache as RAM to a particular memory region, but the disable bit is global, so the other half would just sit idle.
  - antonkochubey 8mo ago
    Thanks – I was wondering how code that initializes DRAM actually runs
KeplerBoy 8mo ago
You can in CUDA. You can have shared memory which is basically L1 cache you have full control over. It's called shared memory because all threads within a block (which reside on a common SM) have fast access to it. The downside: you now have less regular L1 cache.

KellyCriterion 8mo ago

Reminds me somehow on: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu... ;-)