Cache size is constrained by different things at different levels. Cache uses 6 transistors per bit while normal RAM uses 1 transistor and 1 capacitor. You can

Cache size is constrained by different things at different levels. Cache uses 6 transistors per bit while normal RAM uses 1 transistor and 1 capacitor. You can buy a LOT more RAM per dollar than you could buy cache. This has been further exacerbated by poor SRAM scaling with each new fabrication node. It's usually 20-30% instead of 60-80% for other transistors and is actually 0% for TSMC's upcoming N3 node. AMD chose to fab their RDNA 3 GPU die on 5nm, but their SRAM dies on 6nm because the cost savings were much bigger than the effect of a slightly larger die. DRAM typically has a maximum cycle speed of around 300-500MHz due to the capacitors having more restrictive charge/discharge rates (when you see something like 6400MT/s, that's not random access speed, but is instead the aggregate speed of slowly reading thousands of cells in parallel at one time into a cache then quickly sending the results over the wire). Once you move away from bulk storage into high-performance caches, you have to discuss speed, latency, and associativity. As cache gets bigger, keeping lookup times low requires more and more transistors to control the cache and keep it coherent with both the CPU/threads and main memory. If you want to make that controller smaller, it will mean lowering clockspeeds, altering latencies, changing associativity, etc. At a physical level, faster caches will require higher-performance transistors which require more die space and power per transistor too. Sometimes, the tradeoffs aren't what you might expect. A great example is AMD moving from 64kb of I-cache in Zen1 to 32kb I-cache in Zen 2, 3, and 4 because they found the tighter latencies were more important in their uarch than the higher hit rate. There's a further constraint for AMD's 3D chip where the cache reduces heat transfer forcing lower CPU clockspeeds. Unfortunately there's not much room on the substrate to add the cache beside the CPU. Infinity Cache on RDNA3 also uses a significant portion of it's die space for the super-fast interconnect with the GPU die. I suspect a similar thing happens here too.