4 ms·
How L1 and L2 CPU caches work, and why they’re an essential part of modern chips
- ganzuul 12y agoARM is using L3 cache as a DMA-like interconnect, on the SoC side of things.
- shortsightedsid 12y agoI'm not sure about ARMv8 but in ARMv7 SoCs the ARM Subsystem does not include L3. Plus each L1 is tied to each core.
- ganzuul 12y agoI believe this is a relevant source: http://www.arm.com/products/system-ip/interconnect/corelink-ccn-504-cache-coherent-network.php http://www.arm.com/products/system-ip/interconnect/corelink-... The L3 cache is 'integrated', which I presume means it's different names for the same thing.
- shortsightedsid 12y agoAh! Thanks for sharing. It is for ARMv8 which I really need to read up on.
- userbinator 12y agoIn the late 70s/early-to-mid 80s memory was actually faster than the CPU, and this is partly what motivated the RISC philosophy - at the time, instruction decoding was the main bottleneck and there was plenty of memory bandwidth available so it was a good idea to design CPUs to make better use of that. Now that it's the exact opposite, and we have multilevel caches, I wonder how different CPUs would be today if the engineers back then had realised that their surplus of memory performance would be very short-lived and cores would become faster while latencies to access memory continue to increase, and designed for this long term situation.
- colanderman 12y agoDoes memory really bottleneck instruction throughput though? I've never ever seen code that stalled on an instruction fetch. CPUs generally hide latency by auto-prefetching the instruction stream into cache, and memory bandwidth has been keeping up just fine with CPU speed. Hence memory latency rarely hinders instruction throughput, unless your code jumps around unpredictably every few instructions (most doesn't; the only common class of program I can think of which does are emulators/interpreters). Probably the only differences in ISA design/implementation between this universe and one where memory latency tracked CPU speed would be that (a) instructions would need not be prefetched into cache (possibly there'd be no cache), and (b) we'd still have wacky instructions like the TMS9900's X instruction, which executes the instruction at a given memory address. [1] [1] http://en.wikipedia.org/wiki/Texas_Instruments_TMS9900#Instruction_set_and_addressing http://en.wikipedia.org/wiki/Texas_Instruments_TMS9900#Instr...
- sliverstorm 12y agoYou totally can hide latency with auto-prefetching. But there are a few problems. - This depends on correct branch prediction, otherwise you could easily wind up having fetched the wrong fork - L2 latency is, what, 12 cyles? L3 is 28? (Sandy Bridge). To completely hide this latency, we would need the prefetcher to run 28 cycles ahead. Considering caches work in cachelines and not instructions, this means we have to run potentially hundreds of instructions ahead of execution, without mispredicting one branch... Whether your code is going to bottleneck all depends on size and code patterns. If your software fits in the L1 icache, you will only miss when context-switching. If your software is 100MB and does not exhibit good code locality, you might be stalling all over the place.
- ganzuul 12y agoIs this code locality what makes functional style programming perform well?
- colanderman 12y ago
- mmphosis 12y agoAll that complexity (and space) to handle the cache, why not simply add more cache, lots more cache, and don't use RAM at all?
- acveilleux 12y agoCost. More cache means bigger die size and lower yield. Also once the area of the cache gets too big, the latency increases too much. That's why the P3 moved the L2 cache back on the die while it was in a separate package on the P2...
- sliverstorm 12y agoAlso density. DRAM is orders of magnitude more dense per silicon area than SRAM, but you can't generally put DRAM cells on the same chip as the CPU.
- Sharlin 12y agoYes. It's important to realize that at current clock speeds, much of the latency is simply the direct result of the finite speed of light. You simply can't fit gigabytes of memory in a location that's physically close enough to the processor.
- techrat 12y agoHence the location of memory slots always being physically parallel to the CPU socket, squiggles in the traces in the innermost lines from memory socket to CPU (to equal the distance of the longer traces along the edges of the socket to the CPU so timings remain in sync), and why die shrinks are more important for performance than scaling out... If a chip is too physically big, the speed of light can limit its performance before other inefficiencies will. Really fascinating and reinforces how no matter how one can imagine how much "faster" computers will get in the future, there is still that speed limit of information.
- xxs 12y ago>>much of the latency is simply the direct result of the finite speed of light.<< This should be in bold: besides the 6 transistors per bit for L1, there is so much space to fit on a die that's close enough to the core.
- hcrisp 12y agoRelevant article by Francesc Alted, creator of PyTables: "Why Modern CPUs Are Starving And What Can Be Done About It". http://www.pytables.org/docs/CISE-12-2-ScientificPro.pdf http://www.pytables.org/docs/CISE-12-2-ScientificPro.pdf
- jokoon 12y agoIsn't there any C/C++ tutorial on how to avoid cache misses ? I already read the classical argument of having data oriented designs and to avoid linked lists, but I've never really heard of anything concrete on performance in programming. All I hear is "profile profile profile". I guess most of the time it will point out a place where my code is slow, but if I find one, how do I solve it ? For example, can I create some simple bit of code that do a cache miss at will ? Any example of such code ? How do I detect a cache miss ?
- ehsanu1 12y agoThe reason all you hear is "profile profile profile" is that the performance implications of a given code change may sometimes be completely non-obvious, due to all the complex trickery in the processor and cache hierarchy. So what do you do after profiling and finding a performance bottleneck? If you can't identify exactly what the bottleneck's cause is and the solution is not obvious, I suppose all that's left is to do a best guess of the cause and think of a possible solution. Then profile after implementing your solution to check if it made a (positive) difference. And do the profiling/benchmarks even if the solution seems obvious, since it's so easy to be wrong. Detecting cache misses can be done with valgrind: http://valgrind.org/docs/manual/cg-manual.html http://valgrind.org/docs/manual/cg-manual.html There are architecture-specific instruction-level profilers as well. This video mentions a couple and is also a great overview of how non-intuitive performance can be with modern CPUs: http://channel9.msdn.com/Events/Build/2014/4-587 http://channel9.msdn.com/Events/Build/2014/4-587
- TillE 12y agoAccessing contiguous chunks of memory in order minimizes cache misses. It's not a whole lot more complicated than that. This is a good intro video on the subject: https://www.youtube.com/watch?v=16ZF9XqkfRY https://www.youtube.com/watch?v=16ZF9XqkfRY
- jokoon 12y agowoooh thanks a lot for that video! I hope this will bother donald knuth :)
- 12y ago
- bluedino 12y agoThis is probably a simple question, but it may not be obvious to others as well: How effective are caches on a system that is running multiple different processes at one time, much less hundreds? Is each working set that a process uses at any certain time small enough to fit in the cache along with the others? Looking back at caches that used to be 4k or 64k (external) and are now 256k and measured in megabytes (external), I suppose that could be the case.
- pron 12y agoThe number of threads a CPU can run at once is equal to the number of cores (we'll ignore hyperthreading for the sake of simplicity), and the number of cores is equal to the number of (L1) caches. If you're referring to kernel-level multitasking, then these processes don't really run concurrently from the CPUs perspective; in fact, they're switching interval is quite long (relative to the processor speed). However, the cost of bringing in a thread's data into the cache once it's been scheduled by the kernel is not negligible at all, and is part of the significant total task-switching cost.