23 ms·
Why do CPUs have multiple cache levels?
- forgotpwtomain 10y agoI find the real world analogies quite weak and unnecessary. If you haven't read it already, this is worth mentioning: https://people.freebsd.org/~lstewart/articles/cpumemory.pdf https://people.freebsd.org/~lstewart/articles/cpumemory.pdf
- mungoman2 10y agoI think they are great for building intuition. What don't you like about them?
- qq66 10y agoI also dislike most real-world computing analogies because it encourages the use of mental heuristics that are usually just not that relevant to the situation at hand. I remember when there was a big scandal about MBA applicants getting their admissions status by feeding in query strings to the URL, there was a big debate about whether it was like "breaking a lock" or "opening an unlocked door" which doesn't really respect the unique moral reasoning that you need to use to understand the actual situation at hand.
- Sylos 10y agoI generally like real-world analogies, but in this case I do find it too elaborate and unnecessary, too. As a straight answer to the question, it would have sufficed to explain that accessing a larger cache takes more time and resources than accessing a small cache. Then one could have compared that to desk vs. cabinet once to make it visual, but there's no need to extend that analogy for each individual cache level. That just exhausts the reader and makes it near-impossible to tell which parts of the analogy are relevant/accurate and which parts are just fluff to make the analogy work. Alternatively, if you really want to explain each individual detail of caches, then do go with such an elaborate analogy, but then explain at every step to what it corresponds and which part of the analogy is relevant. You shouldn't write out a page-worth of mostly accurate text and then write a paragraph afterwards to explain how the analogy fits. Chances are you've lost half of your readership at that point and many (myself included to be honest) will quit reading at exactly that point, because they feel like you're repeating yourself.
- smallnamespace 10y ago> it would have sufficed to explain that accessing a larger cache takes more time and resources than accessing a small cache. I think the point of the long analogy is to hammer in the intuition that physical locality has a concrete price in the real world, and the cache hierarchy is simply a consequence.
- peyton 10y agoSometimes long analogies like that of the article have trouble making obvious distinctions between design choices and The Way Things Work. For instance, this article's analogy makes assumptions about cache eviction policy and cache coherence that map nicely to an office setting. Other design choices might not map well. So intuition developed here might not be flexible enough to reason about caches in the real world. Not to say the analogy detracts from the article. The author includes a list of caveats. I really enjoyed the piece.
- oolongCat 10y ago>I find the real world analogies quite weak and unnecessary. I actually thought they were good, I always loved when teachers/professors took the time to explain things using real world analogies. Something about them just makes me remember things better.
- amelius 10y agoAlso, their name is already an analogy: pointer.
- EliRivers 10y agoThis kind of thing reminds me of people trying to teach pointers through clumsy analogies to envelopes and letters, or parcels and addresses, or cars and door handles, or some other awful mess. As with pointers, when it comes to caches, a clear explanation without analogies is far superior.
- posterboy 10y agoThe abstract and transparent explanation would involve a lot of maths, in which case a blog post might be the wrong place to start.
- EliRivers 10y agoI'm not convinced it would. They're different kinds of storage, with principal differences in access speed, sharing between cores, physical location and so on. No mathematics needed for a working understanding.
- smallnamespace 10y agoI think you have it backwards. The differences in access speed, density, and power between different types of caches are due to how our physical world works. Therefore,uUsing a physical analogy is totally appropriate since we see the same constraints in different contexts; namely, there's an energetic cost to be paid for increased physical locality, and this is intuitively obvious to most people. The specific details (e.g. eviction policy, cache topology, etc.) are mostly irrelevant for understanding this key point.
- EliRivers 10y agoAnalogies are unhelpful here. I don't need an analogy to help people understand that different kinds of memory have different characteristics. I can just say it. It is easier to understand if I simply state that there are different kinds of memory, and that they have different constraints in terms of speed, density and power. That is really, really easy to understand and from there I can accurately reason about it. I don't need an analogy to understand any of this. Just saying that different kinds of memory have different speed, densities and power consumptions explains everything and provides all the intuition needed.
- i336_ 10y ago> For a “Haswell” or later Core i7 at 3GHz, we’re talking aggregate code+data L1 bandwidths well over 300GB/s per core if you have just the right instruction mix; very unlikely in practice, but you still get bursts of very hot activity sometimes. Reading that reminded me of http://stackoverflow.com/questions/8389648/how-do-i-achieve-the-theoretical-maximum-of-4-flops-per-cycle http://stackoverflow.com/questions/8389648/how-do-i-achieve-.... I don't 100% understand either domain, but I think this link is relevant - it's asking how to achieve the theoretical max of 4 FLOPs per CPU cycle.
- vardump 10y ago> it's asking how to achieve the theoretical max of 4 FLOPs per CPU cycle. Nowadays you can do 32 FLOPs per core per cycle, single precision (counting FMA as add + mul).
- cletus 10y agoGreat SO question. Tanks for the link. Honestly, I'm shocked it hasn't been closed as "not constructive" or "subjective".
- daly 10y agoSuppose you want to make a salad. register: a tomato in your hand level 1 cache: a tomato on the counter level 2 cache: a tomato in the refrigerator level 3 cache: a tomato at the store main memory: a tomato on the plant at the farm disk: a tomato seed being planted
- milcron 10y agoYou need two newline characters for HN to put the text on a new line.
- drwl 10y agoThe post the author wrote goes more in depth than this. Per your analogy, it doesn't quite explain if multiple people (cores) want to make a salad. For example, how does the tomato on the counter and refrigerator get allocated?
- posterboy 10y agoIt's probably over the top of many heads, so a simple explanation is probably what they expect from the title.
- drwl 10y agoI'm confused, are you referring to the answer to the thread question or to the article's explanation?
- posterboy 10y agoObviously the few sentence answer here is much simpler than the (reposted) over my head article discussing cache strategies. If the article contains the answer, it's not up front. It starts with a quote that goes along the lines of "I know why cache is necessary ...", so then I didn't read much further to find out if the explanation will be repeated, because I remember reading the article before. I jumped to the conclusion that the necessity for a cache also explains the necessity for caches of caches. Edit: I was trying to say, the in depth explanation there and the ad-hoc hint given here both have their place. On the other hand, maybe not, if one should read the article first, but who does that, right? /s
- rwmj 10y agoI was "enlightened" many years ago when I asked a colleague (a great electronic engineer) why we didn't do fast task switching by having two sets of registers. His reply was that this would require every regular access to a register to go through an extra gate (to decide which bank of registers you want to hit), making every access slightly slower. Larger registers/caches/memories are slower because they need more address decoding, that time scaling approximately linearly as the storage doubles in size.
- seanmcdirmid 10y agosounds similar to register windows on SPARC, which is rather used for procedure calls. It isn't so much that it is slow, just that when you have a decent level 1 cache it doesn't add much. Right about why small caches are faster.
- sargun 10y agoWhat's the "cost" of registers on an x86-64 chip? How hard would it be to introduce 10 more general purpose registers? Ones that are used by the kernel, say, and one set for user space.
- vardump 10y agoBecause you say kernel, I'm assuming you mean ability to switch a set of registers instead of pushing them on the stack when servicing an interrupt request. Unfortunately you can't get away with just one set of spare registers on x86 interrupt handling. Interrupt can be interrupted by another interrupt. So practically they have to save registers anyways. Unless there'd be one set for each level... So those "10 extra registers" would be practically useless. Having written kernel device driver interrupt handlers, I can also say CPU time is not spent saving and restoring registers, but waiting for glacially slow PCI-e MMIO register loads and stores (don't have exact figures at hand, but I remember one access can take 300-800 nanoseconds, that's thousands of CPU clock cycles). During one such fetch you might be able to store and restore registers tens, if not hundreds of times. X86 interrupt handling is a bit like stopping a freight train to pick up a single letter. However, that kind of feature can be very useful on a microcontroller to help lowering interrupt handling latency.
- Noseshine 10y agoSo this document surely belongs here: What Every Programmer Should Know About Memory https://www.akkadia.org/drepper/cpumemory.pdf https://www.akkadia.org/drepper/cpumemory.pdf You also get an answer for the question asked by the headline of this thread - in great detail (most people will probably skip a lot of details).
- amelius 10y agoI also recommend: https://www.amazon.com/Computer-Architecture-Fifth-Quantitative-Approach/dp/012383872X https://www.amazon.com/Computer-Architecture-Fifth-Quantitat...
- ybaumes 10y agoI though having multiple cache levels was about a trade-off between performances and costs. The closer to the cpus (or the fater cache lvl), the more expensive it is.
- crististm 10y agoYes. I don't know where he gets the idea that a large L1 cache is to a CPU the same as a 150mx150m desk to a human. Address decoding is done in parallel, not sequentially. And desks are as large as people are comfortable to produce and use. Likewise, if the RAM would be as cheap to produce as SRAM like it is as DRAM, it would be as fast as the CPU (since it is using the same technology as the CPU) and we would not need the cache at all. Imagine gigabytes of L1 cache!
- Symmetry 10y agoWell, address decoding can be started in parallel if your page size lets you do virtually indexed, physically tagged caches which applies to only some processors. But that's a separate issue from the relationship between cache size and cache speed. That's governed by three things. First, the larger your cache the more layers of muxing you need to select the data you need, meaning more FO4s of transistor delay. Second, the larger your cache the physically bigger it is. That means more physical distance between the memory location and where it is used. That means more speed of light delay. And third there's the issue of resolving contention for shared versus unshared caches. So despite the fact that you're using the same SRAM in both your L1 and L3 but access to the former takes 4 clock cycle but access to the later takes 80.
- gchadwick 10y agoThere's also the fact that as you get down the cache hierachy the cache becomes more complicated. An L1 does lookups for a single processor, and responds to snoops. An L3 probably has several processors hanging it off and may deal with running the cache coherency protocol (e.g. implements a directory of what lines are where and sends clean or invalidation snoops when someone wants to upgrade a line from shared to unique). As a result you've got layers of buffering, arbitration and hazarding to get through before you can even touch the memory array.
- jgord 10y agoWhen my son was 5 or 6 I had a great discussion about salt containers - the little one you have on the table, the big packet in the pantry, the pallet that gets delivered to the supermarket, the vast piles of salt at the salt mine, and all that salt in the ocean. Next time we had an egg and he wanted salt I scratched my head and asked him what should we do .. take our egg to the supermarket or we could take a pack lunch to the beach, and maybe wave the egg around in the water ? "No daddy, remember, we have a little salt cache in the kitchen." hehe. I guess a lot of the world can be seen thru the glasses of caching data or physical things.
- csours 10y agoThat's a beautiful analogy. I wonder if we would still use small salt shakers if they cost 1000x a large salt container. Edit: Cunningham's Law in action! https://meta.wikimedia.org/wiki/Cunningham%27s_Law https://meta.wikimedia.org/wiki/Cunningham%27s_Law
- CyberDildonics 10y agoFor the amount of salt they contain I would guess that they do.
- zepolen 10y agoFrom 100x to 1250x container (40’) shaker payload: 27,600 kg 100g cost: 1000$ to 5000$ 2$ to 5% cost/kg: 0.04$ to 0.18$/kg $20/kg to $50/kg
- dghughes 10y agoFarming is similar $200/tonne of potatoes and one 25kg bag is $10. Quite a difference in what a farmer gets and what the end products costs although not as bad as salt.
- zepolen 10y agoWtf this isn't Cunningham's. I had the same thought and did the math and decided to post the results since other people were thinking the same thing.
- SixSigma 10y agoOne article worth reading is : Machine perception of time, if only nanoseconds were seconds [1] http://umumble.com/blogs/hardware/machine-perception-of-time,-if-only-nanoseconds-were-seconds/ http://umumble.com/blogs/hardware/machine-perception-of-time...
- dTal 10y agoThis is great.
- rsync 10y agoIsn't cost the reason ? That is, the reason you don't use battery backed DRAM for all of your photos is not because you don't want to, but because 8TB of the stuff would be very expensive. And so most of us have RAM leading to SSD leading to spinning platters. So the reason to have a CPU cache at all is (insert interesting explanations of caches here). But the reason to have more than one CPU cache is because of the relative cost of the first cache, right ? If cost was no object, wouldn't you just have a huge primary cache ?
- vvanders 10y agoNot really, SRAM(what most caches use) isn't just more costly is also consumes a lot more power. You'd be hitting thermal limits much sooner than with traditional DRAM.
- Symmetry 10y agoIs that right? Traditionally DRAM has used much more power because it requires a periodic refresh whereas the power consumption of SRAM is purely leakage. Now, maybe leakage power has grown so much in recent years that this is no longer true but if so I'd find that very surprising. Do you have any numbers?
- vvanders 10y agoIn SRAM you get Leakage + Switching current. For cases like a cache where you're going to be constantly churning bits this can drive up power quite a bit. I don't claim to be an expert so there are probably cases where it's lower. However when I was looking into fast RAM for storing data from an FPGA for a simple Logic Analyzer most of the SRAM was almost 2-4x what DRAM was for power consumption at the same storage size(with SRAM being a blazing fast cycle access).
- Lagged2Death 10y agoA hardware project I worked on some years ago included a chip that contained a small SRAM and a lithium battery, all soldered right onto the board, as a form of nonvolatile storage. Some SRAM can indeed be very low-power, but caches are probably a different animal.
- lamontcg 10y agoIf you're going to combine all the CPU caches into L1 why not also put the RAM and the SSD into the CPU L1 cache as well and just have 1TB of L1 CPU cache? Working out the cost and power consumption and die size of that might be instructive -- as a kind of reductio ad absurdum...
- kristianp 10y agoWhy have L1 caches have been the same size for quite a few generations of Intel Core processors? "Currently Intel's L1D (level 1 data) cache is 512 lines with 64 bytes each, 32 kB. Been that way for a pretty long time. L1D latency with a pointer is mostly 4 cycles. Not sure, but I think having 1024 entries would increase that to 5 cycles.." - vardump, 550 days ago: https://news.ycombinator.com/item?id=9001238 https://news.ycombinator.com/item?id=9001238 The total L1 cache increases as you increase the number of cores though.