3 ms·
Understanding x86_64 Paging
- LAC-Tech 3y agoVery cool. I've actually been trying to learn about this area recently. I know this was aside from the main thrust of the article, but this but made me think: Page Cache Disabled (PCD) – pages descendant of this PGD entry should not enter the CPU’s cache hierarchy, sometimes also called the ‘Uncacheable’ (UC) bit. I guess this is some optimisation where it's decided the page isn't worth taking up valuble CPU cache state. And now I'm wondering what this algorithm is that decides it..
- mark_undoio 3y agoMy guess is that this is useful when interacting with memory-mapped IO devices - individual accesses can have meaning at the device level, so you don't want the caches getting in the way (and e.g. removing / combining accesses or providing stale data).
- kimixa 3y agoAlso useful for memory you know you won't be touching multiple times and just thrash the cache. For example GPU drivers and texture copying - even on integrated GPUs that have cache coherency with the CPU (so don't "need" to manually flush it), there's no point filling the CPU cache with data you're only touching once with effectively a memcpy() on the CPU, you'll just evict more useful data you'll probably need to readback after anyway.
- loeg 3y agoUC writes are very inefficient/slow. They are synchronous and not batched into cache lines. Oldschool GPUs used a UC-adjacent mode called WC (write combine) which was almost identical to UC just with adjacent-writes batched into full cache line writes. I think newer GPUs have dedicated DMA engines for copying to GPU memory that isn't mapped into the host address space but I'm not really familiar with these drivers. For anything that isn't MMIO, you would prefer using ordinary WB caching and non-temporal stores to avoid populating the cache instead of UC mode.
- oasisaimlessly 3y agoI believe that UC writes are indeed slow, but are not actually fully synchronous. PCI "write posting" is a thing: > Writes to MMIO space allow the CPU to continue before the transaction reaches the PCI device. HW weenies call this “Write Posting” because the write completion is “posted” to the CPU before the transaction has reached its destination. https://www.kernel.org/doc/html/v5.6/PCI/pci.html https://www.kernel.org/doc/html/v5.6/PCI/pci.html
- loeg 3y agoYeah, PCI writes can be posted. It's still very very slow and will end up using an inefficient small transaction size on the PCIe side of things. And posting doesn't help with reads, of course. PCI devices that are cache coherent and can cache snoop and otherwise work with WB cache mode can be written a lot faster. UC writes are something like sequentially consistent (program order reordering with other operations is not allowed) which limits the OoO / superscalar magic the CPU can otherwise do to make code fast. I like this series as an intro: https://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-1 https://xillybus.com/tutorials/pci-express-tlp-pcie-primer-t...
- monocasa 3y ago> I think newer GPUs have dedicated DMA engines for copying to GPU memory that isn't mapped into the host address space but I'm not really familiar with these drivers. Not even newer, but instead it's a pretty common feature for GPUs for the past couple decades or so.
- PhilipRoman 3y agoI don't think it's an optimization, for that you would typically use non-temporal instructions which avoid polluting cache with one-off accesses.
- ithkuil 3y agoIs this useful for memory mapped IO?
- loeg 3y agoUC is for MMIO.
- foobiekr 3y agoIt's for mapped I/O primarily.
- commandersaki 3y agoThis is great. I find the x86_64 paging to be very similar to RISC V which I was exposed to poring through xv6 code.
- snvzz 3y agoThere's definitely both "established best practices" and many other legacy ways to do things on x86. RISC-V is quite boring, majorly opting for well-proven approaches. And a lot of its technical value comes from doing just that. Yet x86-64 and ARM would have a hard time justifying their complexity, when RISC-V achieves the same with simplicity, and matches or beats them on objective metrics[0]. 0. https://dl.acm.org/doi/pdf/10.1145/3624062.3624233 https://dl.acm.org/doi/pdf/10.1145/3624062.3624233
- MichaelRo 3y ago[flagged]
- phendrenad2 3y agoEasy on paper, hard to get right when trying to do this in C or Rust!
- drxzcl 3y agoIt diverges more from x86 than I would have expected!
- codeflo 3y agoI’m not deeply familiar with either, though I have written a toy x86_64 bootloader once. There are more levels in x86_64 (which makes sense to me that a larger address space would benefit from a sparser encoding). What else is significantly different?
- loeg 3y agox86 in some modes (PSE but not PAE) has 4MB superpages, whereas in PAE mode or in x86_64 they're 2MB (first level). That's the only one I remember off-hand.
- RavSS 3y agoThere are also 1 GiB pages in long mode, but support depends on the processor and CPUID's output. Almost all modern x86 processors in the last several years should have it, I think.
- monocasa 3y agoSome also technically support it, but it doesn't gain you a whole lot as the TLB will split it down to a 2MB TLB entry. Could save a bit of L2 for the page table entries under the right conditions, but you've already lost if invoking your page walk hardware is in your fast path to begin with.
- deleted 3y ago[deleted]
- bhakunikaran 3y agoReally enjoyed this deep dive into x86_64 paging.