3 ms·

This whole rigmarole is necessary for a single reason: TLBs don't participate in the cache coherency system. Uh, why is that? If they did participate, then th

This whole rigmarole is necessary for a single reason: TLBs don't participate in the cache coherency system. Uh, why is that? If they did participate, then the mere act of writing to the cache line(s) which change the mapping would implicitly invalidate all of the associated entries in all of the system's TLBs. (Handwave, handwave), maybe you still end up needing a barrier similar to the instruction barrier needed when altering the content of executable pages. What's the downside? Is it just power? Or is there something more fundamental about the TLB structure that makes it impractical?

ip26 6y ago
I would love to hear from someone with the real answer but I've always assumed it is legacy baggage from the old days. The impact on performance has probably never been big enough to compel the entire x86 ecosystem to make them coherent.
- monocasa 6y ago
  I imagine it's because the hardware doesn't have enough context to know which virtual address spaces exist in which cores' TLBs. ASIDs were only added in the virtualization instructions on x86, and I can't think of an OS where those are the same namespace across cores. A side reason is that it'd probably heavily complicate the top level TLBs to have "please flush yourself" coming from anywhere other than the core they're attached to, and those are in the critical path between L1 and L2. Totally spitballing here FWIW, I haven't been part of the design process for a TLB.
eqvinox 6y ago
I'm guessing what you mean is for the TLB to get a notification when the physical memory that contains the page mapping it was loaded from is changed. (As opposed to, some weird contortion the other way around where you touch the cache for the mapping that got changed.) It'd probably require holding the cacheline for the TLB entry in the actual cache in at least MESI "S" state. For 5 levels of page tables. And you can't do non-flushing changes (e.g. dirty bit) anymore. I'm no CPU designer but it sounds complicated and bug prone...
- brandmeyer 6y ago
  > I'm guessing what you mean is for the TLB to get a notification when the physical memory that contains the page mapping it was loaded from is changed. Yes, that's right. Let us conceive of an inclusive cache architecture, just for the sake of argument. L2 already maintains a directory listing of all the lines which are present in L1I and L1D, and forwards cache coherency traffic to those caches based on the messages it receives. Expanding this to the ITLB and DTLB would be exactly the same circuits. The hard parts are already done. But I think you're onto something with the hardware-managed dirty bit. Its much less general than the S->E->M transition, and doesn't need the Exclusive state at all. I've done a bit more digging in the background. Turns out that ARMv8's TLB invalidate instructions are broadcasted operations. You don't need IPIs to execute them on every core. So processors do have some snoopy and/or directory management hardware interface for managing TLB shootdown. It just isn't as general-purpose as the rest of the cache maintenance hardware.
  - eqvinox 6y ago
    A separate broadcast makes much more sense on the complexity front... if it were tied to the cache, the TLB wouldn't really care about anything other than specific transitions. It doesn't need the cacheline contents, and one cacheline would generally hold 8 or 16 PTEs (depending on PTE and cacheline size.) You'd also necessarily be binding to the PTE's physical address since page table walks occur on physical addresses. With a broadcast, on the other hand, you get the specific transition event you need, and you can choose whether to broadcast the physical PTE, physical data or virtual (+ASID) address, or both. You can also have an acknowledgement returned if needed.
ncmncm 6y ago
It is because where it matters, it is too often easy to avoid the problem by better system design. And, it doesn't show up in benchmarks, in part because benchmarks don't unmap, or because they are well designed systems that neatly sidestep the problem.
gpderetta 6y ago
Exactly. It is certainly possible to make TLB coherent: IIRC IBM mainframe architecture has coherent TLBs. But probably has a non trivial cost to implement and and hasn't been done yet for most common architectures. I wouldn't be surprised if it became a thing at some point i t the near future though, as it can be a win for high core count CPUs.

ip26 6y ago

I would love to hear from someone with the real answer but I've always assumed it is legacy baggage from the old days. The impact on performance has probably never been big enough to compel the entire x86 ecosystem to make them coherent.

monocasa 6y ago
I imagine it's because the hardware doesn't have enough context to know which virtual address spaces exist in which cores' TLBs. ASIDs were only added in the virtualization instructions on x86, and I can't think of an OS where those are the same namespace across cores. A side reason is that it'd probably heavily complicate the top level TLBs to have "please flush yourself" coming from anywhere other than the core they're attached to, and those are in the critical path between L1 and L2. Totally spitballing here FWIW, I haven't been part of the design process for a TLB.

eqvinox 6y ago

I'm guessing what you mean is for the TLB to get a notification when the physical memory that contains the page mapping it was loaded from is changed. (As opposed to, some weird contortion the other way around where you touch the cache for the mapping that got changed.) It'd probably require holding the cacheline for the TLB entry in the actual cache in at least MESI "S" state. For 5 levels of page tables. And you can't do non-flushing changes (e.g. dirty bit) anymore. I'm no CPU designer but it sounds complicated and bug prone...

brandmeyer 6y ago
> I'm guessing what you mean is for the TLB to get a notification when the physical memory that contains the page mapping it was loaded from is changed. Yes, that's right. Let us conceive of an inclusive cache architecture, just for the sake of argument. L2 already maintains a directory listing of all the lines which are present in L1I and L1D, and forwards cache coherency traffic to those caches based on the messages it receives. Expanding this to the ITLB and DTLB would be exactly the same circuits. The hard parts are already done. But I think you're onto something with the hardware-managed dirty bit. Its much less general than the S->E->M transition, and doesn't need the Exclusive state at all. I've done a bit more digging in the background. Turns out that ARMv8's TLB invalidate instructions are broadcasted operations. You don't need IPIs to execute them on every core. So processors do have some snoopy and/or directory management hardware interface for managing TLB shootdown. It just isn't as general-purpose as the rest of the cache maintenance hardware.
- eqvinox 6y ago
  A separate broadcast makes much more sense on the complexity front... if it were tied to the cache, the TLB wouldn't really care about anything other than specific transitions. It doesn't need the cacheline contents, and one cacheline would generally hold 8 or 16 PTEs (depending on PTE and cacheline size.) You'd also necessarily be binding to the PTE's physical address since page table walks occur on physical addresses. With a broadcast, on the other hand, you get the specific transition event you need, and you can choose whether to broadcast the physical PTE, physical data or virtual (+ASID) address, or both. You can also have an acknowledgement returned if needed.

ncmncm 6y ago

It is because where it matters, it is too often easy to avoid the problem by better system design. And, it doesn't show up in benchmarks, in part because benchmarks don't unmap, or because they are well designed systems that neatly sidestep the problem.

gpderetta 6y ago

Exactly. It is certainly possible to make TLB coherent: IIRC IBM mainframe architecture has coherent TLBs. But probably has a non trivial cost to implement and and hasn't been done yet for most common architectures. I wouldn't be surprised if it became a thing at some point i t the near future though, as it can be a win for high core count CPUs.