4 ms·

RISC-V actually calls out the posibility of a software filled TLB in it's spec. It's somewhere on my list of things I want to go try and do - to create a small

by bpye 5y ago

RISC-V actually calls out the posibility of a software filled TLB in it's spec. It's somewhere on my list of things I want to go try and do - to create a small RV32IC core capable of running user, supervisor and machine mode and have a software TLB for paging.

johndoe0815 5y ago
The RISC-V Privileged Architecture spec (V20211203) [1] advises against a software-managed TLB (section 4.3) due to being a "performance bottleneck on high-performance systems, and [are] especially troublesome with decoupled specialized coprocessors", but this is definitely a topic worth investigating IMO. An interesting related document is Gernot Heiser's "Inside L4/MIPS" [2], which details the design decisions and the implementation of software TLB management for an early version of the L4 microkernel on 64-bit MIPS systems. [1] https://github.com/riscv/riscv-isa-manual/releases/download/Priv-v1.12/riscv-privileged-20211203.pdf https://github.com/riscv/riscv-isa-manual/releases/download/... [2] https://trustworthy.systems/publications/papers/Heiser%3AIL4.pdf https://trustworthy.systems/publications/papers/Heiser%3AIL4...
- Taniwha 5y ago
  More importantly there are no RISCV "reserved for kernel use" registers that can be trashed at any time - the mips register convention leaves 2 of the 31 registers for the kernel to use in exceptions (including TLB misses). To do this for RISCV you would have to build build special compilers and kernels
  - johndoe0815 5y ago
    On RISC-V, I've seen the trick of swapping a register with the m/sscratch CSR in order to obtain one freely useable one. But this, of course, still is more overhead than on MIPS.
    - Taniwha 5y ago
      yup, and a swap is more difficult because (depending on the design) is more likely to generate pipe bubbles. The mips also has (quite simple) magic hardware to convert the TLB miss address into a PTE address so that you can create the address you need in 1 clock, you could do some magic like that where the csr swap saved the reg in scratch and returned the PTE address, then the "write tlb entry" would also swap from scratch - you could do something like (pseudo code) : tlb_fault: movcsr r, pte_addr, r ldd r, (r) movcsr r, tlb_data, r mret (tlb entry would use the saved fault virtual address and the result from the ldd - the "ldd" would fault to a special pde vector if there's no TLB entry, the "movcsr r, tlb_data, r" would fault if you try and write an invalid PTE. BTW I think this only makes sense for simple RISC system - a massive super-scalar OOO system (like mine) would suffer massively if you had to toss all the (100+) instructions in the pipeline when you took a TLB miss
      - Taniwha 5y ago
        (in other words it's a bit of an architectural dead end, like explicit branch delay slots if you want to grow a design past the simple risc stage)
      - Taniwha 5y ago
        You could also expand a TLB miss "ld r1, (r2)" inline to: movcsr r, pte_addr, r ldd r, (r) movcsr r, tlb_data, r ld r1, (r2) With appropriate traps as described above (that's essentially how I handle vectored CLIC interrupts on my RISCV)
  - Taniwha 5y ago
    BTW one of the coolest things is the way that MIPs (and other) systems do 2 level page tables - essentially they map the page tables into their virtual address space (mapped by TLBs just like anything else) and then when they get a miss on a TLB fetch from a PTE they take a recursive trap and fill the TLB entry that matches the upper level PDE A long time ago I worked on an x86 clone built on top of a 64-bit RISC core (pre amd64) we used this trick (and more) to do segmentation as well (to get around the then somewhat bogus Intel paging patents)
    - throwawaylinux 5y ago
      Virtual linear page table. Neat trick. I think ia64 had an option to do that as well. But I've never seen a really good study of them on a decent range of interesting workloads. Primary TLB misses have no dependent memory accesses (just the PTE) but at a cost of another TLB entry. Could be a win in some cases. Nobody really does it anymore though, it could be there is not much appetite for the risk of trying out different virtual memory schemes these days.

by bpye 5y ago

johndoe0815 5y ago

The RISC-V Privileged Architecture spec (V20211203) [1] advises against a software-managed TLB (section 4.3) due to being a "performance bottleneck on high-performance systems, and [are] especially troublesome with decoupled specialized coprocessors", but this is definitely a topic worth investigating IMO. An interesting related document is Gernot Heiser's "Inside L4/MIPS" [2], which details the design decisions and the implementation of software TLB management for an early version of the L4 microkernel on 64-bit MIPS systems. [1] https://github.com/riscv/riscv-isa-manual/releases/download/Priv-v1.12/riscv-privileged-20211203.pdf https://github.com/riscv/riscv-isa-manual/releases/download/... [2] https://trustworthy.systems/publications/papers/Heiser%3AIL4.pdf https://trustworthy.systems/publications/papers/Heiser%3AIL4...

Taniwha 5y ago
More importantly there are no RISCV "reserved for kernel use" registers that can be trashed at any time - the mips register convention leaves 2 of the 31 registers for the kernel to use in exceptions (including TLB misses). To do this for RISCV you would have to build build special compilers and kernels
- johndoe0815 5y ago
  On RISC-V, I've seen the trick of swapping a register with the m/sscratch CSR in order to obtain one freely useable one. But this, of course, still is more overhead than on MIPS.
  - Taniwha 5y ago
    yup, and a swap is more difficult because (depending on the design) is more likely to generate pipe bubbles. The mips also has (quite simple) magic hardware to convert the TLB miss address into a PTE address so that you can create the address you need in 1 clock, you could do some magic like that where the csr swap saved the reg in scratch and returned the PTE address, then the "write tlb entry" would also swap from scratch - you could do something like (pseudo code) : tlb_fault: movcsr r, pte_addr, r ldd r, (r) movcsr r, tlb_data, r mret (tlb entry would use the saved fault virtual address and the result from the ldd - the "ldd" would fault to a special pde vector if there's no TLB entry, the "movcsr r, tlb_data, r" would fault if you try and write an invalid PTE. BTW I think this only makes sense for simple RISC system - a massive super-scalar OOO system (like mine) would suffer massively if you had to toss all the (100+) instructions in the pipeline when you took a TLB miss
    - Taniwha 5y ago
      (in other words it's a bit of an architectural dead end, like explicit branch delay slots if you want to grow a design past the simple risc stage)
    - Taniwha 5y ago
      You could also expand a TLB miss "ld r1, (r2)" inline to: movcsr r, pte_addr, r ldd r, (r) movcsr r, tlb_data, r ld r1, (r2) With appropriate traps as described above (that's essentially how I handle vectored CLIC interrupts on my RISCV)
- Taniwha 5y ago
  BTW one of the coolest things is the way that MIPs (and other) systems do 2 level page tables - essentially they map the page tables into their virtual address space (mapped by TLBs just like anything else) and then when they get a miss on a TLB fetch from a PTE they take a recursive trap and fill the TLB entry that matches the upper level PDE A long time ago I worked on an x86 clone built on top of a 64-bit RISC core (pre amd64) we used this trick (and more) to do segmentation as well (to get around the then somewhat bogus Intel paging patents)
  - throwawaylinux 5y ago
    Virtual linear page table. Neat trick. I think ia64 had an option to do that as well. But I've never seen a really good study of them on a decent range of interesting workloads. Primary TLB misses have no dependent memory accesses (just the PTE) but at a cost of another TLB entry. Could be a win in some cases. Nobody really does it anymore though, it could be there is not much appetite for the risk of trying out different virtual memory schemes these days.