5 ms·

Applications don't get 4GB with a 32-bit address space. The practical split between application and kernel was usually 1-3 or 2-2 with 3-1 being experimental an

Applications don't get 4GB with a 32-bit address space. The practical split between application and kernel was usually 1-3 or 2-2 with 3-1 being experimental and mooted with the switch to 64-bit. Nowadays with VRAM being almost as large as main RAM, you need the larger address space just to map a useful chunk of it in. When you factor in memory fragmentation, you really only had a solid 0.75-1.5GB of space that could be kept continuously in use. That was starting to become a problem even when 32-bit was the only practical option. A lot of games saw a benefit to just having the larger address space, such that they ran better in 64-bit with only 4GB of RAM despite the fatter 64-bit pointers.

vlovich123 1y ago
I believe that's an accident of the evolutionary path chosen with syscalls. If we'd instead gone with a ring buffer approach to make requests, then you'd never need to partition the memory address space; the kernel has its memory and userspace has its and you don't need the situation where the kernel is always mapped.
- kbolino 1y ago
  Hmm. I don't understand how that would work. I think it would be possible for e.g. microkernels to greatly reduce the size of the reservation (though not to eliminate it entirely). However, I can't imagine how you would handle the privilege escalation issue without having at least some system code in the application's virtual address space that's not modifiable by the application.
  - vlovich123 1y ago
    I'm not sure how privilege escalation would be an issue since you'd never escalate privilege in the first place (I'm assuming you're talking about CPU ring privileges and not OS privileges). You'd just enqueue into the shared kernel/user space ring buffer your operations and the kernel would pick them up on its side, but you'd never jump between rings. Such a design may require at least one processor dedicated to running the kernel at all times, so it might not work on a single processor architecture. However, single processor architectures might be supportable by having the "kernel process" go to sleep by arming a timer and the timer interrupt is the only one that's specially mapped so it can modify the page table to resume the kernel (for handling all the ring buffers + scheduling). As you note, there's some reserved address space but it's a trivial amount just to be able to resume running the kernel. I don't think it has anything to do with monolithic vs microkernels.
    - kbolino 1y ago
      True, you don't have to go full microkernel just to have messages passed though a buffer. However, if the buffer is shared by all processes, it does need to have some protection. I guess you could assign one buffer per process (which might end up using a lot of physical RAM), and then just crash the process if it corrupts its own buffer. The bigger issue with this approach might be adapting to asynchrony though.
      - vlovich123 1y ago
        It wouldn't be by all processes. One per process just like with io_uring. Not sure how it would end up being all that much physical RAM - you get a lot more memory mapped when you just start a process. Page faults might be another tricky corner case.
    - adgjlsfhk1 1y ago
      I have for a while wondered why we don't have "security core"s that are are really slow, but don't have caches or speculative execution so you can run security critical code without having to worry about CPU timing attacks
      - adwn 1y ago
        I think that would be of little help if the faster, less-secure cores have access to the same memory system as the secure cores. If the JavaScript engine of your browser runs on the fast cores, and you're visiting a malicious website, then vulnerabilities due to speculative execution on the fast cores could still leak information that was written by the slow cores. And you really wouldn't want to run the JS engine on cores without caches and speculative execution, at least not for everyday browsing.
      - kbolino 1y ago
        What problem are you trying to solve? If you just want constant-time operations, you just need dedicated instructions. Maybe they run on dedicated cores, but that's an implementation detail. I think this is a good idea, and it feels like we're heading this way already, more or less. If you want to address all of the vulnerabilities that have arisen, you need full isolation. As mentioned in a sibling comment, you can't share any memory or caches etc. Each "security core" would have to be fully isolated from every other core, including every other security core. And you also need to segregate out "sensitive" code and data from non-sensitive code and data. When all is said and done, I don't see how we're talking about anything less than scrapping all existing ISAs, and perhaps even von Neumann architecture itself. I'm sure there are some middle-ground solutions between where we are today and this extreme, but I think they're going to look more like TPM/Secure Enclave than special cores, and they're only going to partially address the vulnerabilities.
monocasa 1y ago
It depends on the kernel architecture. 4G/4G kernels weren't the most common thing, but also weren't exactly rare in the grand scheme of things. PowerPC macOS (and x86 in macOS before they officially released Intel based mac hardware) were 4G/4G for example. The way that works under x86 is that you just reserve a couple kernel pages mapped into both address spaces to do the page table swap on interrupts and syscalls. A little expensive, but less than you'd think, and having the kernel and user space not fight for virtual address space provided its own efficiencies to partially make up the difference. We've been moving back to that anyway with Kernel Page Table isolation for spectre mitigations. And 3-1 wasn't really experimental. It was essentially always that way under Linux, and had been supported under Windows since the late 90s.
- kbolino 1y ago
  Yeah, "experimental" may not be the right word, but actually getting to use the 3-1 split required all of the following: at least 3GB of physical RAM (obviously), the O/S booted with /3GB flag, and the application in question linked with /LARGEADDRESSAWARE flag (and not mishandling the high bit of a pointer). Many video games towards the end of the 32-bit era were built this way tbf, though they still generally do better on 64-bit Windows/Wine anyway.

vlovich123 1y ago

I believe that's an accident of the evolutionary path chosen with syscalls. If we'd instead gone with a ring buffer approach to make requests, then you'd never need to partition the memory address space; the kernel has its memory and userspace has its and you don't need the situation where the kernel is always mapped.

kbolino 1y ago
Hmm. I don't understand how that would work. I think it would be possible for e.g. microkernels to greatly reduce the size of the reservation (though not to eliminate it entirely). However, I can't imagine how you would handle the privilege escalation issue without having at least some system code in the application's virtual address space that's not modifiable by the application.
- vlovich123 1y ago
  I'm not sure how privilege escalation would be an issue since you'd never escalate privilege in the first place (I'm assuming you're talking about CPU ring privileges and not OS privileges). You'd just enqueue into the shared kernel/user space ring buffer your operations and the kernel would pick them up on its side, but you'd never jump between rings. Such a design may require at least one processor dedicated to running the kernel at all times, so it might not work on a single processor architecture. However, single processor architectures might be supportable by having the "kernel process" go to sleep by arming a timer and the timer interrupt is the only one that's specially mapped so it can modify the page table to resume the kernel (for handling all the ring buffers + scheduling). As you note, there's some reserved address space but it's a trivial amount just to be able to resume running the kernel. I don't think it has anything to do with monolithic vs microkernels.
  - kbolino 1y ago
    True, you don't have to go full microkernel just to have messages passed though a buffer. However, if the buffer is shared by all processes, it does need to have some protection. I guess you could assign one buffer per process (which might end up using a lot of physical RAM), and then just crash the process if it corrupts its own buffer. The bigger issue with this approach might be adapting to asynchrony though.
    - vlovich123 1y ago
      It wouldn't be by all processes. One per process just like with io_uring. Not sure how it would end up being all that much physical RAM - you get a lot more memory mapped when you just start a process. Page faults might be another tricky corner case.
  - adgjlsfhk1 1y ago
    I have for a while wondered why we don't have "security core"s that are are really slow, but don't have caches or speculative execution so you can run security critical code without having to worry about CPU timing attacks
    - adwn 1y ago
      I think that would be of little help if the faster, less-secure cores have access to the same memory system as the secure cores. If the JavaScript engine of your browser runs on the fast cores, and you're visiting a malicious website, then vulnerabilities due to speculative execution on the fast cores could still leak information that was written by the slow cores. And you really wouldn't want to run the JS engine on cores without caches and speculative execution, at least not for everyday browsing.
    - kbolino 1y ago
      What problem are you trying to solve? If you just want constant-time operations, you just need dedicated instructions. Maybe they run on dedicated cores, but that's an implementation detail. I think this is a good idea, and it feels like we're heading this way already, more or less. If you want to address all of the vulnerabilities that have arisen, you need full isolation. As mentioned in a sibling comment, you can't share any memory or caches etc. Each "security core" would have to be fully isolated from every other core, including every other security core. And you also need to segregate out "sensitive" code and data from non-sensitive code and data. When all is said and done, I don't see how we're talking about anything less than scrapping all existing ISAs, and perhaps even von Neumann architecture itself. I'm sure there are some middle-ground solutions between where we are today and this extreme, but I think they're going to look more like TPM/Secure Enclave than special cores, and they're only going to partially address the vulnerabilities.

monocasa 1y ago

It depends on the kernel architecture. 4G/4G kernels weren't the most common thing, but also weren't exactly rare in the grand scheme of things. PowerPC macOS (and x86 in macOS before they officially released Intel based mac hardware) were 4G/4G for example. The way that works under x86 is that you just reserve a couple kernel pages mapped into both address spaces to do the page table swap on interrupts and syscalls. A little expensive, but less than you'd think, and having the kernel and user space not fight for virtual address space provided its own efficiencies to partially make up the difference. We've been moving back to that anyway with Kernel Page Table isolation for spectre mitigations. And 3-1 wasn't really experimental. It was essentially always that way under Linux, and had been supported under Windows since the late 90s.

kbolino 1y ago
Yeah, "experimental" may not be the right word, but actually getting to use the 3-1 split required all of the following: at least 3GB of physical RAM (obviously), the O/S booted with /3GB flag, and the application in question linked with /LARGEADDRESSAWARE flag (and not mishandling the high bit of a pointer). Many video games towards the end of the 32-bit era were built this way tbf, though they still generally do better on 64-bit Windows/Wine anyway.