5 ms·
Even ignoring CPU die size cost, there's another cost to legacy features: some of those features have register state, and contexts switching it is expensive. S
by amluto 9y ago
Even ignoring CPU die size cost, there's another cost to legacy features: some of those features have register state, and contexts switching it is expensive. Simpler architectures can do crazy things like switching between threads in mere tens of cycles. X86: not so much. The OS needs to deal with DS, ES, FS, and GS, and the microcode needs to deal with CS, SS, and FLAGS. The segment registers are almost useless these days, but saving and restoring them is very slow.
Even x86's model of handling CPL (privilege level) isn't ideal. I've tried to push RISC-V toward a model where user and kernel code have entirely separate address spaces, which would let privilege handling move to the front end and potentially allow pipelining across privilege changes. Amusingly, x86's privilege mechanism is complicated enough that Intel implemented it wrong in a few CPUs with MPX, resulting in a bizarre erratum.
I would love to see x86 add a "clean mode" in which a lot of legacy features are inaccessible to user code. Then kernels could try to run programs in clean mode and, if they don't fault, context switching between clean mode threads would be much faster.
- frozenport 9y agoAlso AVX512! And what the heck are we going to do when we need to save the AVX1024 register state! And we really going to push 1kb to memory per context switch? (I think this is somewhat solved by trapping when the registers are first invoked, and kernel drivers that use enhanced instruction sets actually need to pass flags before using them)
- jandrewrogers 9y agoThis highlights one of the advantages of context-switching in user space. The OS context switch can make few assumptions about the registers that actually need to be preserved once a register set has been used. Many user space context libraries, on the other hand, only preserve the half dozen basic registers per the ABI by default (very fast) and let the programmer optionally preserve the other register sets. In the vast majority of cases, you never need to preserve any additional state at the logical context switch points even when you are using those additional registers in your application.
- frozenport 9y agoAlternatively juat grep the function source and ban self modifying code. Or run self modifying code in a "fuck you" performance sandbox.
- MichaelMoser123 9y agoRight. Boosts set_fcontext even allows to omit the floating point registers (saves a lot of time if the application doesn't do fp)
- amluto 9y agoThis doesn't bother me as much. X86 could add lightweight instructions to clear large parts of the AVX state (a la VZEROALL) along with their associated "in use" flags, and compilers could be taught to emit these instructions whenever they're done using the big registers. Then context switches could mostly avoid touching them.
- flamedoge 9y ago??? it's way more than that. at least 32kb if they keep # of vector registers same.
- legulere 9y agoNowadays CS, SS, DS, and ES are enforced to be 0 in 64 bit mode, leaving just FS and GS which are used for e.g. thread local storage
- amluto 9y agoNot true. The CS, SS, DS, and ES bases are ignored in 64-bit mode, but those registers still exist, still carry their full, extraordinarily complicated values, and are very much observable. Even on AMD Zen, for example, SS handling is screwed up enough that every context switch has added overhead to work around a SYSRET glitch that would otherwise risk executing user code with a messed up bit in the "hidden" part of SS. AMD should have killed these registers entirely in 64-bit user mode, but they didn't.
- yuhong 9y agoMy understanding is that AMD probably originally designed SYSRET assuming that SS contained the correct values already (except SS.DPL of course, which I guess is where the stuff where they set it to RPL comes from). When they designed x86-64 however, they decided to set SS to null selectors in interrupts. Of course, these values are normally ignored in 64-bit long mode. Long mode is also when they fixed SYSCALL/SYSRET by adding an option to mask EFLAGS (and store the old value in R11): http://web.archive.org/web/20120813101603/http://www.x86-64.org:80/pipermail/discuss/2000-October/001019.html http://web.archive.org/web/20120813101603/http://www.x86-64....
- yuhong 9y agoBTW, SYSENTER/SYSEXIT was designed by Andy Glew, which mentioned in https://groups.google.com/d/msg/comp.arch/Bf-4VflljF4/4hNvyrHN8pAJ https://groups.google.com/d/msg/comp.arch/Bf-4VflljF4/4hNvyr... that "(I wanted to avoid changing SS:ESP, so that it could be just a single register write if CS:EIP were co-renamed, but microcode thought that would violate invariants about SS RPL they wanted to maintain." I wonder if part of the problem is that AMD was still new to x86 CPU design back when SYSCALL/SYSRET was developed as part of the K6.
- terryf 9y agoInteresting. I thought they would do register renaming for thread switching as they do for speculative execution. Is this not the case?
- amluto 9y agoI doubt a segment register is easily renamable. It implicitly affects most subsequent operations, and it's not really just a number. (You can find the full data structure documented in the virtualization section of the Intel and AMD docs because virtualization extensions let you read the registers in their entirety [1].). Also, the instructions that write to these regs have very complicated microcode. [1] The AMD interactions between CPL, CS, and SS are so messy that there's still an unresolved crash when KVM runs an older Linux guest and exits all the way out to QEMU to handle, say, a virtio interrupt at the wrong time. QEMU tries and fails to read the state and write it back fully.
- kllrnohj 9y ago> The segment registers are almost useless these days, but saving and restoring them is very slow. Those are only saved/restored when switching between processes not between threads. Context switching when dealing with threads is typically done in software, not in hardware, precisely so that registers that are not used (like the segment registers) don't get needlessly saved & restored. Saving/restoring registers isn't even the expensive part of a context switch anyway. Switching address spaces, cache flushes/misses, etc... are all more expensive than saving/restoring registers.
- bhouston 9y ago> Those are only saved/restored when switching between processes not between threads. Context switching when dealing with threads is typically done in software, not in hardware, precisely so that registers that are not used (like the segment registers) don't get needlessly saved & restored. You are incorrect. Context switching is done for threads in HW. I think you are thinking of cooperative multi-threading (like pre-Windows 95, or nginx or node.js or V8) instead of true context-switching threading.
- dom0 9y ago> Context switching is done for threads in HW. Uhm, x86-64 can't switch between tasks/contexts in hardware.
- kllrnohj 9y agoAll you need for preemptive multithreading is IRQ#0 to generate an interrupt. That's the entirety of it. Preemptive multiprocessing then needs paging and such to make isolated memory a thing as well, but still doesn't need HW context switching.
- xorblurb 9y agoMost registers including x86 segment registers need to be saved during context switches, regardless of its between threads or processes. E.g segment registers used for TLS. There are maybe optimisation for flat segments. > Context switching when dealing with threads is typically done in software, not in hardware I'm not sure any OS exists that does register saving in HW for switches between processes and in SW for switches between threads. Its especially doubtful for Linux, given it does not really have a very strong concept of what is a process vs. what is a thread.
- Animats 9y agoEven x86's model of handling CPL (privilege level) isn't ideal. Because each was retrofitted, each protection ring in x86, from -1 to 3, works differently. When IBM originally invented virtualization (IBM 360/67, late 1960s), they got it right; all the layers work the same, and you can run VM on VM on VM until you run out of memory. That's been continued into today's IBM mainframes.
- bogomipz 9y agoI'm not following the connection, IBM used more than two protections and thats why we have the two unused rings in x86? Can you elaborate?
- amluto 9y agoI'm not talking about the fact that the privilege level is two bits instead of one bit. What I mean is: the privilege level is more or less duplicated in quite a few different pieces of CPU state. There's CPL, CS.RPL, CS.DPL, SS.RPL, and SS.DPL, not to mention CR0.PE. The one that matters is CPL, but CPL itself isn't an architecturally visible register except (I think) on AMD systems with SVM -- it's merely a concept. CS.RPL, CS.DPL, SS.RPL, and SS.DPL all usually match CPL, but I think that all four equalities have nasty corner case exceptions. This results in a bunch of overhead when you change privilege: the CPU needs to update CPL, CS.RPL, CS.DPL, SS.RPL, and SS.DPL. This means that you need to change two of these nasty large legacy registers, the values of which barely matter for practical purposes, every time you enter or exit user mode. Saying there's a ring -1 is a red herring here. There's SMM, which is a mode of execution that's more privileged than ring 0, but it doesn't have a number -1 associated with it, nor is it part of CPL. Similarly, ring 0 is split into root and non-root when virtualization is in play, but that's also a bit beside the point. [1] Check out the Linux commit message for ae9fedc793c4d98aa9bb298585b2b9246096ce65 if you want to see a bit of how nasty this is.
- bogomipz 9y ago>"and the microcode needs to deal with CS, SS, and FLAGS. The segment registers are almost useless these days, but saving and restoring them is very slow." Can you elaborate on whats special in terms of CS, SS and the FLAGs registers in terms of what the microcode must do? >"Even x86's model of handling CPL (privilege level) isn't ideal. I've tried to push RISC-V toward a model where user and kernel code have entirely separate address spaces, which would let privilege handling move to the front end and potentially allow pipelining across privilege changes" I thought that mapping the kernel into the top of of a userland process address space was supposed to be kind of elegant as it prevents a TLB flush. What is does "move to the front end" in your proposed alternative to this arrangement? This sounds interesting. Might you have any links regarding this?
- amluto 9y ago> Can you elaborate on whats special in terms of CS, SS and the FLAGs registers in terms of what the microcode must do? For SYSCALL, SYSRET, SYSENTER, and SYSEXIT, microcode needs to change the values of the registers (including the hidden parts). For entries through the IDT (interrupts, faults, and traps) and IRET, the microcode needs to go through a whole bunch of complicated logic including looking up the requested CS and SS selectors in the GDT or LDT. > I thought that mapping the kernel into the top of of a userland process address space was supposed to be kind of elegant as it prevents a TLB flush. What is does "move to the front end" in your proposed alternative to this arrangement? This sounds interesting. Might you have any links regarding this? Once upon a time, the x86 MMU, and probably most other MMUs, was quite simple. There was a TLB, which is essentially a cache that maps (VA) -> (PA, access rights, cache mode). A memory access looks up the virtual address (VA) in the TLB, then checks the access rights and accesses the physical address. In the event of the TLB miss, the CPU falls back to reading the page tables and fills the TLB. When you change to a different process with different page tables, you flush the TLB. You can either map the whole kernel into every process's address space of you can map a small trampoline that switches to the real kernel's address space and flushes the TLB every time you enter the kernel. From (IIRC) 486 onward, x86 would let you tag a TLB entry as "global", meaning it would survive a normal TLB flush and would only be cleared on a special type of flush. Linux uses global entries for kernel pages so that only user entries get flushed on processes switches. Modern CPUs have a somewhat different arrangement. The TLB maps (VA, address space ID) -> (PA, access rights, cache mode) [1]. On a TLB miss, the CPU will look up the VA in the current process's page tables and add a TLB entry with the appropriate address space ID. On x86, the address space ID is (VPID, PCID) [2]. This means that you can sometimes switch between processes without a TLB flush -- you can change the page tables and the address space ID and leave the TLB intact. I would like to see CPUs arrange it just a bit differently: have a separate address space, complete with a separate tag bit, for user mode and kernel mode. Then user accesses effectively look up (VA, "user", ASID) and kernel accesses effectively look (VA, "kernel", ASID). This would add almost no overhead. As an added benefit, when the front end decodes an instruction, it would know the whole TLB key (i.e. VA, user/kernel, and address space ID) and could decode user and kernel code differently. This means that kernel-mode instructions could be rejected in the front end. More importantly, if you imagine the user/kernel bit as being an extension of the VA, a transition between user and kernel mode is functionally the same thing as an indirect jump and could use all the optimizations, including pipelining, that indirect jumps use. [1] It might be organized internally as (VA) -> (address space ID, PA, access rights, cache mode), but it doesn't really matter. [2] VPID identifies the current VM guest, if any, and PCID is used at the discretion of the kernel. Linux may soon use PCID to allow a few processes to coexist in the TLB at once. grsecurity uses it for a totally different purpose.