5 ms·
Given the multi core, NUMA and Spectre/meltdown reality we’re living in, and the clear benefits of the io_uring approach, why not just have a dedicated core(s)
by vlovich123 5y ago
Given the multi core, NUMA and Spectre/meltdown reality we’re living in, and the clear benefits of the io_uring approach, why not just have a dedicated core(s) to handle “interrupts” which are nothing more than entries in a shared memory table?
- electricshampo1 5y agoThis is essentially the approach taken by https://www.dpdk.org/ https://www.dpdk.org/ (network) and https://spdk.io/ https://spdk.io/ (storage) Anything trying to squeeze perf doing IO intensive work should switch to this model (context permitting of course).
- eklitzke 5y agoThis approach works for I/O devices (and for things like network cards the kernel will typically poll them anyway), but I/O isn't the only thing that generates interrupts. For instance, a processor fault (e.g. divide by zero) should be handled immediately and synchronously since the CPU core generating the fault can't do any useful work until the fault is handled.
- vlovich123 5y agoIs that actually true? Wouldn’t this imply you could launch a DOS attack against cloud providers just generating divisions by zero?
- bonzini 5y agoYou would only attack yourself. The CPU time you're paying for would be spent processing division by zero exceptions.
- vlovich123 5y agoThen the CPU isn’t stopping and is moving on doing other work. Meaning the divide by zero could be processed by a background CPU and doesn’t require immediate handling. Same for page faults.
- bonzini 5y agoYour instance can still be preempted by the hypervisor.
- fulafel 5y agoThe hypervisor is just some software that runs on the same CPU. It wouldn't help if the assertion that the CPU can't continue was true.
- bonzini 5y agoThe CPU can't continue running user code because it jumps to the exception handler. But it can get to the hypervisor at any instruction boundary. (I write a hypervisor for a living :)).
- bananabreakfast 5y agoIncorrect. The "CPU" is stopping and handling the fault. There is no background CPU from your perspective. In a cloud provider you are always in a virtual environment and using a vCPU which is constantly being preempted by the hypervisor. You cannot DOS a hypervisor just by bogging down your virtualized kernel.
- bonzini 5y agoThere are a couple x86 corner cases where you can do so, by causing an endless stream of exceptions. But they are not going to happen in a hot path, so the hypervisors force a vmexit on those exceptions and avoid that the processor freezes forever.
- zaptheimpaler 5y agoThe divide by zero in this case is handled by a virtual CPU implemented in the hypervisor, not the actual physical CPU on the server.
- imtringued 5y agoA site that renders HTML is more computationally expensive than handling a site that does nothing but division by zero.
- fulafel 5y agoThere are mechanisms that can intercede, and there are often more than one cpu thread per core.
- justincormack 5y agoFaults for divide by zero are a terrible legacy thing, Arm etc do not do this, you test for zero if you want to, otherwise you get a value.
- eklitzke 5y agoThat was just an example, there are many other things the CPU can do that will generate a fault (for example, trying to execute an illegal instruction).
- DblPlusUngood 5y agoA better example: a page fault for a non-present page.
- vlovich123 5y agoCould be handled by having the CPU switch to a different process while the kernel CPU faults the data in.
- ben509 5y agoThe CPU doesn't know what processes are, that's handled by the OS. So there still needs to be a fault.
- vlovich123 5y agoYou’re thinking about computer architecture as designed today. There’s no reason there isn’t a common data structure defined that the CPU can use to select a backup process, much how it uses page table data structures in main memory to resolve TLB misses.
- gpderetta 5y agoX86 had the concept of harware assisted context switching. Yet another unsued feature dropped in 64bit mode.
- bonzini 5y agoOr interprocessor interrupts for flushing the TLB, terminating the scheduling quantum, or anything else.
- deleted 5y ago[deleted]
- devit 5y agoThere must be a way to alter a core's instruction pointer from another core or from hardware to support killing processes running untrusted machine code, and to support pre-emptive multithreading without needing to have compilers add a check for preemption on all backward branches and calls. These features are very worth the hassle of providing this capability (which is known as "IPI"s), and once you have that hardware interrupts become pretty much free to support by using the same capability and the OS/user can then decide whether to dedicate a core, to affinity them to a core, to load balance them among all cores or to disable the interrupts and poll instead.
- vlovich123 5y agoI was thinking rather than mucking with instruction pointers you would just send a message back to the other CPU saying “pause & switch to context X”. Technically an interrupt but one that can be handled internally within the CPU.
- ori_b 5y agoWhat do you think "switch to context X" does, if not muck with the instruction pointer? Where do you think mucking with the instruction pointer is handled, if not within the CPU?
- knz42 5y agoThis is the approach proposed in http://doi.org/10.1109/TPDS.2015.2492542 http://doi.org/10.1109/TPDS.2015.2492542 (preprint: https://science.raphael.poss.name/pub/poss.15.tpds.pdf https://science.raphael.poss.name/pub/poss.15.tpds.pdf )
- DSingularity 5y agoAnd old systems like Corey OS.
- bostonsre 5y agoWould guess it would be complicated and difficult to update the kernel to support something like that. Not sure Linus would entertain PRs for custom boards that do something like that. Would think it would need to be an industry wide push for that. But just speculation..
- vlovich123 5y agoWe’re talking about the CPU architecture here, not custom one-off ARM boards. Think x86 or ARM not a Qualcomm SoC. And yes of course. Linus’ opinion would be needed.
- bostonsre 5y agoYea, that could probably be an even bigger impediment to trying to implement a separate interrupt processing unit I guess. My grasp of this area is tenuous, didn't even contemplate that. Would a new physical setup be required for something like this? Or would it all be in the cpu architecture?