7 ms·
Linux: IO without entering the kernel
- rorhug 8y agoWould someone care to summarise the possible benefits of being able to do polled IO, without entering the kernel?
- snaky 8y agoPerformance, performance and performance.
- joatmon-snoo 8y agoSomeone more familiar with kernel workings than me should clarify, but my understanding is that IO generally happens via a syscall which requires the thread/process in question to context switch between userspace and kernel space, which can be very expensive. By enabling IO polling in userspace, you get to avoid that context switching.
- int0x80 8y agoYes. Instead of using a syscall to get/issue events you use a mapped ring buffer. See https://lwn.net/Articles/743714/ https://lwn.net/Articles/743714/
- geofft 8y agoThe motivating benefit is performance, but a side one the author mentioned on Twitter https://twitter.com/axboe/status/1073320502532263936 https://twitter.com/axboe/status/1073320502532263936 is sidestepping Meltdown and similar vulnerabilities from having the kernel and the OS in the same address space (even though they're separated by a privilege boundary). In a scheme like this, you can theoretically dedicate one core to the application and a separate one to the kernel, and minimize speculation, cache sharing, etc. between the two. The application and the kernel share a portion of memory, so the kernel doesn't ever run on the application's CPU. This is questionably practical for a general-purpose machine, but for a server system used entirely as a hypervisor, or web server, or file aerver, or something, it might fit really well.
- cyphar 8y agoBut wouldn't having the kernel pinned to a different core hurt performance due to NUMA, or through having to do lots of cross-calls?
- geofft 8y agoDepends on the use case; keep in mind that syscalls are slow, too. If you have an application that does significant computation on lots of data (think a scientific calculation/simulation), having another core on the same socket read ahead from disk to RAM might be much more efficient than pausing computation to read synchronously. Or if you're a file server that is just passing things back to the kernel's network layer, you might not even need to see the contents of RAM yourself.
- farazbabar 8y agoIn traditional I/O, a hardware interrupt is triggered whenever data arrives at hardware boundary and the interrupt can get serviced by any core that is available to the scheduler. One can imagine how much overhead is involved in context switching whatever that core was doing before, setting up the registers, moving data and then relinquishing the core back to OS - in this model, dedicated cores serve I/O in a memory mapped ring buffer like data structure sized to your application needs. There is no allocation/deallocation overhead, no management beyond moving a pointer and no context switching. If you can spare the cores, this can significantly improve performance. In one use-case, I was able to quadruple the performance on a 32 core xeon by installing 4 10gbps ethernet cards and dedicating the first eight cores to I/O (2 per interface). This is all about latency but with proper care, it also improves throughput.
- barbecue_sauce 8y agoDo you have to write your own software to do this, or can it be accomplished through OS configuration?
- deleted 8y ago[deleted]
- franblas 8y agoIs it dangerous to perform some IO bypassing the kernel ? I suppose not regarding this commit, but the kernel isn't there to check calls of the application ?
- int0x80 8y agoThe kernel handles the IO. You just don't need to enter the kernel (syscall) to issue it, it is done via a kernel<->user memory mapped ring buffer. Roughly, I don't really know the details well. See https://lwn.net/Articles/743714/ https://lwn.net/Articles/743714/
- k__ 8y agoSo you don't "ask" the kernel, but pump your data inside a special memory space where the kernel will perform IO with it?
- tinus_hn 8y agoI think the kernel provides results in a special memory space where you can read it without using system calls. Performing operations in the kernel on a memory area writeable by user space sounds really dangerous.
- int0x80 8y agoIt is possible. As you say, not completely problem free and indeed has raised some discussions regarding security implications. AFAICT documentation is scarce (non-existent). Here is a previous patch prepping for this one, with sample code to both submit/reap events: https://patchwork.kernel.org/patch/10712819/ https://patchwork.kernel.org/patch/10712819/
- lkurusa 8y agoThe idea of avoiding context switches by using memory regions to communicate with the kernel reminds me of FlexSC[0], which was a way to have "async" system calls and "flexible scheduling" of them. [0]: https://www.usenix.org/legacy/events/osdi10/tech/full_papers/Soares.pdf https://www.usenix.org/legacy/events/osdi10/tech/full_papers...
- geofft 8y agoI've never totally followed why AIO is its own set of interfaces instead of a generic async system call mechanism based on something like this. Write a bunch of requests to a page in the form of the registers that would make up the syscal, then wait for one to be completed using a futex or something (where futex(2) remains an actual system call).
- naasking 8y agoYou just reinvented async-await/continuations. It's a great idea.
- kbwt 8y agoSo you need to dedicate two cores, making them unavailable for general-purpose tasks in order to reduce the syscall overhead for a single application?
- jpgvm 8y agoIf you are coding your application against libaio chances are the entire point of the system is to run that application. High performance software generally does divide the system up so that certain cores serve certain roles. i.e IRQ handling etc. This is often done to exploit NUMA architecture, ensuring that memory accesses only happen against the local memory controllers and locally attached PCIE where possible. If you are going to this point you are rarely if ever concerned with sharing the system with anyone else.
- reacharavindh 8y agoI don't know if it is the case, but, I would happily trade off two cores to get better IO in most cases in our nodes servicing IO bound jobs. Otherwise, those CPU cores are just idling anyway because of IO.. With the current growth of CPU leaning towards more and more cores with ever smaller improvements in single core performance, dedicating spare cores for such perm improvements is not at all a bad idea.
- kbwt 8y agoThe only thing this really improves is latency, because you can always batch more I/O into a single syscall.
- beagle3 8y agoYou can batch 1000 writes to the same socket into a single syscall, but you can’t batch writing to 1000 different sockets into one syscall. (And it is actually a common occurrence to write the same message to 100 sockets at the same time - e.g. in an IRC server with 100 people in A room)
- 8y ago
- jbverschoor 8y agoWhat ever happened to singularity, where everything ran in the same space?
- geofft 8y agoMorphed into Midori, which never made it past being a research project. http://joeduffyblog.com/2015/11/03/blogging-about-midori/ http://joeduffyblog.com/2015/11/03/blogging-about-midori/ has some posts about other things they explored in Midori.
- gpderetta 8y agoDoes spectre completely kill the hope of single address space OSes ever seeing the light of day?
- nickpsecurity 8y agoJX OS took that approach for most of its protection with the code being open source: http://www4.cs.fau.de/Projects/JX/ http://www4.cs.fau.de/Projects/JX/
- gbrown_ 8y agoThe following may also be of interest to readers. https://lore.kernel.org/linux-block/20181117235317.7366-1-axboe@kernel.dk/ https://lore.kernel.org/linux-block/20181117235317.7366-1-ax...
- d33 8y agoI wonder if this kind of approach could be generalized to all syscalls - pushing them asynchronously to shared memory, then receiving output in arbitrary order once kernel core takes care of it. Is this feasible? Does it make sense to expand this beyond IO? My reasoning is that we already usually have more cores than we need, why not dedicate some specifically for kernel stuff?
- hornetblack 8y agoThere's a concept called exception-less syscalls, which might be similar to what your thinking.
- d33 8y agoMy idea is to avoid context switching like in VDSO, but with a generalized solution that would allow this sort of behavior for all system calls. Could you point me to any write-ups about exception-less syscalls?
- anonymousDan 8y agoSee the FlexSC paper at the bottom here: https://www.usenix.org/conference/osdi10/flexsc-flexible-system-call-scheduling-exception-less-system-calls https://www.usenix.org/conference/osdi10/flexsc-flexible-sys... or more recently SCONE in the context of Intel SGX: https://www.usenix.org/conference/osdi16/technical-sessions/presentation/arnautov https://www.usenix.org/conference/osdi16/technical-sessions/...
- int0x80 8y agoThere is mmap for read/write (some of the most used syscalls). There is also the vdso page [1] for things like gettimeofday etc. [1] http://man7.org/linux/man-pages/man7/vdso.7.html http://man7.org/linux/man-pages/man7/vdso.7.html
- blattimwind 8y agoThat's almost literally IOCPs.
- glenrivard 8y agoI believe this is similar to what we will see with Zircon as a default. Why I believe multi core performance with Zircon will exceed the same with Linux. The big question is single core? You do not have the benefit. But I do think Google will do their own CPU optimized for Zircon. Do hope they use RISC-V for the ISA. They did with the PVC.
- spatulon 8y agoBTW, you appear to be shadowbanned. I vouched for your post to un-kill it.
- glenrivard 8y agoThanks! Have no idea why banned and was never given any explanation why. But it is what it is. Guess I could just make a new account.
- nkurz 8y agoI don't know if the explanation is true, but from the outside it looks like you were given a very clear explanation: sctb 67 days ago [-] We've banned this latest shill account. Please do not create any more to abuse Hacker News with a Google agenda. https://news.ycombinator.com/item?id=18179838 https://news.ycombinator.com/item?id=18179838.
- baybal2 8y agoA little bit of a misnomer. You have to mention that IO and driver interaction is still being done in kernel. Just intermediary buffers are remaining MMAPed in both kernel and userspace. That's a step forward from syscalls and excessively frequent context switching, but it is not a real case of userspace driving hardware directly. The later still holds a lot of promise, but it will require hardware to be more smart on its own, and be able to support access concurrency. Something like that is already being done by hardware supporting virtualisable I/O
- FullyFunctional 8y agoThis is what DSSD did since ~ 2014; drive NVMe directly to the D5 from user app, never incurring a context switch (in fact the D5 was mapping into memory on all the PCIe connected clients). Latency was astonishingly low, dominated by PCIe switches for writes and NAND reads for, eh, reads. EDIT: typo
- termie 8y agoThis is promising, but for dedicated high-performance I/O I am much more interested in how the SPDK is progressing. Avoiding syscalls is great, but without any benchmarks I wonder about the benefits over mmap and vectorizing. You still have to contend with locking, interrupts, copying, and ring 0 abstraction impedence. [1] https://spdk.io https://spdk.io
- jandrewrogers 8y agoSPDK follows the same pattern as DPDK (for networking) but it has much less of a practical use case in my experience. You can easily outperform mmap() etc using io_submit() interface that already exists, saturating the capabilities of fast storage hardware, and the overhead of io_submit() is nominal if you are using it well. SPDK won't meaningfully increase your I/O throughput and the API is significantly more difficult to use. I've played around with SPDK in database storage engines and I find it to be inferior for the purpose compared to io_submit() in practical implementations. The number of IOPS required to make SPDK worth considering would be indicative of a more fundamental design flaw in your storage architecture, at least for the foreseeable future. DPDK, by contrast, can dramatically improve networking throughput and performance. While also not the most friendly API to use, it undeniably improves CPU efficiency for packet processing and without an obviously better alternative.
- devwastaken 8y agoDo you have a high performing io_submit() example?
- jandrewrogers 8y agoA high-performance io_submit()/O_DIRECT implementation looks similar to a more explicit reimplementation of mmap() and friends, so the performance does not come from the API per se but what it enables. The primary difference is that manipulating/measuring the fine-grained runtime internal state of mmap() is difficult/expensive, whereas with io_submit() you have almost perfect visibility into and control of the entire internal state of your ersatz mmap(). With io_submit() you never block on page faults or write backs, since your scheduler explicitly controls when they happen and knows when they complete. Admittedly there is a large implementation gap between the io_submit() APIs and a high-performing mmap() replacement. Throughput in database-y software is largely driven by the effectiveness of the scheduling -- doing exactly the right thing at exactly the right time. The fine-grained awareness and control of your instantaneous disk I/O and cache state makes it possible to build efficient schedules with io_submit(). With mmap() you are never entirely sure what is going under the hood and Linux will often decide to do suboptimal things at suboptimal times, or choose to ignore you -- most control of mmap() behavior is advisory only.
- ksec 8y agoIs there something similar available on FreeBSD?
- en4bz 8y agoSo this is basically packet MMAP but for disks?