5 ms·
If the problem is slow switching between user space and kernel space, why not have a syscall that allows you to batch process multiple syscalls? It might work
by correct_horse 7y ago
If the problem is slow switching between user space and kernel space, why not have a syscall that allows you to batch process multiple syscalls?
It might work like this: user space allocates a buffer, fills it with syscall numbers and arguments, then passes it to the kernel. The kernel executes each syscall in series, placing the return values in the array originally containing arguments. This syscall would not return to user space until all syscalls in the buffer are processed.
Does this idea have any potential?
- nominatronic 7y agoThis is basically the idea of io_uring [0], although that mechanism is specific to certain I/O operations rather than being a generic queuing system for syscalls. [0] https://lwn.net/Articles/776703/ https://lwn.net/Articles/776703/
- wahern 7y agoBefore io_uring there were many failed proposals for a general purpose, asynchronous interface to wrap blocking syscalls. One of these, "syslets", was literally a mechanism for batching arbitrary syscalls: https://lwn.net/Articles/221887/ https://lwn.net/Articles/221887/ > Syslets are small, simple, lightweight programs (consisting of system-calls, 'atoms') that the kernel can execute autonomously (and, not the least, asynchronously), without having to exit back into user-space. .... > Syslets consist of 'syslet atoms', where each atom represents a single system-call. These atoms can be chained to each other: serially, in branches or in loops.
- correct_horse 7y agoThanks for sharing. It's encouraging that something at least resembling what I came up with was considered.
- anonymousDan 7y agoAny thoughts on why they failed?
- wahern 7y agoI don't think the kernel complexity and maintenance burden was deemed worth the effort. But now 1) Spectre has made syscalls more expensive and 2) Linux has destroyed all the competition and so there's alot more pressure to add features for what effectively would only ever be used by niche use cases--you don't add io_uring to benefit the Node.js crowd, you add it for Amazon and Netflix and other consumers that have have no other avenues for optimization. To be fair, io_uring seems to be a pretty sane design. But, for example, for many years anything with a shared ring buffer between userspace and the kernel was suspect. Now there are several such interfaces. Overall tolerance has grown, especially on the part of Linus. The kernel is much larger, and Linus seems to have become much more deferential to subsystem maintainers as his ability to understand and opine on code has slipped beyond his grasp.
- eternalban 7y agoThe user api: https://people.redhat.com/mingo/syslet-patches/patches/async-user-api-defs.patch https://people.redhat.com/mingo/syslet-patches/patches/async... User space memory mapping (naturally left to the user) seems to be the error prone bit here. search for "sys_umem" here: https://people.redhat.com/mingo/syslet-patches/patches/async-core-code.patch https://people.redhat.com/mingo/syslet-patches/patches/async... I understand that this is low level stuff for experienced developers, but this approach basically asks the programmer to play linker. Entirely doable but I wonder what it looks like at scale (LOC).
- JoshTriplett 7y agoLong-term, io_uring will almost certainly going to provide more operations than just I/O.
- the8472 7y agoBatched virtual memory updates (mmap/munmap/madvise) could be a boon for allocators and garbage collectors since the cost of TLB shootdowns could be amortized. Similarly if all the syscalls done between fork and exec could be squeezed into a single uring chain without any intervening userspace code then the kernel could skip some costly virtual memory operations, bringing us closer to posix_spawn behavior.
- JoshTriplett 7y ago> Batched virtual memory updates (mmap/munmap/madvise) could be a boon for allocators and garbage collectors since the cost of TLB shootdowns could be amortized. And ideally also the cost of mmap_sem (the lock for an address space's memory map), which can be one of the most contended locks in a workload. > Similarly if all the syscalls done between fork and exec could be squeezed into a single uring chain without any intervening userspace code then the kernel could skip some costly virtual memory operations, bringing us closer to posix_spawn behavior. I'd love to see a way of doing this with io_uring.
- koolba 7y agoThe problem is that the calls are related. The first gives you the file handle, the second reads it, the third closes it. You need the handle in advance to queue the second and third. This idea reminds me of pipelined Redis commands. It does work to boost total throughout, but it’s significantly more complicated to use in practice.
- correct_horse 7y ago> You need the handle in advance to queue the second and third whoops. I probably should have thought of that.
- jzwinck 7y agoImagine a method like this: 1. Enqueue A = openat(...) 2. Enqueue B = read(A, buf, sz) 3. Enqueue close(A) 4. Commit, return B to userspace A and B are placeholders, and A is never known to userspace, it is simply chained within the enqueued commands.
- danobi 7y agoWhat if openat() or read() fails? More generally, what if there's a chain of syscalls and you _want_ one of them to fail?
- JoshTriplett 7y agoWith the "specific fd" approach, if openat fails, then read and close will harmlessly fail with EBADF ("Bad file descriptor"). Once you process the failure of openat, you'll just ignore the failure of read and close.
- yxhuvud 7y agoWhat will happen though, if the fd is already used by some other area of userspace? Like what if some lib you link to also happen to hardcore the use of the same fd. Sounds like unfunny problems to debug. In the case of uring I suppose it is possible to link the calls to fail at the first failure.