3 ms·
There are multiple problems "kernel bypasses" solves. Generally speaking: (A) prevent kernel from touching data and only operate on metadata (to reduce cache
by majke 5y ago
There are multiple problems "kernel bypasses" solves.
Generally speaking:
(A) prevent kernel from touching data and only operate on metadata (to reduce cache misses/memory bandwidth)
(B) in routing situations prevent application from copying data
(C) busy-polling - save cpu on doing everything in application as opposed to jumping to/from kernel context, for better latency
(D) allow manual buffer management
For example, splice or tcp_mmap() can be used to "receive" data from networking stack into application managed place. splice and MSG_ZEROCOPY can be used to "transmit" data into networking stack. The problem with MSG_ZEROCOPY is that, while buffers belong to application, the API doesn't tell you when it's done sending and it's possible to repurpose the buffers.
This io_uring article seem to indicate this fault has been fixed in io_uring zerocopy API.
These API's are pretty much (A), prevent kernel from touching data, and only look at metadata.
Due to API limitations it's hard or even impossible to do (B) in application. But with better API's we'll get there.
To get (C), you need raw hardware access basically.
But the holly grail is (D): the buffer-reuse problem. Ideally you'd want to submit buffer to networking card (or network stack), get data from NIC directly into there, then get a handle to this data in application, perform routing decision, submit that buffer for transmission in some other NIC, and eventually get a completion notification. At which point the buffer could be submitted for RX again. The userspace zerocopy API's are far away from this ideal scenario.
So no. Zerocopy kernel API's don't immediately solve all problems typical kernel bypass solution aims for. Having staid that, I don't think many users have the problems true kernel bypass solves.
- drewg123 5y agoFor example, splice or tcp_mmap() can be used to "receive" data from networking stack into application managed place. splice and MSG_ZEROCOPY can be used to "transmit" data into networking stack. The problem with MSG_ZEROCOPY is that, while buffers belong to application, the API doesn't tell you when it's done sending and it's possible to repurpose the buffers. This io_uring article seem to indicate this fault has been fixed in io_uring zerocopy API. This is the hard part of a zero-copy API, and why its hard to make a zero-copy API that's a drop-in replacement for sockets and can magically enable naive applications to have zero-copy transmits. I attempted to do zero-copy sockets in FreeBSD ~20+ years ago (https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.34.8745&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.34...). What's not mentioned in the paper is that the change was an overall net-negative, because applications would frequently trigger a COW fault when attempting to write to a buffer owned by the kernel.
- a-dub 5y agoit almost seems like zero copy design is a larger design principle that has to permeate how all user and kernel code is structured in order to really see the benefits. i've been thinking about these sorts of things (kernel/user data barriers and high performance ipc) a lot lately as i've been wondering what a high performance, high security microkernel might look like.