21 ms·

Exactly my point, the kernel shouldn’t have to be bypassed. It should just be fast.

ignoramous 8y ago
(not an expert, just someone keeping abreast with development in the networking world) There are inherent problems with relying on the OS in an elastic environment. Kernel's TCP/IP stack could be made to expose very specific APIs to help with NetworkFunctionVirtualization (NFV) and Software-defined Networking (SDN), but the advantages are not clear: 1. Vendors to rely on Kernel devs to do the their biding for their NFV/SDN deployments, and wrestle with various other priorities Kernel might already have. 2. Build/deploy the updated Kernel fleet wide, with those changes (which now include other Kernel changes vendor may or may not want). 3. Cater to strict Kernel community guidelines and expect to be driven by the established processes governed by designated gatekeepers, costing vendors time and speed to market. With a userspace solution, esp since it allows vendors to be performant by simply interfacing with NICs, the vendors have a measure of control on their own destiny: 1. Deploy at will, at pace. 2. Ability to do custom packet/flow (virtualization), on-the-fly (SDN) even, without requiring any assistance from any other part of the system not in their control. 3. Reduce external factors that might influence their timelines or priorities. The generic problem, however, with the Kernel's TCP/IP stack needing improvement is orthogonal to this. That effort will have little to no bearing on the direction the Networking industry is moving, imo. --- Google Maglev (network load balancer): https://ai.google/research/pubs/pub44824 https://ai.google/research/pubs/pub44824 Dropbox MagicPocket (bypassing OS' VirtualFileSystem for storage at scale): https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pocket/ https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pock... Facebook Haystack (an example of problems working with VFS at scale and a solution to it): https://code.fb.com/core-data/needle-in-a-haystack-efficient-storage-of-billions-of-photos/ https://code.fb.com/core-data/needle-in-a-haystack-efficient... NFV: https://www.electronics-notes.com/articles/connectivity/nfv-network-functions-virtualisation/what-is-nfv-basics.php https://www.electronics-notes.com/articles/connectivity/nfv-... VNF: https://www.electronics-notes.com/articles/connectivity/nfv-network-functions-virtualisation/virtualized-network-functions-vnf.php https://www.electronics-notes.com/articles/connectivity/nfv-...
convolvatron 8y ago
there are a couple pretty core assumptions that make that hard: read() interfaces expect the kernel to put the data where you asked. its hard but not impossible to coordinate with the NIC to demultiplex the packet and get it into the right place without a copy. almost all of the time you just want to see the data, you dont care that it lands in this particular spot. getting control flow in and out of the kernel is expensive. target address need to be checked. registers often need to be saved. things like the kernel firewall take a certain amount of work to determine the packet status there are lots of mitigations, but at some point you run into a conflict between general purpose multi-process functions and performance. if you ran your application on a dedicated arm next to the nic I bet you'd see a difference. maybe that shouldn't be so hard.
- CoolGuySteve 8y ago
  That still doesn't add up though. System call overhead is something like 500ns to 1usec depending on the call. Copying a large 1500 byte UDP MTU from ram is pessimistically along the lines of 2.5 usec (1500 bytes / 64 byte cacheline * 100ns L3 access). So a simple UDP recv() + copy from network card buffer to userspace should be about 3-4usec. In the real world, it's more like 6 or 7usec the last time I timed it. Admittedly, the unix API could use a refresh. A lot of unix calls like getaddrinfo are perfectly happy to allocate and give you a buffer. A "fast_recv" could easily do the same and just hand you a page from a kernel page pool directly that you then have to free. By avoiding the buffer copying, you could get something in the same ballpark as kernel bypass but without all the bespoke code and frameworks in every application. And in the end, that's kind of the point of having an "Operating System" in the first place.
  - ignoramous 8y ago
    > So a simple UDP recv() + copy from network card buffer to userspace should be about 3-4usec. In the real world, it's more like 6 or 7usec the last time I timed it. Not sure where the Linux kernel lags (may be at scale it kind of tails out?)-- if you read Google's paper on their network load balancer, Maglev, where section 5.2.1 specifically calls out 30% performance degradation without bypass on smaller packet sizes. > By avoiding the buffer copying, you could get something in the same ballpark as kernel bypass but without all the bespoke code and frameworks in every application. May be this article from Cloudflare helps paint a proper picture of why certain kind of applications may need to keep bypassing the kernel forever because their requirements are very specific and not because the kernel can't be made to go faster (ex: a virtual router/switch, or a load-balancer): https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp-stack/ https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp...
TomMarius 8y ago
Merely passing from user to kernel space is time consuming, it is correct to bypass the kernel.
pixl97 8y ago
No. Dont play games with userspace/kernelspace translation. This always ends in security flaws that allow complete take over of the machine.
peterwwillis 8y ago
Everything should just be fast, but it usually isn't. So how are you going to deal with that as a business? Hire kernel hackers and spend expensive time and money to fine-tune one version of a kernel that'll never go upstream? Or pay a vendor for a really fast NIC and integrate it into your app? (If you're Google, you hire the kernel hackers, but if you're trying to make money off a product and ship it next quarter, you buy the NIC)
- CoolGuySteve 8y ago
  Well right now everyone who cares about latency (including me) is spending time integrating some kind of kernel bypass library into their application that's custom and/or proprietary to a single card vendor and often pretty finicky to configure and integrate correctly. It's a massive waste of resources. If the kernel could figure out DRI for graphics then they can figure out universal passthrough networking. All the bypass libraries seem to use the same buffer pool + spinlock slot architecture already. And by your logic, how does anything ever get into the kernel? Nobody should ever contribute anything because "upstream".
  - peterwwillis 8y ago
    I'm just saying that fixing the kernel is unlikely to happen unless it's in someone's business interest, and it's usually not. Hell, I bet there's kernel forks out there where people have already solved this, but it won't go upstream, because lawyers.

(not an expert, just someone keeping abreast with development in the networking world) There are inherent problems with relying on the OS in an elastic environment. Kernel's TCP/IP stack could be made to expose very specific APIs to help with NetworkFunctionVirtualization (NFV) and Software-defined Networking (SDN), but the advantages are not clear: 1. Vendors to rely on Kernel devs to do the their biding for their NFV/SDN deployments, and wrestle with various other priorities Kernel might already have. 2. Build/deploy the updated Kernel fleet wide, with those changes (which now include other Kernel changes vendor may or may not want). 3. Cater to strict Kernel community guidelines and expect to be driven by the established processes governed by designated gatekeepers, costing vendors time and speed to market. With a userspace solution, esp since it allows vendors to be performant by simply interfacing with NICs, the vendors have a measure of control on their own destiny: 1. Deploy at will, at pace. 2. Ability to do custom packet/flow (virtualization), on-the-fly (SDN) even, without requiring any assistance from any other part of the system not in their control. 3. Reduce external factors that might influence their timelines or priorities. The generic problem, however, with the Kernel's TCP/IP stack needing improvement is orthogonal to this. That effort will have little to no bearing on the direction the Networking industry is moving, imo. --- Google Maglev (network load balancer): https://ai.google/research/pubs/pub44824 https://ai.google/research/pubs/pub44824 Dropbox MagicPocket (bypassing OS' VirtualFileSystem for storage at scale): https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pocket/ https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pock... Facebook Haystack (an example of problems working with VFS at scale and a solution to it): https://code.fb.com/core-data/needle-in-a-haystack-efficient-storage-of-billions-of-photos/ https://code.fb.com/core-data/needle-in-a-haystack-efficient... NFV: https://www.electronics-notes.com/articles/connectivity/nfv-network-functions-virtualisation/what-is-nfv-basics.php https://www.electronics-notes.com/articles/connectivity/nfv-... VNF: https://www.electronics-notes.com/articles/connectivity/nfv-network-functions-virtualisation/virtualized-network-functions-vnf.php https://www.electronics-notes.com/articles/connectivity/nfv-...

convolvatron 8y ago

there are a couple pretty core assumptions that make that hard: read() interfaces expect the kernel to put the data where you asked. its hard but not impossible to coordinate with the NIC to demultiplex the packet and get it into the right place without a copy. almost all of the time you just want to see the data, you dont care that it lands in this particular spot. getting control flow in and out of the kernel is expensive. target address need to be checked. registers often need to be saved. things like the kernel firewall take a certain amount of work to determine the packet status there are lots of mitigations, but at some point you run into a conflict between general purpose multi-process functions and performance. if you ran your application on a dedicated arm next to the nic I bet you'd see a difference. maybe that shouldn't be so hard.

CoolGuySteve 8y ago
That still doesn't add up though. System call overhead is something like 500ns to 1usec depending on the call. Copying a large 1500 byte UDP MTU from ram is pessimistically along the lines of 2.5 usec (1500 bytes / 64 byte cacheline * 100ns L3 access). So a simple UDP recv() + copy from network card buffer to userspace should be about 3-4usec. In the real world, it's more like 6 or 7usec the last time I timed it. Admittedly, the unix API could use a refresh. A lot of unix calls like getaddrinfo are perfectly happy to allocate and give you a buffer. A "fast_recv" could easily do the same and just hand you a page from a kernel page pool directly that you then have to free. By avoiding the buffer copying, you could get something in the same ballpark as kernel bypass but without all the bespoke code and frameworks in every application. And in the end, that's kind of the point of having an "Operating System" in the first place.
- ignoramous 8y ago
  > So a simple UDP recv() + copy from network card buffer to userspace should be about 3-4usec. In the real world, it's more like 6 or 7usec the last time I timed it. Not sure where the Linux kernel lags (may be at scale it kind of tails out?)-- if you read Google's paper on their network load balancer, Maglev, where section 5.2.1 specifically calls out 30% performance degradation without bypass on smaller packet sizes. > By avoiding the buffer copying, you could get something in the same ballpark as kernel bypass but without all the bespoke code and frameworks in every application. May be this article from Cloudflare helps paint a proper picture of why certain kind of applications may need to keep bypassing the kernel forever because their requirements are very specific and not because the kernel can't be made to go faster (ex: a virtual router/switch, or a load-balancer): https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp-stack/ https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp...

TomMarius 8y ago

Merely passing from user to kernel space is time consuming, it is correct to bypass the kernel.

pixl97 8y ago

No. Dont play games with userspace/kernelspace translation. This always ends in security flaws that allow complete take over of the machine.

peterwwillis 8y ago

Everything should just be fast, but it usually isn't. So how are you going to deal with that as a business? Hire kernel hackers and spend expensive time and money to fine-tune one version of a kernel that'll never go upstream? Or pay a vendor for a really fast NIC and integrate it into your app? (If you're Google, you hire the kernel hackers, but if you're trying to make money off a product and ship it next quarter, you buy the NIC)

CoolGuySteve 8y ago
Well right now everyone who cares about latency (including me) is spending time integrating some kind of kernel bypass library into their application that's custom and/or proprietary to a single card vendor and often pretty finicky to configure and integrate correctly. It's a massive waste of resources. If the kernel could figure out DRI for graphics then they can figure out universal passthrough networking. All the bypass libraries seem to use the same buffer pool + spinlock slot architecture already. And by your logic, how does anything ever get into the kernel? Nobody should ever contribute anything because "upstream".
- peterwwillis 8y ago
  I'm just saying that fixing the kernel is unlikely to happen unless it's in someone's business interest, and it's usually not. Hell, I bet there's kernel forks out there where people have already solved this, but it won't go upstream, because lawyers.