4 ms·

That still doesn't add up though. System call overhead is something like 500ns to 1usec depending on the call. Copying a large 1500 byte UDP MTU from ram is p

That still doesn't add up though. System call overhead is something like 500ns to 1usec depending on the call. Copying a large 1500 byte UDP MTU from ram is pessimistically along the lines of 2.5 usec (1500 bytes / 64 byte cacheline * 100ns L3 access). So a simple UDP recv() + copy from network card buffer to userspace should be about 3-4usec. In the real world, it's more like 6 or 7usec the last time I timed it. Admittedly, the unix API could use a refresh. A lot of unix calls like getaddrinfo are perfectly happy to allocate and give you a buffer. A "fast_recv" could easily do the same and just hand you a page from a kernel page pool directly that you then have to free. By avoiding the buffer copying, you could get something in the same ballpark as kernel bypass but without all the bespoke code and frameworks in every application. And in the end, that's kind of the point of having an "Operating System" in the first place.

ignoramous 8y ago
> So a simple UDP recv() + copy from network card buffer to userspace should be about 3-4usec. In the real world, it's more like 6 or 7usec the last time I timed it. Not sure where the Linux kernel lags (may be at scale it kind of tails out?)-- if you read Google's paper on their network load balancer, Maglev, where section 5.2.1 specifically calls out 30% performance degradation without bypass on smaller packet sizes. > By avoiding the buffer copying, you could get something in the same ballpark as kernel bypass but without all the bespoke code and frameworks in every application. May be this article from Cloudflare helps paint a proper picture of why certain kind of applications may need to keep bypassing the kernel forever because their requirements are very specific and not because the kernel can't be made to go faster (ex: a virtual router/switch, or a load-balancer): https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp-stack/ https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp...

ignoramous 8y ago

> So a simple UDP recv() + copy from network card buffer to userspace should be about 3-4usec. In the real world, it's more like 6 or 7usec the last time I timed it. Not sure where the Linux kernel lags (may be at scale it kind of tails out?)-- if you read Google's paper on their network load balancer, Maglev, where section 5.2.1 specifically calls out 30% performance degradation without bypass on smaller packet sizes. > By avoiding the buffer copying, you could get something in the same ballpark as kernel bypass but without all the bespoke code and frameworks in every application. May be this article from Cloudflare helps paint a proper picture of why certain kind of applications may need to keep bypassing the kernel forever because their requirements are very specific and not because the kernel can't be made to go faster (ex: a virtual router/switch, or a load-balancer): https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp-stack/ https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp...