3 ms·

So... the driver and device level seems happy here, but is anyone else creeped out by "asynchronous sendfile()"? I mean, how do you even specify that? You hav

by ajross 4y ago

So... the driver and device level seems happy here, but is anyone else creeped out by "asynchronous sendfile()"? I mean, how do you even specify that? You have a giant file you want dumped down the pipe, so you call it, and... just walk away? How do you report errors? What happens to all the data buffered if the other side resets the connection? What happens if the connection just stalls? In synchronous IO paradigms, this is all managed by the application with whatever logic the app author wants to implement. You can report the errors, ping monitoring, whatever. But with this async thing, what's the API for that? Do you have to write kernel code that lives above the driver to implement devops logic? How would one even document that? +1 for the technical wizardry, but seems like it's going to be a long road from here to an "OS" feature that can be documented.

drewg123 4y ago
This has been a feature upstream in FreeBSD for roughly 6 years. If there is a connection RST, then the data buffered in the kernel is released (either freed immediately, or put into the page cache, depending on SF_NOCACHE). sendfile_iodone() is called for completion. If there is no error, it marks the mbufs on the socket buffer holding the pages that were recently brought in as ready, and pokes the TCP stack to send them. If there was an error, it calls TCP's pru_abort() function to tear down the connection and release what's sitting on the socket buffer. See https://github.com/freebsd/freebsd-src/blob/main/sys/kern/kern_sendfile.c#L289 https://github.com/freebsd/freebsd-src/blob/main/sys/kern/ke...
zackmorris 4y ago
This was my thought too. I've been struggling with the concept of "if you don't have anything nice to say, don't say anything at all" lately, because I've been programming too long and just see poison pills and better alternatives everywhere I look. But I believe that async is an anti-pattern. From the article: * When an nginx worker is blocked, it cannot service other requests * Solutions to prevent nginx from blocking like aio or thread pools scale poorly Nothing against nginx (I use it all the time, it's great) but I probably would have used a synchronous blocking approach. The bottleneck there would be artificial limits on stuff like I/O and the number of available sockets or processes. So.. why isn't anyone addressing these contrived limits of sync blocking I/O at a fundamental level? We pretend that context switching overhead is real, but it's not. It's an artifact of poorly written kernels from 30+ years ago (especially in Windows) where too many registers and too much thread state must be saved while swapping threads. We're basically all working around the fact that the big players have traditionally dragged their feet on refactoring that latency. And that some of the more performant approaches like atomic operations using compare and swap (CAS) on thread-safe queues beat locks/mutexes/semaphores. And that content-addressable memory with multiple busses or even network storage beats vertical scaling optimizations. So I dunno, once again this feels like kind of a drink-the-kool-aid article. If we had a better sync blocking foundation, then a simple blocking shell script could serve video and this whole PDF basically goes away. Rinse, repeat with most web programming too, where miles-long async code becomes a single deterministic blocking function that anyone can understand. I'm kind of reaching the point where I expect more from big companies to fix the actual root causes that force these async workarounds. I kind of gave up on stuff like that over the last 10 years, so am behind the times on improvements to sync blocking kernel code. I'd love to hear if anyone knows of an OS that excels at that.
- 0x457 4y ago
  Slide 25 shows benchmark between "old" sendfile and "new" sendfile: https://www.slideshare.net/facepalmtarbz2/new-sendfile-in-english https://www.slideshare.net/facepalmtarbz2/new-sendfile-in-en... > but I probably would have used a synchronous blocking approach. Well, send a patch, then.
- wmf 4y ago
  I probably would have used a synchronous blocking approach Then Varnish is probably more your style. (A discussion between phk and drewg would be fascinating to watch.) We pretend that context switching overhead is real, but it's not. This sounds crackpot to be honest. Linux has put a lot of effort into optimizing context switching (that's why they have NPTL instead of M:N) and I assume FreeBSD has as well. ...this whole PDF basically goes away Sync vs. async doesn't solve any of the NUMA or TLS issues that this whole PDF is about.
- toast0 4y ago
  If you rely on blocking I/O, you've got to deal with blocked processes as i/o isn't instant. You can deal with that by having lots of threads or processes doing the work, but coordinating many threads can be difficult. With async sendfile, you let the kernel manage all of this without needing userspace to do anything, and the kernel is well organized to manage pushing data to the right place when disk i/o completes. If you just want to write code in blocking style, check out Erlang, but know that under the hood it massages most of the blocking into aggregated select/kqueue/epoll/etc calls.
0x457 4y ago
Here is announcement for the feature: https://www.nginx.com/blog/nginx-and-netflix-contribute-new-sendfile2-to-freebsd/ https://www.nginx.com/blog/nginx-and-netflix-contribute-new-... And here are the slides explaining it: https://www.slideshare.net/facepalmtarbz2/new-sendfile-in-english https://www.slideshare.net/facepalmtarbz2/new-sendfile-in-en... There are video of various talks by Gleb Smirnoff explaining all this magic on YouTube. The feature is fully documented in `man 2 sendfile`, it was part of the patch that did the work.

by ajross 4y ago

drewg123 4y ago

This has been a feature upstream in FreeBSD for roughly 6 years. If there is a connection RST, then the data buffered in the kernel is released (either freed immediately, or put into the page cache, depending on SF_NOCACHE). sendfile_iodone() is called for completion. If there is no error, it marks the mbufs on the socket buffer holding the pages that were recently brought in as ready, and pokes the TCP stack to send them. If there was an error, it calls TCP's pru_abort() function to tear down the connection and release what's sitting on the socket buffer. See https://github.com/freebsd/freebsd-src/blob/main/sys/kern/kern_sendfile.c#L289 https://github.com/freebsd/freebsd-src/blob/main/sys/kern/ke...

zackmorris 4y ago

This was my thought too. I've been struggling with the concept of "if you don't have anything nice to say, don't say anything at all" lately, because I've been programming too long and just see poison pills and better alternatives everywhere I look. But I believe that async is an anti-pattern. From the article: * When an nginx worker is blocked, it cannot service other requests * Solutions to prevent nginx from blocking like aio or thread pools scale poorly Nothing against nginx (I use it all the time, it's great) but I probably would have used a synchronous blocking approach. The bottleneck there would be artificial limits on stuff like I/O and the number of available sockets or processes. So.. why isn't anyone addressing these contrived limits of sync blocking I/O at a fundamental level? We pretend that context switching overhead is real, but it's not. It's an artifact of poorly written kernels from 30+ years ago (especially in Windows) where too many registers and too much thread state must be saved while swapping threads. We're basically all working around the fact that the big players have traditionally dragged their feet on refactoring that latency. And that some of the more performant approaches like atomic operations using compare and swap (CAS) on thread-safe queues beat locks/mutexes/semaphores. And that content-addressable memory with multiple busses or even network storage beats vertical scaling optimizations. So I dunno, once again this feels like kind of a drink-the-kool-aid article. If we had a better sync blocking foundation, then a simple blocking shell script could serve video and this whole PDF basically goes away. Rinse, repeat with most web programming too, where miles-long async code becomes a single deterministic blocking function that anyone can understand. I'm kind of reaching the point where I expect more from big companies to fix the actual root causes that force these async workarounds. I kind of gave up on stuff like that over the last 10 years, so am behind the times on improvements to sync blocking kernel code. I'd love to hear if anyone knows of an OS that excels at that.

0x457 4y ago
Slide 25 shows benchmark between "old" sendfile and "new" sendfile: https://www.slideshare.net/facepalmtarbz2/new-sendfile-in-english https://www.slideshare.net/facepalmtarbz2/new-sendfile-in-en... > but I probably would have used a synchronous blocking approach. Well, send a patch, then.
wmf 4y ago
I probably would have used a synchronous blocking approach Then Varnish is probably more your style. (A discussion between phk and drewg would be fascinating to watch.) We pretend that context switching overhead is real, but it's not. This sounds crackpot to be honest. Linux has put a lot of effort into optimizing context switching (that's why they have NPTL instead of M:N) and I assume FreeBSD has as well. ...this whole PDF basically goes away Sync vs. async doesn't solve any of the NUMA or TLS issues that this whole PDF is about.
toast0 4y ago
If you rely on blocking I/O, you've got to deal with blocked processes as i/o isn't instant. You can deal with that by having lots of threads or processes doing the work, but coordinating many threads can be difficult. With async sendfile, you let the kernel manage all of this without needing userspace to do anything, and the kernel is well organized to manage pushing data to the right place when disk i/o completes. If you just want to write code in blocking style, check out Erlang, but know that under the hood it massages most of the blocking into aggregated select/kqueue/epoll/etc calls.

0x457 4y ago

Here is announcement for the feature: https://www.nginx.com/blog/nginx-and-netflix-contribute-new-sendfile2-to-freebsd/ https://www.nginx.com/blog/nginx-and-netflix-contribute-new-... And here are the slides explaining it: https://www.slideshare.net/facepalmtarbz2/new-sendfile-in-english https://www.slideshare.net/facepalmtarbz2/new-sendfile-in-en... There are video of various talks by Gleb Smirnoff explaining all this magic on YouTube. The feature is fully documented in `man 2 sendfile`, it was part of the patch that did the work.