5 ms·

Sendfile: a system call for web developers to know about

52 points by Symmetry 11y ago

tyingq 11y ago
Related...Apache, nginx, and probably other web servers can be coerced into handling sendfile() for you. https://tn123.org/mod_xsendfile/ https://tn123.org/mod_xsendfile/ https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/# https://www.nginx.com/resources/wiki/start/topics/examples/x...
- dullgiulio 11y ago
  Also any webserver written in Go uses sendfile transparently when sending mmap-able data down to a socket.
- mixmastamyk 11y ago
  That nginx config is hard to grok. Is it really that complicated and not just "sendfile = on" ?
  - tyre 11y ago
    You are correct! There is a simple sendfile in the http core module. http://nginx.org/en/docs/http/ngx_http_core_module.html#sendfile http://nginx.org/en/docs/http/ngx_http_core_module.html#send...
  - dividuum 11y ago
    If you have a public url for a file, then you can just point your users to that url and use sendfile=on. Nginx will then use sendfile. But consider the usecase where you have a bigger download hidden behind some kind of authentication. You'd do that authentication in your normal backend and return the X-Accel-Redirect header in your response. Nginx will then take over and send the data using sendfile.
  - tyingq 11y ago
    I linked to the "harder to grok" piece because that's what would give you programmatic control by setting an X header. That's as opposed to calling sendfile() yourself from your app. Helpful if you're trying to do something more than just serve up static files...like maybe in a page cache, where you also want dynamic control over other headers, etc. The simpler sendfile=on would just be instructing nginx to use it for static files.
luckydude 11y ago
This is really old but the article mentions splice() and here's the original thinking on that syscall. Linux didn't take all the ideas but they took some: http://www.mcvoy.com/lm/bitmover/lm/papers/splice.pdf http://www.mcvoy.com/lm/bitmover/lm/papers/splice.pdf
- joveian 11y ago
  Or even earlier there is "Exploiting In-Kernel Data Paths to Improve I/O Throughput and CPU Availability" by Fall and Pasquale: https://cseweb.ucsd.edu/~pasquale/Research/Papers/usenix93.pdf https://cseweb.ucsd.edu/~pasquale/Research/Papers/usenix93.p...
amelius 11y ago
Seems quite useless to me. For, what if you want to prefix the file with a header first, or add a trailer? Or what if you want to escape certain parts, or zip the file before sending? Or if you want to send the file incrementally? It is better (more powerful) to have simple primitives, and work up from them. If you really want something like sendfile, you could also just find the right library.
- julik 11y ago
  It is in fact very very useful but underutilized, for a very specific purpose - serving very large amounts of data from pretty much any runtime (like Ruby or Python). With those runtimes it works great because you can avoid stuffing the contents of the files you send out into the VM memory, greatly reducing memory and GC pressure. You can also cache the computation product in files, and then send those files out via sendfile() - and the computation can happen on a background thread or in a parallel process.
- luckydude 11y ago
  Any well written server application eventually comes down to how fast can you copy the data. This avoids a copy so it can double performance. As for headers/trailers, it's a file descriptor, nothing prevents you from writing to it before or after the sendfile() call. Simple primitives are fine but this is faster. Unless you are going to do the page flipping tricks that Irix did, you can't approach this speed (and Irix tricks aren't as fast as this, better than the bcopy() but not as fast as this).
  - Matthias247 11y ago
    Reality is sometimes a bit different from theory here. Let's say you want to send a small amount of data (1k) from a file to network, some header before it and some trailer behind it. With sendfile you would do three system calls, two normal writes and a sendfile. Without it you would read the file, copy the content, the header and the trailer into a single buffer and issue a single write syscall with it (of course you have the read call before). Due to the lower number of write calls this could be faster then sendfile, as copying a small amount of data might yield a lower overhead then multiple write requests. Of course that situation will change depending on the amount of data - so I think sendfile is one possible optimization that should be benchmarked for the particular application before using it everywhere.

52 points by Symmetry 11y ago

Related...Apache, nginx, and probably other web servers can be coerced into handling sendfile() for you. https://tn123.org/mod_xsendfile/ https://tn123.org/mod_xsendfile/ https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/# https://www.nginx.com/resources/wiki/start/topics/examples/x...

dullgiulio 11y ago
Also any webserver written in Go uses sendfile transparently when sending mmap-able data down to a socket.
mixmastamyk 11y ago
That nginx config is hard to grok. Is it really that complicated and not just "sendfile = on" ?
- tyre 11y ago
  You are correct! There is a simple sendfile in the http core module. http://nginx.org/en/docs/http/ngx_http_core_module.html#sendfile http://nginx.org/en/docs/http/ngx_http_core_module.html#send...
- dividuum 11y ago
  If you have a public url for a file, then you can just point your users to that url and use sendfile=on. Nginx will then use sendfile. But consider the usecase where you have a bigger download hidden behind some kind of authentication. You'd do that authentication in your normal backend and return the X-Accel-Redirect header in your response. Nginx will then take over and send the data using sendfile.
- tyingq 11y ago
  I linked to the "harder to grok" piece because that's what would give you programmatic control by setting an X header. That's as opposed to calling sendfile() yourself from your app. Helpful if you're trying to do something more than just serve up static files...like maybe in a page cache, where you also want dynamic control over other headers, etc. The simpler sendfile=on would just be instructing nginx to use it for static files.

luckydude 11y ago

This is really old but the article mentions splice() and here's the original thinking on that syscall. Linux didn't take all the ideas but they took some: http://www.mcvoy.com/lm/bitmover/lm/papers/splice.pdf http://www.mcvoy.com/lm/bitmover/lm/papers/splice.pdf

joveian 11y ago
Or even earlier there is "Exploiting In-Kernel Data Paths to Improve I/O Throughput and CPU Availability" by Fall and Pasquale: https://cseweb.ucsd.edu/~pasquale/Research/Papers/usenix93.pdf https://cseweb.ucsd.edu/~pasquale/Research/Papers/usenix93.p...

amelius 11y ago

Seems quite useless to me. For, what if you want to prefix the file with a header first, or add a trailer? Or what if you want to escape certain parts, or zip the file before sending? Or if you want to send the file incrementally? It is better (more powerful) to have simple primitives, and work up from them. If you really want something like sendfile, you could also just find the right library.

julik 11y ago
It is in fact very very useful but underutilized, for a very specific purpose - serving very large amounts of data from pretty much any runtime (like Ruby or Python). With those runtimes it works great because you can avoid stuffing the contents of the files you send out into the VM memory, greatly reducing memory and GC pressure. You can also cache the computation product in files, and then send those files out via sendfile() - and the computation can happen on a background thread or in a parallel process.
luckydude 11y ago
Any well written server application eventually comes down to how fast can you copy the data. This avoids a copy so it can double performance. As for headers/trailers, it's a file descriptor, nothing prevents you from writing to it before or after the sendfile() call. Simple primitives are fine but this is faster. Unless you are going to do the page flipping tricks that Irix did, you can't approach this speed (and Irix tricks aren't as fast as this, better than the bcopy() but not as fast as this).
- Matthias247 11y ago
  Reality is sometimes a bit different from theory here. Let's say you want to send a small amount of data (1k) from a file to network, some header before it and some trailer behind it. With sendfile you would do three system calls, two normal writes and a sendfile. Without it you would read the file, copy the content, the header and the trailer into a single buffer and issue a single write syscall with it (of course you have the read call before). Due to the lower number of write calls this could be faster then sendfile, as copying a small amount of data might yield a lower overhead then multiple write requests. Of course that situation will change depending on the amount of data - so I think sendfile is one possible optimization that should be benchmarked for the particular application before using it everywhere.