3 ms·

I’m trying to convince Antirez to use this technique in redis: https://twitter.com/antirez/status/1081197002573139968 https://twitter.com/antirez/status/1081197

by majke 8y ago

I’m trying to convince Antirez to use this technique in redis: https://twitter.com/antirez/status/1081197002573139968 https://twitter.com/antirez/status/1081197002573139968 > There was a problem with that… I don’t remember what exactly, but it was like, the structures you fill did not match how Redis stored the data or something. I need to try again soon or later because the speedup in Redis would be huge.

hoytech 8y ago
Hi Marek, great article as usual. > (2) Pre-warm the disk cache with posix_fadvise(2) and hope for the best. Just as an aside, there is also the option of mlock(2) to improve this technique as implemented in the flash webserver (https://www.usenix.org/legacy/events/usenix99/full_papers/pai/pai.pdf https://www.usenix.org/legacy/events/usenix99/full_papers/pa...). Also some people use vmtouch (https://hoytech.com/vmtouch/ https://hoytech.com/vmtouch/) for similar things. Regarding this new interface, the LWN article says: > AIO poll operations always operate in the "one-shot" mode; once a poll notification has been generated, a new IOCB_CMD_POLL IOCB must be submitted for that file descriptor if further notifications are needed. Does this mean that in the (I think fairly common) case of wanting to persist a poll indefinitely until (for example) a socket is closed, with this new interface you will need to re-issue a poll after every IO operation? If so would that mitigate some/any of the performance benefit you described? Lastly just wanted to point out the following research in case you haven't seen it before (and the various other designs it cites): https://www.usenix.org/legacy/event/atc11/tech/final_files/Soares.pdf https://www.usenix.org/legacy/event/atc11/tech/final_files/S... Cheers!
drewg123 8y ago
I wonder if Linux has considered the technique that FreeBSD uses for async sendfile? The FreeBSD sendfile() syscall is async (mostly; there is a potential to block reading metadata in VOP_BMAP). It launches a request to read disk blocks into pages, and attaches the pages to the socket buffer via mbufs, but marks them M_NOTREADY so that the protocol layer cannot send them. When the disk I/O completes, the mbufs are marked ready in the context of the disk interrupt thread, and then the proto output routine (eg, tcp_output()) is called. This is more-or-less fire and forget for nginx. (yes, it does check the status via kqueue, but it does not block) This was a huge performance win for us at Netflix, and the not-ready concept is the basis for our kernel-tls layer (mbufs are left not-ready, and encrypted after the disk read, and then the crypto thread calls the proto output).
- khc 8y ago
  at cloudflare, we mostly don't stream large files so this is probably not a big win in our case
- loeg 8y ago
  async sendfile is highly specialized for ... sendfile. It'd be nice to have a more general approach in FreeBSD as well as Linux. Honestly, NT has a great model here we could adopt.

by majke 8y ago

hoytech 8y ago

Hi Marek, great article as usual. > (2) Pre-warm the disk cache with posix_fadvise(2) and hope for the best. Just as an aside, there is also the option of mlock(2) to improve this technique as implemented in the flash webserver (https://www.usenix.org/legacy/events/usenix99/full_papers/pai/pai.pdf https://www.usenix.org/legacy/events/usenix99/full_papers/pa...). Also some people use vmtouch (https://hoytech.com/vmtouch/ https://hoytech.com/vmtouch/) for similar things. Regarding this new interface, the LWN article says: > AIO poll operations always operate in the "one-shot" mode; once a poll notification has been generated, a new IOCB_CMD_POLL IOCB must be submitted for that file descriptor if further notifications are needed. Does this mean that in the (I think fairly common) case of wanting to persist a poll indefinitely until (for example) a socket is closed, with this new interface you will need to re-issue a poll after every IO operation? If so would that mitigate some/any of the performance benefit you described? Lastly just wanted to point out the following research in case you haven't seen it before (and the various other designs it cites): https://www.usenix.org/legacy/event/atc11/tech/final_files/Soares.pdf https://www.usenix.org/legacy/event/atc11/tech/final_files/S... Cheers!

drewg123 8y ago

I wonder if Linux has considered the technique that FreeBSD uses for async sendfile? The FreeBSD sendfile() syscall is async (mostly; there is a potential to block reading metadata in VOP_BMAP). It launches a request to read disk blocks into pages, and attaches the pages to the socket buffer via mbufs, but marks them M_NOTREADY so that the protocol layer cannot send them. When the disk I/O completes, the mbufs are marked ready in the context of the disk interrupt thread, and then the proto output routine (eg, tcp_output()) is called. This is more-or-less fire and forget for nginx. (yes, it does check the status via kqueue, but it does not block) This was a huge performance win for us at Netflix, and the not-ready concept is the basis for our kernel-tls layer (mbufs are left not-ready, and encrypted after the disk read, and then the crypto thread calls the proto output).

khc 8y ago
at cloudflare, we mostly don't stream large files so this is probably not a big win in our case
loeg 8y ago
async sendfile is highly specialized for ... sendfile. It'd be nice to have a more general approach in FreeBSD as well as Linux. Honestly, NT has a great model here we could adopt.