5 ms·

Ok, I understand why you might "offload" read() from disk backed files file descriptors. But why "sendfile"? I thought sendfile is properly async in linux... Un

by majke 9y ago

Ok, I understand why you might "offload" read() from disk backed files file descriptors. But why "sendfile"? I thought sendfile is properly async in linux... Unless I'm getting something wrong. "aio_write" -> wasn't the idea of AIO operations to actually be truly async? Maybe it's because it requires XFS. Anyway, I'm confused.

taf2 9y ago
From the article "Although Linux provides a kind of asynchronous interface for reading files, it has a couple of significant drawbacks. One of them is alignment requirements for file access and buffers, but NGINX handles that well. But the second problem is worse. The asynchronous interface requires the O_DIRECT flag to be set on the file descriptor, which means that any access to the file will bypass the cache in memory and increase load on the hard disks. That definitely doesn’t make it optimal for many cases."
jsn 9y ago
From what I can see, sendfile isn't really async, it still returns the number of bytes written or error status, and nginx probably needs to wait for that to e.g. log the results, handle short writes, or proceed with whatever next operation must be done to the socket (like closing it). No idea why aio_write needs offloading, though.
- taf2 9y ago
  They say why in the article "The asynchronous interface requires the O_DIRECT flag to be set on the file descriptor, which means that any access to the file will bypass the cache in memory and increase load on the hard disks" They even go into how no interface has been surfaced yet that allows determining whether a file is in cache or not. Also how FreeBSD's aio interface doesn't have the same limitations
  - dom0 9y ago
    I never tried, but mmap and mincore could work (or not, as one of the many special cases in this area). I expect that's not a viable approach for this application, anyway :)
josefbacik 9y ago
Sendfile isn't async, it just avoids the user space copy, and in some cases the copy from the fd to the socket. Aio doesn't require xfs, just O_DIRECT, which can be tricky to get right and more work than userspace guys want to deal with.
nemanjaboric 9y ago
No, sendfile is not necessarily non-blocking (note the difference between async and non-blocking IO), it still blocks the caller if the reading side is a disk device. The advantage of using sendfile is that you need to copy the data from the kernel-space (source) to the user-space (program) and then back to the kernel-space.

by majke 9y ago

taf2 9y ago

From the article "Although Linux provides a kind of asynchronous interface for reading files, it has a couple of significant drawbacks. One of them is alignment requirements for file access and buffers, but NGINX handles that well. But the second problem is worse. The asynchronous interface requires the O_DIRECT flag to be set on the file descriptor, which means that any access to the file will bypass the cache in memory and increase load on the hard disks. That definitely doesn’t make it optimal for many cases."

jsn 9y ago

From what I can see, sendfile isn't really async, it still returns the number of bytes written or error status, and nginx probably needs to wait for that to e.g. log the results, handle short writes, or proceed with whatever next operation must be done to the socket (like closing it). No idea why aio_write needs offloading, though.

taf2 9y ago
They say why in the article "The asynchronous interface requires the O_DIRECT flag to be set on the file descriptor, which means that any access to the file will bypass the cache in memory and increase load on the hard disks" They even go into how no interface has been surfaced yet that allows determining whether a file is in cache or not. Also how FreeBSD's aio interface doesn't have the same limitations
- dom0 9y ago
  I never tried, but mmap and mincore could work (or not, as one of the many special cases in this area). I expect that's not a viable approach for this application, anyway :)

josefbacik 9y ago

Sendfile isn't async, it just avoids the user space copy, and in some cases the copy from the fd to the socket. Aio doesn't require xfs, just O_DIRECT, which can be tricky to get right and more work than userspace guys want to deal with.

nemanjaboric 9y ago

No, sendfile is not necessarily non-blocking (note the difference between async and non-blocking IO), it still blocks the caller if the reading side is a disk device. The advantage of using sendfile is that you need to copy the data from the kernel-space (source) to the user-space (program) and then back to the kernel-space.