4 ms·

Rob Pike deserves the benefit of your doubt when discussing UNIX implementations. http://en.wikipedia.org/wiki/Rob_Pike http://en.wikipedia.org/wiki/Rob_Pike

by kmavm 16y ago

Rob Pike deserves the benefit of your doubt when discussing UNIX implementations. http://en.wikipedia.org/wiki/Rob_Pike http://en.wikipedia.org/wiki/Rob_Pike The read(2)/write(2) solution from userspace would involve no copying as well, assuming a competent kernel implementation. The only "overheads" to speak of relative to what's possible with sendfile("2") would be those associated with writing one contiguous page table entry for every 4KB of data to set up the mappings; since sendfile(2) requires that you use mmap'able input, it probably incurs the same overheads. Fine, let's say doing the right thing with read(2)/write(2) is hard, even really hard, and sendfile(2) is faster today. Making expedient shortcuts to performance in the system call API has historically not turned out well in UNIX. People write software which depends on this interface, and that software may well outlive any existing hardware and its quirks.

acqq 16y ago
Classic read and write pointer can have any address, it doesn't have to be a multiple of kernel page size or whatever. Is OS supposed to implicitly do a new "hidden" mmap whenever it sees an address that's multiple of the page size? And to do the lookups for matching chunk still mmaped for every write? How long are you going to keep such relations at all in the system with more reads and writes in progress? I don't see how that concept can be made efficient, and Pike doesn't actually explain in the post how he'd do it. sendfile looks ugly and inelegant but solves a real problem.
rlpb 16y ago
> The read(2)/write(2) solution from userspace would involve no copying as well, assuming a competent kernel implementation. Wouldn't the userspace application have to ensure that the buffers being passed to read(2) and write(2) are page-aligned?
- deleted 16y ago
  [deleted]
- wmf 16y ago
  I think the kernel could copy the un-page-aligned head and tail while remapping the body.
  - jedbrown 16y ago
    Which kernels actually do this? Also, if I read(2), it's supposed to be putting stuff in my buffer (obtained with malloc or whatever). It would seem very complicated to change the mapping for addresses in this buffer. And if I write to the buffer, it must be copy-on-write (in fact, it has to copy the page if I dereference any pointer into it, because most hardware doesn't distinguish between read-only and read-write at the instruction level. It just seems like if the kernel really did this level of mapping tricks, that MPI implementations could do something better without this patch. http://lkml.org/lkml/2010/9/14/468 http://lkml.org/lkml/2010/9/14/468
    - luckydude 16y ago
      IRIX did page flipping on the way out to users (read(2)) and copy-on-write page borrowing on the way in from users (write(2)). That's for networking. For file I/O, if you used O_DIRECT, then the data was dma-ed to/from your pages directly, never went through the kernel's file cache at all. All the file performance came from getting many disk DMA engines running in parallel.
      - caf 16y ago
        The "copy-on-write borrowing" on write() is where the problem lies. To turn the writable pages into COWs you have to update the process's page tables to remove write permission, which requires a TLB flush. These kinds of games are a net loss, at least on modern hardware. (And then the app likely reuses the buffer for the next `read()` anyway, requiring either replacing the page or faulting in a fresh one and doing the C in COW).
        luckydude 16y ago
        Huh. Like to see some numbers on that. Not arguing, just a little surprised. So you have data that shows bcopy() to be faster than the copy-on-write stuff? As for the buffers, known problem, all the apps that used this sort of thing cycled through N buffers for exactly that reason. I think N=4 was way more than enough.
- dfox 16y ago
  I don't see why the buffers have to be specifically page aligned. But there certainly exists some hardware, that requires some specific (unrelated to page size) alignment of it's DMA buffers. But more importantly, such buffers have to be placed in physical memory such that they are accessible to said hardware, which often means in lowest 4GB of physical address space and both kernel (does not know what just allocated page is going to be used for) and user space (cannot influence page allocation) has no way to ensure this without copying pages. IOMMU should solve this problem, but then you have additional costs coming from reprogramming IOMMU.
- aliguori 16y ago
  No. If the act of write() is to construct and transmit and SKB, and SKB can be created from a scatter/gather list of pages plus an offset into the first page. But this implies that you can track the SKBs lifecycle in the kernel such that you can know when the SKB is done being used and therefore can allow write() to return. The trouble is, a write() to a socket doesn't actually generate an SKB. Instead it copies data to a socket buffer and then the socket buffers are used to generate SKBs. Some of this is meant as an optimization to create large packets even from small writes. But of course, a copy is a copy and sometimes this buffering is not actually helpful to userspace.
wmf 16y ago
It's unwise to go against Rob Pike and Keith Adams, but I'm going to give it a try... The read(2)/write(2) solution from userspace would involve no copying as well, assuming a competent kernel implementation. AFAIK that was tried and it's slower than sendfile; sometimes it's even slower than copying data. software may well outlive any existing hardware and its quirks. Virtual memory operations like modifying page tables appear to be consistently getting slower relative to other operations, so I would expect that sendfile/splice will always be faster.
- kmavm 16y ago
  Modifying page tables is just a store on x86; since you would be reducing protections (going from unmapped, to mapped read-only), there is no need for TLB flush or shootdown (both of which are, admittedly, getting relatively more costly).
  - wmf 16y ago
    Apps tend to reuse buffers, so if you zero-copy read() into a buffer that's already populated you'd have to drop the existing pages and replace them with new ones. Then a zero-copy write() has to mark the pages as COW. Would this require a TLB flush? It seems like it would.
    - kmavm 16y ago
      I initially thought that was a killer point, but on second thought, if the memory is already mapped writably the kernel can avoid remappings entirely by directing the I/O into the already-mapped pages.
      - aliguori 16y ago
        Problem is you don't know which recv() a packet is destined for until you actually receive it and can parse the header and apply all of the filtering rules. But what you describe is exactly how O_DIRECT achieves zero/copy for read/write. Of course, if you assume multiple receive queues that are programmable, you can short cut the netfilter code. That was roughly the idea behind VJ channels.
andrewf 16y ago
Realistically, you'd also have to track the allocation, and deallocation, of the range the file is being mmap(2)d into. sendfile(2) (it appears, I don't really spend time in the Linux kernel) piggybacks on the kernel's pipe code; it allocates some non-contiguous buffer pages in kernel space, but it doesn't map any new pages, and it doesn't allocate enough pages for the entire file. It's also been around (and presumably making things faster) for over ten years. That's a long time to let a hypothetical future "best" triumph over an achievable "okay".
crux_ 16y ago
> The read(2)/write(2) solution from userspace would involve no copying as well, assuming a competent kernel implementation. Even though you've got many others chiming in as to why this might not be so, I'll add my own: A userspace program can modify a buffer after data is read in, then write the modified version out -- this implies a local copy or, at the very least, the semantics of a local copy.
loewenskind 16y ago
I know who he is and gave him the benefit of the doubt. That's why I said "I'm not sure he's understanding what this is." instead of just assuming the person had no idea what they were talking about. >The read(2)/write(2) solution from userspace would involve no copying as well I don't believe this is the case as at all. First you have the context switch problem and second of all the reason sendfile can be fast is because it's possible that the kernel never touches the data. It just assigns the disk DMA to a kernel buffer, and then gives the kernel buffer location to the NIC driver to read from. All data reading and writing can be done completely with the hardware with no CPU involvement. read/write can't do it this way because you have the option of doing something with the data you get back from read. Well, it could optimistically hope that nothing will in fact be done with the data but that assumption will be wrong in nearly every single usage.

by kmavm 16y ago

acqq 16y ago

Classic read and write pointer can have any address, it doesn't have to be a multiple of kernel page size or whatever. Is OS supposed to implicitly do a new "hidden" mmap whenever it sees an address that's multiple of the page size? And to do the lookups for matching chunk still mmaped for every write? How long are you going to keep such relations at all in the system with more reads and writes in progress? I don't see how that concept can be made efficient, and Pike doesn't actually explain in the post how he'd do it. sendfile looks ugly and inelegant but solves a real problem.

rlpb 16y ago

> The read(2)/write(2) solution from userspace would involve no copying as well, assuming a competent kernel implementation. Wouldn't the userspace application have to ensure that the buffers being passed to read(2) and write(2) are page-aligned?

deleted 16y ago
[deleted]
wmf 16y ago
I think the kernel could copy the un-page-aligned head and tail while remapping the body.
- jedbrown 16y ago
  Which kernels actually do this? Also, if I read(2), it's supposed to be putting stuff in my buffer (obtained with malloc or whatever). It would seem very complicated to change the mapping for addresses in this buffer. And if I write to the buffer, it must be copy-on-write (in fact, it has to copy the page if I dereference any pointer into it, because most hardware doesn't distinguish between read-only and read-write at the instruction level. It just seems like if the kernel really did this level of mapping tricks, that MPI implementations could do something better without this patch. http://lkml.org/lkml/2010/9/14/468 http://lkml.org/lkml/2010/9/14/468
  - luckydude 16y ago
    IRIX did page flipping on the way out to users (read(2)) and copy-on-write page borrowing on the way in from users (write(2)). That's for networking. For file I/O, if you used O_DIRECT, then the data was dma-ed to/from your pages directly, never went through the kernel's file cache at all. All the file performance came from getting many disk DMA engines running in parallel.
    - caf 16y ago
      The "copy-on-write borrowing" on write() is where the problem lies. To turn the writable pages into COWs you have to update the process's page tables to remove write permission, which requires a TLB flush. These kinds of games are a net loss, at least on modern hardware. (And then the app likely reuses the buffer for the next `read()` anyway, requiring either replacing the page or faulting in a fresh one and doing the C in COW).
      - luckydude 16y ago
        Huh. Like to see some numbers on that. Not arguing, just a little surprised. So you have data that shows bcopy() to be faster than the copy-on-write stuff? As for the buffers, known problem, all the apps that used this sort of thing cycled through N buffers for exactly that reason. I think N=4 was way more than enough.
dfox 16y ago
I don't see why the buffers have to be specifically page aligned. But there certainly exists some hardware, that requires some specific (unrelated to page size) alignment of it's DMA buffers. But more importantly, such buffers have to be placed in physical memory such that they are accessible to said hardware, which often means in lowest 4GB of physical address space and both kernel (does not know what just allocated page is going to be used for) and user space (cannot influence page allocation) has no way to ensure this without copying pages. IOMMU should solve this problem, but then you have additional costs coming from reprogramming IOMMU.
aliguori 16y ago
No. If the act of write() is to construct and transmit and SKB, and SKB can be created from a scatter/gather list of pages plus an offset into the first page. But this implies that you can track the SKBs lifecycle in the kernel such that you can know when the SKB is done being used and therefore can allow write() to return. The trouble is, a write() to a socket doesn't actually generate an SKB. Instead it copies data to a socket buffer and then the socket buffers are used to generate SKBs. Some of this is meant as an optimization to create large packets even from small writes. But of course, a copy is a copy and sometimes this buffering is not actually helpful to userspace.

wmf 16y ago

It's unwise to go against Rob Pike and Keith Adams, but I'm going to give it a try... The read(2)/write(2) solution from userspace would involve no copying as well, assuming a competent kernel implementation. AFAIK that was tried and it's slower than sendfile; sometimes it's even slower than copying data. software may well outlive any existing hardware and its quirks. Virtual memory operations like modifying page tables appear to be consistently getting slower relative to other operations, so I would expect that sendfile/splice will always be faster.

kmavm 16y ago
Modifying page tables is just a store on x86; since you would be reducing protections (going from unmapped, to mapped read-only), there is no need for TLB flush or shootdown (both of which are, admittedly, getting relatively more costly).
- wmf 16y ago
  Apps tend to reuse buffers, so if you zero-copy read() into a buffer that's already populated you'd have to drop the existing pages and replace them with new ones. Then a zero-copy write() has to mark the pages as COW. Would this require a TLB flush? It seems like it would.
  - kmavm 16y ago
    I initially thought that was a killer point, but on second thought, if the memory is already mapped writably the kernel can avoid remappings entirely by directing the I/O into the already-mapped pages.
    - aliguori 16y ago
      Problem is you don't know which recv() a packet is destined for until you actually receive it and can parse the header and apply all of the filtering rules. But what you describe is exactly how O_DIRECT achieves zero/copy for read/write. Of course, if you assume multiple receive queues that are programmable, you can short cut the netfilter code. That was roughly the idea behind VJ channels.

andrewf 16y ago

Realistically, you'd also have to track the allocation, and deallocation, of the range the file is being mmap(2)d into. sendfile(2) (it appears, I don't really spend time in the Linux kernel) piggybacks on the kernel's pipe code; it allocates some non-contiguous buffer pages in kernel space, but it doesn't map any new pages, and it doesn't allocate enough pages for the entire file. It's also been around (and presumably making things faster) for over ten years. That's a long time to let a hypothetical future "best" triumph over an achievable "okay".

crux_ 16y ago

> The read(2)/write(2) solution from userspace would involve no copying as well, assuming a competent kernel implementation. Even though you've got many others chiming in as to why this might not be so, I'll add my own: A userspace program can modify a buffer after data is read in, then write the modified version out -- this implies a local copy or, at the very least, the semantics of a local copy.

loewenskind 16y ago

I know who he is and gave him the benefit of the doubt. That's why I said "I'm not sure he's understanding what this is." instead of just assuming the person had no idea what they were talking about. >The read(2)/write(2) solution from userspace would involve no copying as well I don't believe this is the case as at all. First you have the context switch problem and second of all the reason sendfile can be fast is because it's possible that the kernel never touches the data. It just assigns the disk DMA to a kernel buffer, and then gives the kernel buffer location to the NIC driver to read from. All data reading and writing can be done completely with the hardware with no CPU involvement. read/write can't do it this way because you have the option of doing something with the data you get back from read. Well, it could optimistically hope that nothing will in fact be done with the data but that assumption will be wrong in nearly every single usage.