4 ms·

> The read(2)/write(2) solution from userspace would involve no copying as well, assuming a competent kernel implementation. Wouldn't the userspace application

• deleted 16y ago
  [deleted]
• wmf 16y ago
  I think the kernel could copy the un-page-aligned head and tail while remapping the body.
  • jedbrown 16y ago
    Which kernels actually do this? Also, if I read(2), it's supposed to be putting stuff in my buffer (obtained with malloc or whatever). It would seem very complicated to change the mapping for addresses in this buffer. And if I write to the buffer, it must be copy-on-write (in fact, it has to copy the page if I dereference any pointer into it, because most hardware doesn't distinguish between read-only and read-write at the instruction level. It just seems like if the kernel really did this level of mapping tricks, that MPI implementations could do something better without this patch. http://lkml.org/lkml/2010/9/14/468 http://lkml.org/lkml/2010/9/14/468
    • luckydude 16y ago
      IRIX did page flipping on the way out to users (read(2)) and copy-on-write page borrowing on the way in from users (write(2)). That's for networking. For file I/O, if you used O_DIRECT, then the data was dma-ed to/from your pages directly, never went through the kernel's file cache at all. All the file performance came from getting many disk DMA engines running in parallel.
      • caf 16y ago
        The "copy-on-write borrowing" on write() is where the problem lies. To turn the writable pages into COWs you have to update the process's page tables to remove write permission, which requires a TLB flush. These kinds of games are a net loss, at least on modern hardware. (And then the app likely reuses the buffer for the next `read()` anyway, requiring either replacing the page or faulting in a fresh one and doing the C in COW).
        • luckydude 16y ago
          Huh. Like to see some numbers on that. Not arguing, just a little surprised. So you have data that shows bcopy() to be faster than the copy-on-write stuff? As for the buffers, known problem, all the apps that used this sort of thing cycled through N buffers for exactly that reason. I think N=4 was way more than enough.
• dfox 16y ago
  I don't see why the buffers have to be specifically page aligned. But there certainly exists some hardware, that requires some specific (unrelated to page size) alignment of it's DMA buffers. But more importantly, such buffers have to be placed in physical memory such that they are accessible to said hardware, which often means in lowest 4GB of physical address space and both kernel (does not know what just allocated page is going to be used for) and user space (cannot influence page allocation) has no way to ensure this without copying pages. IOMMU should solve this problem, but then you have additional costs coming from reprogramming IOMMU.
• aliguori 16y ago
  No. If the act of write() is to construct and transmit and SKB, and SKB can be created from a scatter/gather list of pages plus an offset into the first page. But this implies that you can track the SKBs lifecycle in the kernel such that you can know when the SKB is done being used and therefore can allow write() to return. The trouble is, a write() to a socket doesn't actually generate an SKB. Instead it copies data to a socket buffer and then the socket buffers are used to generate SKBs. Some of this is meant as an optimization to create large packets even from small writes. But of course, a copy is a copy and sometimes this buffering is not actually helpful to userspace.