3 ms·

> Doesn't linux do shared-read-only mapping of pages on read(2) and only copy if you modify? Eh? Do you have a reference for this? I'd be pretty interested to

> Doesn't linux do shared-read-only mapping of pages on read(2) and only copy if you modify? Eh? Do you have a reference for this? I'd be pretty interested to learn more. This sounds like a very weird optimization to me, because, as you point out, it would only work if your buffer (and file offset) was page-aligned. That doesn't happen by accident, so "normal" programs that use read() would almost never get a benefit. To get guaranteed page alignment, you probably have to use mmap() to allocate the buffer in the first place, so... why wouldn't you just mmap the file, then? In fact, you could arguably translate a page-aligned pread() into mmap() in userspace, using MAP_FIXED to force the mapping to be placed at your buffer address. No need for kernel help? So why would a kernel implement this optimization? It seems like a lot of risk (of unexpected side effects) with little potential benefit.

skbpush 6y ago
A small jumping point that you can start from. https://en.wikipedia.org/wiki/Copy-on-write https://en.wikipedia.org/wiki/Copy-on-write The only reason I remember is from first encountering this in code and wondering what the hell the "COW" flag was. >To get guaranteed page alignment, you probably have to use mmap() Most modern implementations of malloc will call mmap in the background.
- kentonv 6y ago
  Yes, I know what copy-on-write is. You can get copy-on-write semantics by passing MAP_PRIVATE to mmap(). So, again, you could literally replace your pread() calls with mmap() calls with MAP_FIXED and MAP_PRIVATE and get the described optimization, without kernel help -- so long as your buffer is page-aligned. If your buffer isn't page aligned, then no such optimization is possible in either the kernel or userspace. So the question is: Why would the kernel implement this optimization if (1) programs that aren't explicitly designed for it (by using page-aligned buffers) won't get the benefit, and (2) programs design for it could implement the same optimization in userspace anyway? > Most modern implementations of malloc will call mmap in the background. Well of course they do. But malloc() won't promise you a page-aligned address, regardless of how it's implemented. If you need a page-aligned address you have to call mmap() directly. (Or I suppose you could malloc() one more page than you need, and then find the page-aligned subset, but it's a lot easier to just call mmap()...)
  - tedunangst 6y ago
    1. It's not too hard to page align a buffer. If you malloc a big buffer, almost certainly page aligned anyway. So requires very little effort. 2. You can try the same with mmap, but large files won't fit in (32 bit) memory, and read() is cheaper than mapping and unmapping as you go.
    - kentonv 6y ago
      > If you malloc a big buffer, almost certainly page aligned anyway. Not necessarily. Remember that malloc() must produce a pointer which can later be passed to free() without any other details. free() needs some way to figure out, at the very least, how big the allocation is, to properly free it. Many allocators, such as glibc's, accomplish this by placing metadata in the bytes immediately before the allocated block -- which implies that a large allocation won't be page-aligned. Some other allocators, such as tcmalloc, manage to place the metadata somewhere else, and in that case maybe they return page-aligned buffers, but it's hardly safe to assume. (I just ran a quick test and verified -- a 4MB malloc() under glibc is typically 16 bytes off of the page boundary, but under tcmalloc it is aligned.) > 2. You can try the same with mmap, but large files won't fit in (32 bit) memory, and read() is cheaper than mapping and unmapping as you go. This thread is discussing the claim that the Linux kernel transparently "optimizes" large read() calls into mmap(), with me arguing that doesn't make a lot of sense as an optimization. If you're saying that this "optimization" would actually be slower, then that supports my point. You may be right. munmap() (or remapping with MAP_FIXED) triggers TLB shootdowns, which are rather expensive in multithreaded programs. This is another reason why "optimizing" read() to mmap() sounds awfully dubious to me. That said, shootdowns would have a constant cost per read(), independent of the size. For a sufficiently gigantic read, the cost of the shootdown would be less than the cost of copying data into the target memory, so the "optimization" would possibly start to make sense at that point?
labawi 6y ago
You are likely correct. I knew you could do zero-copy, but it seems special syscalls are necessary for the stated reasons. Calls like mmap, splice, vmsplice, tee, sendfile, io_uring, ..

skbpush 6y ago

A small jumping point that you can start from. https://en.wikipedia.org/wiki/Copy-on-write https://en.wikipedia.org/wiki/Copy-on-write The only reason I remember is from first encountering this in code and wondering what the hell the "COW" flag was. >To get guaranteed page alignment, you probably have to use mmap() Most modern implementations of malloc will call mmap in the background.

kentonv 6y ago
Yes, I know what copy-on-write is. You can get copy-on-write semantics by passing MAP_PRIVATE to mmap(). So, again, you could literally replace your pread() calls with mmap() calls with MAP_FIXED and MAP_PRIVATE and get the described optimization, without kernel help -- so long as your buffer is page-aligned. If your buffer isn't page aligned, then no such optimization is possible in either the kernel or userspace. So the question is: Why would the kernel implement this optimization if (1) programs that aren't explicitly designed for it (by using page-aligned buffers) won't get the benefit, and (2) programs design for it could implement the same optimization in userspace anyway? > Most modern implementations of malloc will call mmap in the background. Well of course they do. But malloc() won't promise you a page-aligned address, regardless of how it's implemented. If you need a page-aligned address you have to call mmap() directly. (Or I suppose you could malloc() one more page than you need, and then find the page-aligned subset, but it's a lot easier to just call mmap()...)
- tedunangst 6y ago
  1. It's not too hard to page align a buffer. If you malloc a big buffer, almost certainly page aligned anyway. So requires very little effort. 2. You can try the same with mmap, but large files won't fit in (32 bit) memory, and read() is cheaper than mapping and unmapping as you go.
  - kentonv 6y ago
    > If you malloc a big buffer, almost certainly page aligned anyway. Not necessarily. Remember that malloc() must produce a pointer which can later be passed to free() without any other details. free() needs some way to figure out, at the very least, how big the allocation is, to properly free it. Many allocators, such as glibc's, accomplish this by placing metadata in the bytes immediately before the allocated block -- which implies that a large allocation won't be page-aligned. Some other allocators, such as tcmalloc, manage to place the metadata somewhere else, and in that case maybe they return page-aligned buffers, but it's hardly safe to assume. (I just ran a quick test and verified -- a 4MB malloc() under glibc is typically 16 bytes off of the page boundary, but under tcmalloc it is aligned.) > 2. You can try the same with mmap, but large files won't fit in (32 bit) memory, and read() is cheaper than mapping and unmapping as you go. This thread is discussing the claim that the Linux kernel transparently "optimizes" large read() calls into mmap(), with me arguing that doesn't make a lot of sense as an optimization. If you're saying that this "optimization" would actually be slower, then that supports my point. You may be right. munmap() (or remapping with MAP_FIXED) triggers TLB shootdowns, which are rather expensive in multithreaded programs. This is another reason why "optimizing" read() to mmap() sounds awfully dubious to me. That said, shootdowns would have a constant cost per read(), independent of the size. For a sufficiently gigantic read, the cost of the shootdown would be less than the cost of copying data into the target memory, so the "optimization" would possibly start to make sense at that point?

labawi 6y ago

You are likely correct. I knew you could do zero-copy, but it seems special syscalls are necessary for the stated reasons. Calls like mmap, splice, vmsplice, tee, sendfile, io_uring, ..