3 ms·

A small jumping point that you can start from. https://en.wikipedia.org/wiki/Copy-on-write https://en.wikipedia.org/wiki/Copy-on-write The only reason I remem

A small jumping point that you can start from. https://en.wikipedia.org/wiki/Copy-on-write https://en.wikipedia.org/wiki/Copy-on-write The only reason I remember is from first encountering this in code and wondering what the hell the "COW" flag was. >To get guaranteed page alignment, you probably have to use mmap() Most modern implementations of malloc will call mmap in the background.

kentonv 6y ago
Yes, I know what copy-on-write is. You can get copy-on-write semantics by passing MAP_PRIVATE to mmap(). So, again, you could literally replace your pread() calls with mmap() calls with MAP_FIXED and MAP_PRIVATE and get the described optimization, without kernel help -- so long as your buffer is page-aligned. If your buffer isn't page aligned, then no such optimization is possible in either the kernel or userspace. So the question is: Why would the kernel implement this optimization if (1) programs that aren't explicitly designed for it (by using page-aligned buffers) won't get the benefit, and (2) programs design for it could implement the same optimization in userspace anyway? > Most modern implementations of malloc will call mmap in the background. Well of course they do. But malloc() won't promise you a page-aligned address, regardless of how it's implemented. If you need a page-aligned address you have to call mmap() directly. (Or I suppose you could malloc() one more page than you need, and then find the page-aligned subset, but it's a lot easier to just call mmap()...)
- tedunangst 6y ago
  1. It's not too hard to page align a buffer. If you malloc a big buffer, almost certainly page aligned anyway. So requires very little effort. 2. You can try the same with mmap, but large files won't fit in (32 bit) memory, and read() is cheaper than mapping and unmapping as you go.
  - kentonv 6y ago
    > If you malloc a big buffer, almost certainly page aligned anyway. Not necessarily. Remember that malloc() must produce a pointer which can later be passed to free() without any other details. free() needs some way to figure out, at the very least, how big the allocation is, to properly free it. Many allocators, such as glibc's, accomplish this by placing metadata in the bytes immediately before the allocated block -- which implies that a large allocation won't be page-aligned. Some other allocators, such as tcmalloc, manage to place the metadata somewhere else, and in that case maybe they return page-aligned buffers, but it's hardly safe to assume. (I just ran a quick test and verified -- a 4MB malloc() under glibc is typically 16 bytes off of the page boundary, but under tcmalloc it is aligned.) > 2. You can try the same with mmap, but large files won't fit in (32 bit) memory, and read() is cheaper than mapping and unmapping as you go. This thread is discussing the claim that the Linux kernel transparently "optimizes" large read() calls into mmap(), with me arguing that doesn't make a lot of sense as an optimization. If you're saying that this "optimization" would actually be slower, then that supports my point. You may be right. munmap() (or remapping with MAP_FIXED) triggers TLB shootdowns, which are rather expensive in multithreaded programs. This is another reason why "optimizing" read() to mmap() sounds awfully dubious to me. That said, shootdowns would have a constant cost per read(), independent of the size. For a sufficiently gigantic read, the cost of the shootdown would be less than the cost of copying data into the target memory, so the "optimization" would possibly start to make sense at that point?

kentonv 6y ago

Yes, I know what copy-on-write is. You can get copy-on-write semantics by passing MAP_PRIVATE to mmap(). So, again, you could literally replace your pread() calls with mmap() calls with MAP_FIXED and MAP_PRIVATE and get the described optimization, without kernel help -- so long as your buffer is page-aligned. If your buffer isn't page aligned, then no such optimization is possible in either the kernel or userspace. So the question is: Why would the kernel implement this optimization if (1) programs that aren't explicitly designed for it (by using page-aligned buffers) won't get the benefit, and (2) programs design for it could implement the same optimization in userspace anyway? > Most modern implementations of malloc will call mmap in the background. Well of course they do. But malloc() won't promise you a page-aligned address, regardless of how it's implemented. If you need a page-aligned address you have to call mmap() directly. (Or I suppose you could malloc() one more page than you need, and then find the page-aligned subset, but it's a lot easier to just call mmap()...)

tedunangst 6y ago
1. It's not too hard to page align a buffer. If you malloc a big buffer, almost certainly page aligned anyway. So requires very little effort. 2. You can try the same with mmap, but large files won't fit in (32 bit) memory, and read() is cheaper than mapping and unmapping as you go.
- kentonv 6y ago
  > If you malloc a big buffer, almost certainly page aligned anyway. Not necessarily. Remember that malloc() must produce a pointer which can later be passed to free() without any other details. free() needs some way to figure out, at the very least, how big the allocation is, to properly free it. Many allocators, such as glibc's, accomplish this by placing metadata in the bytes immediately before the allocated block -- which implies that a large allocation won't be page-aligned. Some other allocators, such as tcmalloc, manage to place the metadata somewhere else, and in that case maybe they return page-aligned buffers, but it's hardly safe to assume. (I just ran a quick test and verified -- a 4MB malloc() under glibc is typically 16 bytes off of the page boundary, but under tcmalloc it is aligned.) > 2. You can try the same with mmap, but large files won't fit in (32 bit) memory, and read() is cheaper than mapping and unmapping as you go. This thread is discussing the claim that the Linux kernel transparently "optimizes" large read() calls into mmap(), with me arguing that doesn't make a lot of sense as an optimization. If you're saying that this "optimization" would actually be slower, then that supports my point. You may be right. munmap() (or remapping with MAP_FIXED) triggers TLB shootdowns, which are rather expensive in multithreaded programs. This is another reason why "optimizing" read() to mmap() sounds awfully dubious to me. That said, shootdowns would have a constant cost per read(), independent of the size. For a sufficiently gigantic read, the cost of the shootdown would be less than the cost of copying data into the target memory, so the "optimization" would possibly start to make sense at that point?