4 ms·

Quoting another user, jcranmer [1]: > "The fundamental operation of mmap is to add new entries to the page table of a process, and the precise properties of th

Quoting another user, jcranmer [1]: > "The fundamental operation of mmap is to add new entries to the page table of a process, and the precise properties of those entries are heavily dependent on what the arguments to mmap are. > When you mmap a regular file, you're essentially adding an entry to the page table that shares the data with the kernel's filesystem cache." Now, understanding this requires some understanding of an OS's virtual memory function, and what "page tables" are. Those are what the OS use to track a process' memory, whose granularity is in "pages" (historically 4kB on Linux, though others use larger granularity such as 16kB). mmap() has a a lot of flags [2] that affect the properties of those mapped pages. It is the swiss-army knife of memory management on Linux. Some of those properties allow you to share memory with other processes (MAP_SHARED | MAP_ANONYMOUS), or just allocate memory (MAP_ANONYMOUS) or, by default, map an (open) file specified by the `fd` argument. (fun fact! On linux, when you malloc(), you don't actually get memory, just an IOU from the kernel. Only when you access that memory, ie accessing those memory pages, does the kernel actually make the effort of allocating you that memory.) [1] https://news.ycombinator.com/item?id=35412842 https://news.ycombinator.com/item?id=35412842 [2] https://linux.die.net/man/2/mmap https://linux.die.net/man/2/mmap

xiphias2 4y ago
One thing I don't understand is that if I read a gigabyte from a file with the read call, why the kernel can't see that it's unused allocated memory, create copy on write pages, which could just be shared with the disk cache as long as the data is aligned (which it should be after a malloc + read call). It seems the intuitive way to implement read if there's a complex memory subsystem that does all these great things anyways.
- remexre 4y ago
  malloc() and the kernel would need to conspire to know this before you call read, when malloc() gives you back the memory in the first place
  - xiphias2 4y ago
    I guess you haven't read my parent: malloc doesn't give you memory, just reserves pages to be used in the future.
    - dfox 4y ago
      malloc(3) does not know or care about whether the OS actually commits the VM allocation that it got from somewhere (be it mmap(2) or sbrk(2)), apart from the fact that most implementations of malloc would write some kind of book-keeping header before the returned object (observe that on amd64 glibc, malloc(3) for large allocations returns pointers that are offset +0x10 from page alignment, that is where the two word allocated block header lives). The fact that the memory might not really exist is mostly invisible to userspace (as in, there is no portable way to find out that it does not exist). The OS somehow detecting that argument to read(2) is page aligned block of uncommited meory and transparently creating MAP_PRIVATE mapping from that seems like nifty idea, but userspace tends to not have page-aligned buffers that often (see the aforementioned 0x10 offset for one reason why) and applications that intentionally allocates IO buffers with some particular alignment do so in order to bypass the block cache for particular IO patterns and are exactly the kinds of applications that tend to use mmap(2) for IO where that does not matter.
      - comex 4y ago
        > most implementations of malloc would write some kind of book-keeping header before the returned object (observe that on amd64 glibc, malloc(3) for large allocations returns pointers that are offset +0x10 from page alignment, that is where the two word allocated block header lives). Hmm. You're right about glibc, but macOS for instance reliably returns page-aligned pointers for large allocations. I wonder how Windows behaves.
- Joker_vD 4y ago
  > it's unused allocated memory Well, first of all, it's not: malloc may have previously stored its internal structures in that memory so it may be dirty. Second, AFAIK if you ask malloc() for a gigabyte, it'll internally call mmap() on an anonymous file and will give you its result.
  - xiphias2 4y ago
    I see, that's how it's implemented. But from what I understand there may be nothing in POSIX forbidding the kernel to do this optimization, just kernel authors decided against it (which is understandable).
    - comex 4y ago
      That's right. Indeed, if the optimization is implemented properly it should have no observable behavioral difference except for performance (better in some cases, worse in others) which is not really something POSIX concerns itself with. So POSIX doesn't forbid it. I can think of one edge case not mentioned by others, though. One of the main benefits of mmap is that the data can be paged out while not used. But when you try to page the data back in, the attempt may fail, or it may return different data than was originally there. This is particularly likely with network filesystems or external storage. With mmap, these situations result in, respectively, the process receiving SIGBUS, or the contents of memory changing out from under it. The former is usually a crash (unless the program installs a signal handler), and the latter can be worse than a crash, if either program logic or compiler optimizations make assumptions about memory staying consistent when not written to. A program using mmap is opting in to that risk. A program using malloc and read is not. That said, it may be reasonable to trust that the system’s main disk(s) (however you define that) won’t misbehave this way, and limit the optimization to files mapped from them. After all, if swap is enabled, the same potential risks exist for any memory that’s swapped out. Even without swap, if the root partition starts failing, the system is not going to stay up for long. Or, the optimization could be enabled for other disks, but with the caveat that the pages would be swapped to the swap file/partition rather than paged out normally.
      - 4y ago
- andrewf 4y ago
  That's what mmap is for :) When a read call completes, you know you've successfully read the data from the underlying device. If you have swap disabled you know you can read and write from that memory quickly. With mmap'd data, a memory read/write can trigger a page fault and I/O, and if you encounter an error like a network filesystem not responding in time, the program sees a segfault.
- ElectricalUnion 4y ago
  > if I read a gigabyte from a file with the read call `mmap` signals the kernel that you want to use the data someday. You `mmap` it, it automatically handles fetching and throwing unused bytes away. `read` signals the kernel that you want to do something with the data now. You `read` it, you need it in the memory now. > why the kernel can't see that it's unused allocated memory Only a in-process garbage collector would really know what part of your memory is unused. The kernel can't really know for sure.

xiphias2 4y ago

One thing I don't understand is that if I read a gigabyte from a file with the read call, why the kernel can't see that it's unused allocated memory, create copy on write pages, which could just be shared with the disk cache as long as the data is aligned (which it should be after a malloc + read call). It seems the intuitive way to implement read if there's a complex memory subsystem that does all these great things anyways.

remexre 4y ago
malloc() and the kernel would need to conspire to know this before you call read, when malloc() gives you back the memory in the first place
- xiphias2 4y ago
  I guess you haven't read my parent: malloc doesn't give you memory, just reserves pages to be used in the future.
  - dfox 4y ago
    malloc(3) does not know or care about whether the OS actually commits the VM allocation that it got from somewhere (be it mmap(2) or sbrk(2)), apart from the fact that most implementations of malloc would write some kind of book-keeping header before the returned object (observe that on amd64 glibc, malloc(3) for large allocations returns pointers that are offset +0x10 from page alignment, that is where the two word allocated block header lives). The fact that the memory might not really exist is mostly invisible to userspace (as in, there is no portable way to find out that it does not exist). The OS somehow detecting that argument to read(2) is page aligned block of uncommited meory and transparently creating MAP_PRIVATE mapping from that seems like nifty idea, but userspace tends to not have page-aligned buffers that often (see the aforementioned 0x10 offset for one reason why) and applications that intentionally allocates IO buffers with some particular alignment do so in order to bypass the block cache for particular IO patterns and are exactly the kinds of applications that tend to use mmap(2) for IO where that does not matter.
    - comex 4y ago
      > most implementations of malloc would write some kind of book-keeping header before the returned object (observe that on amd64 glibc, malloc(3) for large allocations returns pointers that are offset +0x10 from page alignment, that is where the two word allocated block header lives). Hmm. You're right about glibc, but macOS for instance reliably returns page-aligned pointers for large allocations. I wonder how Windows behaves.
Joker_vD 4y ago
> it's unused allocated memory Well, first of all, it's not: malloc may have previously stored its internal structures in that memory so it may be dirty. Second, AFAIK if you ask malloc() for a gigabyte, it'll internally call mmap() on an anonymous file and will give you its result.
- xiphias2 4y ago
  I see, that's how it's implemented. But from what I understand there may be nothing in POSIX forbidding the kernel to do this optimization, just kernel authors decided against it (which is understandable).
  - comex 4y ago
    That's right. Indeed, if the optimization is implemented properly it should have no observable behavioral difference except for performance (better in some cases, worse in others) which is not really something POSIX concerns itself with. So POSIX doesn't forbid it. I can think of one edge case not mentioned by others, though. One of the main benefits of mmap is that the data can be paged out while not used. But when you try to page the data back in, the attempt may fail, or it may return different data than was originally there. This is particularly likely with network filesystems or external storage. With mmap, these situations result in, respectively, the process receiving SIGBUS, or the contents of memory changing out from under it. The former is usually a crash (unless the program installs a signal handler), and the latter can be worse than a crash, if either program logic or compiler optimizations make assumptions about memory staying consistent when not written to. A program using mmap is opting in to that risk. A program using malloc and read is not. That said, it may be reasonable to trust that the system’s main disk(s) (however you define that) won’t misbehave this way, and limit the optimization to files mapped from them. After all, if swap is enabled, the same potential risks exist for any memory that’s swapped out. Even without swap, if the root partition starts failing, the system is not going to stay up for long. Or, the optimization could be enabled for other disks, but with the caveat that the pages would be swapped to the swap file/partition rather than paged out normally.
    - 4y ago
andrewf 4y ago
That's what mmap is for :) When a read call completes, you know you've successfully read the data from the underlying device. If you have swap disabled you know you can read and write from that memory quickly. With mmap'd data, a memory read/write can trigger a page fault and I/O, and if you encounter an error like a network filesystem not responding in time, the program sees a segfault.
ElectricalUnion 4y ago
> if I read a gigabyte from a file with the read call `mmap` signals the kernel that you want to use the data someday. You `mmap` it, it automatically handles fetching and throwing unused bytes away. `read` signals the kernel that you want to do something with the data now. You `read` it, you need it in the memory now. > why the kernel can't see that it's unused allocated memory Only a in-process garbage collector would really know what part of your memory is unused. The kernel can't really know for sure.