4 ms·

I wish people used mmap less. Creating a new memory mapping can be pretty expensive! On both Windows and Linux, it involves taking a process-wide reader-writer

I wish people used mmap less. Creating a new memory mapping can be pretty expensive! On both Windows and Linux, it involves taking a process-wide reader-writer lock in exclusive mode (meaning you get to sit and wait behind page faults), doing a bunch of VMA tree manipulation work, doing various kinds of bookkeeping (hello, rmap!) and then, after you return to userspace, entering the kernel again in response to VM faults just to fill in a few pages by doing, inside the kernel, what amounts to a read (2) anyway! Sure, if you use mmap, you get to look at the page cache pages directly instead of copying from them into some application-provided buffer, but most of the time, it's not worth the cost. There are exceptions of course, but you should always default to conventional reads.

mandreyel 8y ago
This is a valid point. My use case was very frequent reads of large files at pretty much unpredictable positions, so in theory mmap seemed justified. However, I never got around thoroughly testing this assumption, and may indeed just have been better off using read(2) and its variants. You seem very experienced, so I hope you don't mind a question. In my use case the files were as large as tens of gigabytes and I was creating read-only mappings of 256KB-1MB chunks in them, keeping the mmap handles around according to a cache policy and RAM usage limit. Do you think in this case using mmap could in theory introduce performance gains? [edit: typo]
- gmueckl 8y ago
  I think that this is the wrong way to use mmap. Just map the whole file at once. The operating system will automatically read the pages you access from disk. And if memory gets tight, these pages will be flushed to disk if they are dirty and then discarded before the system starts paging. These mmapped pages essentially live in the disk cache.
  - quotemstr 8y ago
    > The operating system will automatically read the pages you access from disk. And if memory gets tight, these pages will be flushed to disk if they are dirty and then discarded before the system starts paging. You can tell that you understand how modern OS memory management works when you realize that the OS "automatically read[ing] the pages...from disk" and "flush[ing them] to disk" on memory pressure is paging whether those pages are anonymous pages or mmaped file pages. :-) [Edit: flushing dirty file-paged pages is analogous to swapping anonymous memory to the swapfile. Discarding clean file-backed pages is a bit like discarding anonymous pages that have been made unused through munmap, process death, etc.] But to the GP's point: you don't need (except to conserve address space) to limit file mapping size. I think he really wants something like MADV_FREE. But it's complicated.
    - gmueckl 8y ago
      Therr is a subtle difference between anonymous and mapped readonly pages: the later can be discarded right away because their contents were read from permanemt storage to begin with. Anonymous pages need to be written to disk first and that is significantly slower.
  - mandreyel 8y ago
    Hmm, I see, I haven't come across this suggestion but I may not have researched this topic well enough. I'll try your suggestion next time. I appreciate the responses--learned something new today!
  - jnordwick 8y ago
    Yeah +1 for someone who understands mmap.
- quotemstr 8y ago
  Random access to large files is a legitimate use case! LMDB [1] uses a similar technique, and it works well for them. But depending on the specific application, explicitly application-managed caching via O_DIRECT IO with something like threaded pread or AIO might be even better, because with this explicit model, you control the cache sizing and eviction policy, and it's certainly possible with application-level knowledge to do better than the generic kernel-level LRU/active/inactive/kinda-sorta-works-heuristic stuff can do without application-specific knowledge. Another advantage of using application-managed caching is the ability to take advantage of things like huge pages (which can drastically reduce TLB miss rates), whereas with conventional mmap of conventional files, you're limited to regular 4kB (or whatever) small pages and associated management overhead. (There's no reason in principle filesystems can't use huge pages for page cache, but AFAIK, nobody does it yet.) OTOH, kernel management of page cache allows for better integration of cache eviction with system memory pressure signals and allows for multiple users of a single file to share the memory mirroring the contents of that file. > Do you think in this case using mmap could in theory introduce performance gains? It depends. The right approach depends on a lot of factors, including workload and developer complexity budget. It's funny, really: the more experience you get, the less likely you are to say "$SOLUTION is the bestest evar!" and the more often you say "well, it really depends, so I can't give you an answer". What really strikes me as needless is someone using mmap to read a 10kB ~/.myapplication.lol.ini file or something. [1] http://www.lmdb.tech/doc/ http://www.lmdb.tech/doc/
  - mandreyel 8y ago
    Oh, I missed this response (still a little overwhelmed). I'm so glad I checked again because there is some precious wisdom in there. Thank you for taking the time to write it down. And it indeed seems like I was misusing mmap...well, next time I'll know better!
- cbsmith 8y ago
  Nothing wrong with using mmap for this, though I'd just do a big map of the whole file. That said, you can achieve similar results with readv...
gmueckl 8y ago
If you are reading large chunks of data at once - and you should - then the same process happens, but it is hidden inside the memory manager. I, for one, used memory mapping in the past to significantly speed up code that operates on gigantic data sets. I guess that your mileage does vary.
osivertsson 8y ago
Another problem with mmap is that there is no good way to handle I/O errors. The application will get a signal (SIGSEGV/SIGBUS, can't remember), and no information about what the problem could possibly be. Most applications do not catch these signals and will instead just terminate. Even if you do catch the signal there is a real challenge to know what caused the signal and to keep consistent book-keeping to be able to perform any sane action in response. At a previous employer we started seeing this problem when scaling things in production which was no fun.
- pradn 8y ago
  A library that helped manage MMAP errors would be extremely helpful.
  - quotemstr 8y ago
    I wanted to help address this problem with my signal sharing API proposal. Unfortunately, the glibc people have the attitude that nobody should be using signals, and so they refuse to improve the signals API at all. I strongly disagree. https://www.facebook.com/notes/daniel-colascione/toward-shareable-posix-signals/10157129032641102/ https://www.facebook.com/notes/daniel-colascione/toward-shar...
    - spc476 8y ago
      There's a reason for that---signals are horribly abused as a general purpose signaling mechanism. Originally they were for signaling of actual problems in the code (SIGSEGV, SIGILL) but later signals were not (SIGWINCH I'm looking at you!). I read the page you linked to, and just off the top of my head, trying to manage paging by catching SIGSEGV is how do you determine that it's in response to a real bug (say, dereferencing an undefined pointer)? In my opinion, by the time you get a SIGSEGV, you can't trust the program at all. While it might be nice to have a process handle page faults itself, I think a better API than signal() is required.
      - quotemstr 8y ago
jabl 8y ago
Case in point, a sort-of general purpose IO library was using mmap as the OS interface, and presenting a read/write style API. We had various bugs related to this (e.g. extending or truncating a mmap'ed file is slightly tricky to do correctly etc.). I ripped out the mmap codepath, fixing the mmap-related bugs, and for some benchmarks performance improved by a factor of 20. Now, there are certainly places where mmap is awesome. E.g. if you can push the mmap semantics up to the application level, or you need the sharing semantics etc.
gok 8y ago
The cost of the second kernel trip (on the first page fault) is often mitigated by speculative read-ahead, or the fact that a given page is often in the UBC already. And file-backed memory doesn't contribute to dirty memory. And mmap() makes it easy to use read-only memory, which catches memory corruption bugs. Plus it's easier to use huge pages, which reduces TLB pressure. I <3 mmap
- jzwinck 8y ago
  Memory mapped I/O actually prevents the use of huge pages if the backing storage is an actual disk. See https://lwn.net/Articles/718102/ https://lwn.net/Articles/718102/ Do you know of a way around that, aside from building your own patched kernel?
  - gok 8y ago
    Yeah you need to move your files to tmpfs, but I still consider that easier than dealing with making it happen with malloc+read
- nemo1618 8y ago
  >mmap() makes it easy to use read-only memory, which catches memory corruption bugs You might appreciate this toy Go package I hacked together: https://github.com/lukechampine/freeze https://github.com/lukechampine/freeze It uses mmap and mprotect to "freeze" Go objects; if you try to modify a frozen object, the program crashes.
  - gok 8y ago
    Nice!

mandreyel 8y ago

This is a valid point. My use case was very frequent reads of large files at pretty much unpredictable positions, so in theory mmap seemed justified. However, I never got around thoroughly testing this assumption, and may indeed just have been better off using read(2) and its variants. You seem very experienced, so I hope you don't mind a question. In my use case the files were as large as tens of gigabytes and I was creating read-only mappings of 256KB-1MB chunks in them, keeping the mmap handles around according to a cache policy and RAM usage limit. Do you think in this case using mmap could in theory introduce performance gains? [edit: typo]

gmueckl 8y ago
I think that this is the wrong way to use mmap. Just map the whole file at once. The operating system will automatically read the pages you access from disk. And if memory gets tight, these pages will be flushed to disk if they are dirty and then discarded before the system starts paging. These mmapped pages essentially live in the disk cache.
- quotemstr 8y ago
  > The operating system will automatically read the pages you access from disk. And if memory gets tight, these pages will be flushed to disk if they are dirty and then discarded before the system starts paging. You can tell that you understand how modern OS memory management works when you realize that the OS "automatically read[ing] the pages...from disk" and "flush[ing them] to disk" on memory pressure is paging whether those pages are anonymous pages or mmaped file pages. :-) [Edit: flushing dirty file-paged pages is analogous to swapping anonymous memory to the swapfile. Discarding clean file-backed pages is a bit like discarding anonymous pages that have been made unused through munmap, process death, etc.] But to the GP's point: you don't need (except to conserve address space) to limit file mapping size. I think he really wants something like MADV_FREE. But it's complicated.
  - gmueckl 8y ago
    Therr is a subtle difference between anonymous and mapped readonly pages: the later can be discarded right away because their contents were read from permanemt storage to begin with. Anonymous pages need to be written to disk first and that is significantly slower.
- mandreyel 8y ago
  Hmm, I see, I haven't come across this suggestion but I may not have researched this topic well enough. I'll try your suggestion next time. I appreciate the responses--learned something new today!
- jnordwick 8y ago
  Yeah +1 for someone who understands mmap.
quotemstr 8y ago
Random access to large files is a legitimate use case! LMDB [1] uses a similar technique, and it works well for them. But depending on the specific application, explicitly application-managed caching via O_DIRECT IO with something like threaded pread or AIO might be even better, because with this explicit model, you control the cache sizing and eviction policy, and it's certainly possible with application-level knowledge to do better than the generic kernel-level LRU/active/inactive/kinda-sorta-works-heuristic stuff can do without application-specific knowledge. Another advantage of using application-managed caching is the ability to take advantage of things like huge pages (which can drastically reduce TLB miss rates), whereas with conventional mmap of conventional files, you're limited to regular 4kB (or whatever) small pages and associated management overhead. (There's no reason in principle filesystems can't use huge pages for page cache, but AFAIK, nobody does it yet.) OTOH, kernel management of page cache allows for better integration of cache eviction with system memory pressure signals and allows for multiple users of a single file to share the memory mirroring the contents of that file. > Do you think in this case using mmap could in theory introduce performance gains? It depends. The right approach depends on a lot of factors, including workload and developer complexity budget. It's funny, really: the more experience you get, the less likely you are to say "$SOLUTION is the bestest evar!" and the more often you say "well, it really depends, so I can't give you an answer". What really strikes me as needless is someone using mmap to read a 10kB ~/.myapplication.lol.ini file or something. [1] http://www.lmdb.tech/doc/ http://www.lmdb.tech/doc/
- mandreyel 8y ago
  Oh, I missed this response (still a little overwhelmed). I'm so glad I checked again because there is some precious wisdom in there. Thank you for taking the time to write it down. And it indeed seems like I was misusing mmap...well, next time I'll know better!
cbsmith 8y ago
Nothing wrong with using mmap for this, though I'd just do a big map of the whole file. That said, you can achieve similar results with readv...

gmueckl 8y ago

If you are reading large chunks of data at once - and you should - then the same process happens, but it is hidden inside the memory manager. I, for one, used memory mapping in the past to significantly speed up code that operates on gigantic data sets. I guess that your mileage does vary.

osivertsson 8y ago

Another problem with mmap is that there is no good way to handle I/O errors. The application will get a signal (SIGSEGV/SIGBUS, can't remember), and no information about what the problem could possibly be. Most applications do not catch these signals and will instead just terminate. Even if you do catch the signal there is a real challenge to know what caused the signal and to keep consistent book-keeping to be able to perform any sane action in response. At a previous employer we started seeing this problem when scaling things in production which was no fun.

pradn 8y ago
A library that helped manage MMAP errors would be extremely helpful.
- quotemstr 8y ago
  I wanted to help address this problem with my signal sharing API proposal. Unfortunately, the glibc people have the attitude that nobody should be using signals, and so they refuse to improve the signals API at all. I strongly disagree. https://www.facebook.com/notes/daniel-colascione/toward-shareable-posix-signals/10157129032641102/ https://www.facebook.com/notes/daniel-colascione/toward-shar...
  - spc476 8y ago
    There's a reason for that---signals are horribly abused as a general purpose signaling mechanism. Originally they were for signaling of actual problems in the code (SIGSEGV, SIGILL) but later signals were not (SIGWINCH I'm looking at you!). I read the page you linked to, and just off the top of my head, trying to manage paging by catching SIGSEGV is how do you determine that it's in response to a real bug (say, dereferencing an undefined pointer)? In my opinion, by the time you get a SIGSEGV, you can't trust the program at all. While it might be nice to have a process handle page faults itself, I think a better API than signal() is required.
    - quotemstr 8y ago

jabl 8y ago

Case in point, a sort-of general purpose IO library was using mmap as the OS interface, and presenting a read/write style API. We had various bugs related to this (e.g. extending or truncating a mmap'ed file is slightly tricky to do correctly etc.). I ripped out the mmap codepath, fixing the mmap-related bugs, and for some benchmarks performance improved by a factor of 20. Now, there are certainly places where mmap is awesome. E.g. if you can push the mmap semantics up to the application level, or you need the sharing semantics etc.

gok 8y ago

The cost of the second kernel trip (on the first page fault) is often mitigated by speculative read-ahead, or the fact that a given page is often in the UBC already. And file-backed memory doesn't contribute to dirty memory. And mmap() makes it easy to use read-only memory, which catches memory corruption bugs. Plus it's easier to use huge pages, which reduces TLB pressure. I <3 mmap

jzwinck 8y ago
Memory mapped I/O actually prevents the use of huge pages if the backing storage is an actual disk. See https://lwn.net/Articles/718102/ https://lwn.net/Articles/718102/ Do you know of a way around that, aside from building your own patched kernel?
- gok 8y ago
  Yeah you need to move your files to tmpfs, but I still consider that easier than dealing with making it happen with malloc+read
nemo1618 8y ago
>mmap() makes it easy to use read-only memory, which catches memory corruption bugs You might appreciate this toy Go package I hacked together: https://github.com/lukechampine/freeze https://github.com/lukechampine/freeze It uses mmap and mprotect to "freeze" Go objects; if you try to modify a frozen object, the program crashes.
- gok 8y ago
  Nice!