5 ms·

What's fast about mmap?

• rishabhaiover 10mo ago
  it allows the program to reference memory without having to manage it in the heap space. it would make the program faster in a memory managed language, otherwise it would reduce the memory footprint consumed by the program.
  • jeffbee 10mo ago
    You mean it converts an expression like `buf[i]` into a baroque sequence of CPU exception paths, potentially involving a trap back into the kernel.
    • rishabhaiover 10mo ago
      I don't fully understand the under the hood mechanics of mmap, but I can sense that you're trying to convey that mmap shouldn't be used a blanket optimization technique as there are tradeoffs in terms of page fault overheads (being at the mercy of OS page cache mechanics)
      • jibal 10mo ago
        I think he's conveying that he doesn't know what he's talking about. buf[i] generates the same code regardless of whether mmap is being used. The first access to a page will cause a trap that loads the page into memory, but this is also true if the memory is read into.
      • StilesCrisis 10mo ago
        Tradeoffs such as "if an I/O error occurs, the program immediately segfaults." Also, I doubt you're I/O bound to the point where mmap noticeably better than read, but I guess it's fine for an experiment.
        • jibal 10mo ago
          An I/O error on a mmapped file causes a SIGBUS, which the program can catch and report. And I/O bound programs are I/O bound whereas programs that aren't, aren't, so it really isn't meaningful to talk about whether "you" are I/O bound to the point that it's significant--maybe you are, maybe you aren't. I agree about experimentation.
• kennethallen 10mo ago
  Two big advantages: You avoid an unnecessary copy. Normal read system call gets the data from disk hardware into the kernel page cache and then copies it into the buffer you provide in your process memory. With mmap, the page cache is mapped directly into your process memory, no copy. All running processes share the mapped copy of the file. There are a lot of downsides to mmap: you lose explicit error handling and fine-grained control of when exactly I/O happens. Consult the classic article on why sophisticated systems like DBMSs do not use mmap: https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/
  • saidinesh5 10mo ago
    This is a very interesting link. I didn't expect mmap to be less performant than read() calls. I now wonder which use cases would mmap suit better - if any... > All running processes share the mapped copy of the file. So something like building linkers that deal with read only shared libraries "plugins" etc ..?
    • squirrellous 10mo ago
      One reason to use shared memory mmap is to ensure that even if your process crashes, the memory stays intact. Another is to communicate between different processes.
    • kennethallen 10mo ago
      mmap is better when: * You want your program to crash on any I/O error because you wouldn't handle them anyway * You value the programming convenience of being able to treat a file on disk as if the entire thing exists in memory * The performance is good enough for your use. As the article showed, sequential scan performance is as good as direct I/O until the page cache fills up *from a single SSD*, and random access performance is as good as direct I/O until the page cache fills up *if you use MADV_RANDOM*. If your data doesn't fit in memory, or is across multiple storage devices, or you don't correctly advise the OS about your access patterns, mmap will probably be much slower To be clear, normal I/O still benefits from the OS's shared page cache, where files that other processes have loaded will probably still be in memory, avoiding waiting on the storage device. But each normal I/O process incurs the space and time cost of a copy into its private memory, unlike mmap.