4 ms·
As a word of warning, mmap is fine if the semantics match the application. mmap is not a good idea for a general purpose read()/write() replacement, e.g. as ad
by jabl 6y ago
As a word of warning, mmap is fine if the semantics match the application.
mmap is not a good idea for a general purpose read()/write() replacement, e.g. as advocated in the 1994 "alloc stream facility" paper by Krieger et al. I worked with an I/O library that followed this strategy, and we had no end of trouble how to robustly deal with resizing files, and also how to do the windowing in a good way (this was in the time where we needed to care about systems with 32-bit pointers, VM space getting tight, but still needed to care about files larger than 2 GB). And then we needed the traditional read/write fallback path anyway, in order to deal with special files like tty's, pipes etc. In the end I ripped out the mmap path, and we saw a perf improvement in some benchmark by x300.
- nlitened 6y agoIs it still the case in 64-bit systems?
- jabl 6y agoExcept for running out of VM space, all the other issues are still there. And even if you have (for the time being) practically unlimited VM space, you may still not want to mmap a file of unbounded size, since setting up all those mappings takes quite a lot of time if you're using the default 4 kB page size. So you probably want to do some kind of windowing anyway. But then if the access pattern is random and the file is large, you have to continually shift the window (munmap + mmap) and performance goes down the drain. So I don't think going to 64-bit systems tilts the balance in favor of mmap.
- pocak 6y agoLinux allocates page tables lazily, and fills them lazily. The only upfront work is to mark the virtual address range as valid and associated with the file. I'd expect mapping giant files to be fast enough to not need windowing.
- jabl 6y agoGood point, scratch that part of my answer. There are still some cases where you'd not want unlimited VM mapping, but those are getting a bit esoteric and at least the most obvious ones are in the process of getting fixed.
- iforgotpassword 6y agoAlso error handling. read and write can return errors, but what happens when you write to a mmaped pointer and the underlying file system has some issue? Assigning a value to a variable cannot return an error. So you get a fine SIGBUS to your application and it crashes. Just the other day I used imagemagick and it always crashed with a SIGBUS and just when I started googling the issue I remembered mmap, noticed that the partition ran out of space, freed up some more and the issue was gone. So you might want to set up a handler for that signal, but now the control flow suddenly jumps to another function if an error occurs, and you have to somehow figure out where in your program the error occurred and then what? Then you remember that longjmp exists and you end up with a steaming pile of garbage code. Only use mmap if you absolutely must. Don't just "mmap all teh files" as it's the new cool thing you learned about.
- jabl 6y agoIndeed. The issue with file resizing I mentioned was mostly related to error handling (what if another process/thread/file descriptor/ truncates the file, etc.). But yes, there are of course other errors as well, like the fs running out of space you mention.
- justin66 6y agoThere’s nothing wrong with using a read only mmap in conjunction with another method for writes.
- iforgotpassword 6y agoYou have exactly the same problem on a read error.
- justin66 6y agoNot the problem you described in your second paragraph.
- iforgotpassword 6y ago
- searealist 6y agoWhat year / hardware / kernel version are you talking about?
- jabl 6y agoOh uh, IIRC 2004/2005 or thereabouts. Personally I was using PC HW running an up to date Linux distro, as I guess was the vast majority of the userbase, but there was a long tail of all kinds of weird and wonderful targets where the software was deployed.
- deleted 6y ago[deleted]