6 ms·
Claiming "mmap is faster than system calls" is dangerous. I once worked for a company where they also heard someone say "mmap is faster than read/write" and as
by AshamedCaptain 6y ago
Claiming "mmap is faster than system calls" is dangerous.
I once worked for a company where they also heard someone say "mmap is faster than read/write" and as a consequence rewrite their while( read() ) loop into the following monstrosity:
1. mmap a 4KB chunk of the file
2. memcpy it into the destination buffer
3. munmap the 4KB chunk
4. repeat until eof
This is different from the claim in the article -- the above monstrosity is individually mmaping each 4KB block, while I presume the article's benchmark is mmaping the entire file in memory at once, which makes much more sense.
After I claimed the "monstrosity" was absurdly stupid, someone pointed to a benchmark they made and found that the "monstrosity" version was actually faster. To me, this made no sense. The monstrosity has triple the syscall overhead vs the read() version, requires manipulating page tables for every 4KB block and as a consequence had several page faults for each 4KB block of the file. Yet it was true: their benchmarks showed the monstrosity version to be slightly faster.
The idealist in me couldn't stand this and I reverted this change, using for my own (unrelated) experiments a binary which used the older, classic, read() loop instead of mmap.
Eventually I noticed I was getting results much faster using my build on my single-socket Xeon than they were getting on their $$$ server farms. Despite what the benchmark said.
Turns out, the "monstrosity" was indeed faster, but if you had several of these binaries running concurrently on the same machine, they would all slow down each other, as if the kernel was having scale issues with multiple processes constantly changing their page tables. The thing would slow down to single-core levels of performance.
I still have no idea why the benchmark was apparently slightly faster, but obviously they were checking it either isolated or on machines where the other processes where running read() loops. I guess that by wasting more kernel CPU time on yourself you may starve other processes in the system leaving more user time for yourself. But once every process does it, the net result is still significantly lowered performance for everyone.
Just yet another anecdote for the experience bag...
- searealist 6y agoYour anecdote doesn't follow your warning. Using mmap in an unusual way (to read chunks) on presumably legacy hardware doesn't generalize to using it in the obvious way (mmap entire files or at least larger windows) on modern hardware.
- Mathnerd314 6y agoI think the story is to always benchmark first, and also to make sure your benchmarks reflect real-world use. What's dangerous is assuming something is faster without benchmarking.
- searealist 6y agoI think many people reading that anecdote may come away with the idea that mmap is bad (and a monstrosity even) and read is good rather than your interpretation that you should benchmark better. I dislike this kind of muddying the waters and I hope my comment provides another perspective for readers.
- Talanes 6y agoThe whole point of the story seems to be that you shouldn't just read a story like this or the linked article and take away a simplistic data point like "mmap is faster" or "mmap is bad"
- anonunivgrad 6y agoBest place to start is to have a good mental model of how things work and why they would be performant or not for a particular use case. Otherwise you're just taking shots in the dark.
- AshamedCaptain 6y agoIndeed, my warning is about being cautious when making generalized claims.
- searealist 6y agoIf someone claimed running was faster than walking and then I told a story about how I once saw someone running in snow shoes on the grass and it was slower than walking then that would just be muddying the waters.
- labawi 6y agoWas it perhaps a multi-threaded task? Because that would almost definitely crawl. In general, unmapping expensive, much more expensive than mapping memory, because you need to do a TLB-shootdown/flush/whatever to make sure a cached version of the old mapping is not used. A read/write does a copy, so no need to mess with mappings and TLBs, hence it can scale very well.
- AshamedCaptain 6y agoIt was multi-processing. I guess mmap(or at least munmap) also needs to send an IPI even if no other processor currently has the same VM, to avoid race conditions.
- CyberDildonics 6y agoIf someone hears "mmap is faster than system calls" and then mmaps and munmaps 4KB chunks at a time in a loop, not realizing that mmap and munmap are actually system calls and that the benefit is not about calling those functions as much as possible, there is no saving them. That's not the fault of a 'dangerous' claim, that's the fault of people who go heads first into something without taking 20 minutes to understand what they are doing or 20 minutes to profile after.
- anonunivgrad 6y agoYep, there's no substitute for a qualitative understanding of the system.
- 411111111111111 6y agoYou'd need significantly more time then 20 minutes to form an informed opinion on the topic of you don't already know basically everything about it. The only thing you could do in that timespan is reading a single summary on the topic and hope that this includes all relevant information. Which is unlikely and the reason why things are often mistakenly taken out of context. And as the original comment mentioned: they did benchmark and it showed an improvement. They just didn't stresstest it, but that's unlikely doable within 20 minutes either.
- CyberDildonics 6y agoIn 20 minutes you can read what mmap does and see that you you can map a file and copy it like memory. In another 20 minutes you can compile and run a medium sized program. Neither of those is enough time for someone to go deep into something, but you can look up the brand new thing you're using and see where your bottlenecks are.
- rini17 6y agoBut without going deeper you are likely to have preconceived notions like "mmapping a gigabyte file into a memory actually uses a gigabyte of memory, oh noes!" I can totally see the thinking behind this case.
- alexchamberlain 6y agoThe kernel can, in theory, estimate what page you're going to load next, so loading 4KB at a time may not have the page faults you'd expect.
- jabl 6y agoOn Linux mmap_sem contention is a well-known concurrency bottleneck, you may have been hitting that. Multiple efforts over the years have failed to fix it, IIRC. I guess one day they'll find a good solution, but until then, take care.
- stanfordkid 6y agoIsn't the whole point of mmap to randomly access the data needed in memory? Did they think memcpy is a totally free operation or something, without any side effects?
- piyh 6y agoOut of curiosity, what was the use case where they were trying to get these marginal gains out of their program?
- beached_whale 6y agommap is a more ergonomic interface than read too. How often are people copying a file to a local buffer, or the whole file a chunk at a time, in order to use the file like an array of bytes. mmap gives us a range of bytes right off the start. Even if not optimal, the simplicity in usage often means less room for bugs.
- lathiat 6y agoBe mindful that can give you Time Of Check/Time Of Use (TOCTOU) security bugs though. The buffer can change out from under you.
- icedchai 6y agoThat can sometimes be a feature, not a bug. I've worked on systems that used memory mapped files as a form of persistent shared memory between processes.
- beached_whale 6y agoThats the open/lock mode too. Same as without
- jeffbee 6y agoThe fact that you once met some idiots who wrote code that wouldn't pass the review of any experienced Unix programmer doesn't really invalidate the fact that faulting in a mapped page is generally faster than getting the same data with a page-sized read. The one thing doesn't have anything to do with the other.
- KMag 6y agoRight, especially if the alignment within the file is a multiple of the page size, and it's a shared mapping, many OSes can make a zero-copy read directly from non-volatile storage into the process's memory (buffer cache pages mapped directly into the process's address space).
- xxs 6y ago"munmap" is actually expensive as it has to flush the TLB
- pmiller2 6y agoI'm also puzzled as to why the benchmark of the monstrosity would have been faster. How is it not doing the exact same work as read()'ing 4KB, and then writing out the same to the destination file? And, as you said, 3x the system call overhead doesn't seem good, either....