18 ms·
Why mmap is faster than system calls (2019)
- aloknnikhil 6y agoPrevious discussion: https://news.ycombinator.com/item?id=24842648 https://news.ycombinator.com/item?id=24842648
- deleted 2y ago[deleted]
- tucnak 6y agoAm I right to assume that Alexandra is well–known in the field? I've never heard the name.
- lionsdan 6y agohttps://www.ece.ubc.ca/faculty/alexandra-sasha-fedorova https://www.ece.ubc.ca/faculty/alexandra-sasha-fedorova
- stingraycharles 6y agoApparently she’s a researcher and MongoDB consultant. https://www.ece.ubc.ca/~sasha/ https://www.ece.ubc.ca/~sasha/
- deleted 6y ago[deleted]
- bzb6 6y agoIt’s an ad.
- bigdict 6y agoHold up. Isn't mmap a system call?
- chrisseaton 6y ago> Hold up. Isn't mmap a system call? That's not what they mean. You set up a memory map with the mmap system call, yes, but that's not the point. The point is then that you can read and write mapped files by reading and writing memory addresses directly - you do not have to use a system call to perform each read and write.
- bzb6 6y agoSo like DMA?
- _0ffh 6y agoI think that comparison would be more confusing than helpful.
- ndesaulniers 6y agoDMA is more like a bulk memory transfer operation usually facilitated by specific hardware that generally is asynchronous and requires manual synchronization. Usually hardware devices perform DMAs of memory regions, like a memcpy() but between physical memories. Memory mappings established via mmap() more so set up the kernel to map in pages when faults on accesses occur. In this case you're not calling into the kernel, the TLB is generating an interrupt when you go to read an address referring to memory that's not yet paged in, which the kernel than handles and restores control flow to userspace without userspace being any wiser (unless userspace is keeping track of time). Handling page faults is faster than the syscalls involved in read() calls, it would seem.
- magicalhippo 6y agoNo. Using mmap is pretending you read your entire file into memory but all the memory got swapped to disk, and so the pages has to be swapped back into memory when you access them. Except you didn't do the initial read into memory, and instead of the system swap file it's using the file you actually want to read when paging data back into memory.
- deleted 6y ago
- jws 6y agoSummary: Mostly syscalls and mmap do the same things just substituting a page fault for a syscall to get to kernel mode, but… In user space her code is using AVX optimized memory copy instructions which are not accessible in kernel mode yielding a significant speed up. Bonus summary: She didn’t use the mmapped data in place in order to make a more apples-to-apples comparison. If you can use the data in place then you will get even better performance.
- anaisbetts 6y ago*She, not he
- jws 6y agoThanks. Curse this language. I just want to refer to people! It’s simple encapsulation and abstraction. I shouldn’t have to care about implementation details irrelevant to the context.
- deleted 6y ago[deleted]
- ryanianian 6y ago"They" is an acceptable gender-neutral pronoun.
- FentanylFloyd 6y agoit's an idiotic newspeakism, lol it's well enough that 'you' can be both singular and plural, we don't need another one
- kortilla 6y agoAcceptable to some, still not frequent enough though to be normalized.
- deleted 6y ago
- noncoml 6y agoThe meat of it is: the functions (for copying data) used for syscall and mmap are very different, and not only in the name. __memmove_avx_unaligned_erms, called in the mmap experiment, is implemented using Advanced Vector Extensions (AVX) (here is the source code of the functions that it relies on). The implementation of copy_user_enhanced_fast_string, on the other hand, is much more modest. That, in my opinion, is the huge reason why mmap is faster. Using wide vector instructions for data copying effectively utilizes the memory bandwidth, and combined with CPU pre-fetching makes mmap really really fast. Why can’t the kernel implementation use AVX? Well, if it did, then it would have to save and restore those registers on each system call, and that would make domain crossing even more expensive. So this was a conscious decision in the Linux kernel
- cwzwarich 6y agoWouldn't REP MOVSB be as fast as an AVX memcpy for 4 KB sizes on recent Intel CPUs?
- adzm 6y agoIt should be, I think, though it's a complicated question whose answer varies on so much, cpu architecture and how it is used. There is a great discussion on it here, too. https://stackoverflow.com/questions/43343231/enhanced-rep-movsb-for-memcpy https://stackoverflow.com/questions/43343231/enhanced-rep-mo...
- justin_ 6y agoThe glibc implementation[0] uses Enhanced REP MOVSB when the array is long enough. It takes a few cycles to start up the ERMS feature, so it's only used on longer arrays. Edit: Wait a minute... if this is true, then how can AVX be responsible for the speed up? Is it related to the size of the buffers being copied internally? [0] Line 48 here: http://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S;h=cf645dd7ffe8785b731cd1332e7f504ab8f5ba8b;hb=88b57b8ed41d5ecf2e1bdfc19556f9246a665ebb http://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_...
- 6y ago
- xoo1 6y agoIt's all very benchmark-chasing theoretical. In practice performance is more complicated and mmap is this weird corner-case over engineered inconsistent optimization thing that often wastes or even "leaks" memory which can be used for actually important for performance caches, it's also awful at error handling and so on. I had to literally patch LevelDB to disable mmap on amd64 once, which eliminated OOMs on those servers, allowed me to run way more LevelDB instances and overall improved performance so significantly, that I had to write this comment.
- CalChris 6y agoNeither mmap() nor read()/write() leak memory.
- jstimpfle 6y agoBut they might "leak" it. What parent meant is that as an mmap() user you have no control how much of a mapping takes actual memory while you're visiting random memory-mapped file locations. Is that documented somewhere?
- rowanG077 6y agoThat's called a space leak not a memory leak.
- AnotherGoodName 6y agoThe paging system will only page in what's being used right now though and paging out has zero cost. Old data will naturally be paged out. To put it directly the answer is each mmap file will need 1 page of physical memory (the area currently being read/written). There may be old pages left around since there's no reason for the OS to page anything out unless some other application asked for the memory. But if they do mmap will go to 1 page just fine and there's zero cost to paging out. I feel mmap gets a bad reputation when people look at memory usage tools that look at total virtual memory allocated. I can mmap a 100GB of files, use 0 physical memory and a lot of memory usage tools will report 100GB of memory usage of a certain type (virtual memory allocated). You then get articles about application X using GB of memory. Anyone trying to correct this is ignored. Google Chrome is somewhat unfairly hit by this. All those articles along the lines of "Why is Google using 4GB with no tabs after i viewed some large PDFs". The answer is that it reserved 4GB of 'addresses' that it has mapped to files. If another application wants to use that memory there's zero cost to paging out those files from memory. The OS is designed to do this and it's what mmap is for.
- CyberRabbi 6y agoI believe if you turn PTI off the syscall numbers for sequential copies would be a lot higher.
- deleted 6y ago[deleted]
- jabl 6y agoAs a word of warning, mmap is fine if the semantics match the application. mmap is not a good idea for a general purpose read()/write() replacement, e.g. as advocated in the 1994 "alloc stream facility" paper by Krieger et al. I worked with an I/O library that followed this strategy, and we had no end of trouble how to robustly deal with resizing files, and also how to do the windowing in a good way (this was in the time where we needed to care about systems with 32-bit pointers, VM space getting tight, but still needed to care about files larger than 2 GB). And then we needed the traditional read/write fallback path anyway, in order to deal with special files like tty's, pipes etc. In the end I ripped out the mmap path, and we saw a perf improvement in some benchmark by x300.
- nlitened 6y agoIs it still the case in 64-bit systems?
- jabl 6y agoExcept for running out of VM space, all the other issues are still there. And even if you have (for the time being) practically unlimited VM space, you may still not want to mmap a file of unbounded size, since setting up all those mappings takes quite a lot of time if you're using the default 4 kB page size. So you probably want to do some kind of windowing anyway. But then if the access pattern is random and the file is large, you have to continually shift the window (munmap + mmap) and performance goes down the drain. So I don't think going to 64-bit systems tilts the balance in favor of mmap.
- pocak 6y agoLinux allocates page tables lazily, and fills them lazily. The only upfront work is to mark the virtual address range as valid and associated with the file. I'd expect mapping giant files to be fast enough to not need windowing.
- jabl 6y agoGood point, scratch that part of my answer. There are still some cases where you'd not want unlimited VM mapping, but those are getting a bit esoteric and at least the most obvious ones are in the process of getting fixed.
- jstimpfle 6y ago> Why can’t the kernel implementation use AVX? Well, if it did, then it would have to save and restore those registers on each system call, and that would make domain crossing even more expensive. So this was a conscious decision in the Linux kernel. I don't follow. So a syscall that could profit from AVX can't use it because then all syscalls would have to restore AVX registers? Why can't the restoring just happen specifically in those syscalls that make use of AVX?
- jeffbee 6y agoYou'd have to have a static analysis of which syscalls can transitively reach which functions, which is probably not possible because linux uses tables of function pointers for many purposes. Also if thread 1 enters the kernel, suspends waiting for some i/o, and the kernel switches to thread 2, how would it know it needed to restore thread 2's registers because of AVX activity of thread 1? And if it did how would it have known to save them?
- jstimpfle 6y agoNot a kernel person, but how about a flag for the thread data structure?
- jeffbee 6y agoYeah actually now that I'm part way through that first cup of coffee, the 2nd part of my comment doesn't make sense, the kernel already has to do a heavier save of a task's register state when it switches tasks.
- xymostech 6y agoI think by "each system call" she meant it like "every time it calls read()", since it would be read() that was using the AVX registers. Since the example program just calls read() over and over, this could add a significant amount of overhead.
- jabl 6y agoI vaguely recall that the Linux kernel has used lazy save/restore of FP registers since way back when.
- silvestrov 6y agoWhy doesn't Intel CPUs implement a modern version of Z80's LDIR instruction (a memmove in a single instruction)? Then the kernel wouldn't have to save any registers. (I'd really like if she had documented exactly which CPU/system she used for benchmarking).
- jeffbee 6y agoIntel CPUs have REP MOVS, which is basically the same thing.
- beagle3 6y agoIt's called REP MOVSB (or MOVSW, MOVSD, maybe also MOVSQ?). It has existed since the 8086 day; and for reasons I don't know, it supposedly works well for big blocks these days (>1K or so) but is supposedly slower than register moves for small blocks.
- JoshTriplett 6y ago> it supposedly works well for big blocks these days (>1K or so) but is supposedly slower than register moves for small blocks. On current processors with Fast Short REP MOVSB (FSRM), REP MOVSB is the fastest method for all sizes. On processors without FSRM, but with ERMS, REP MOVSB is faster for anything longer than ~128 bytes.
- beagle3 6y agoThanks! Is there a simple rule-of-the-thumb about when can one rely on FSRM?
- JoshTriplett 6y agoYou should check the corresponding CPUID bit, but in general, Ice Lake and newer.
- sedatk 6y agoLDIR is slower than unrolling multiple LDI instructions by the way.
- jFriedensreich 6y agoMade me think about LMMD (https://en.m.wikipedia.org/wiki/Lightning_Memory-Mapped_Database https://en.m.wikipedia.org/wiki/Lightning_Memory-Mapped_Data...) and wonder why mmap didn't seem to have catched on more in storage engines
- ricardo81 6y ago*LMDB I use it a bit. The transactional aspect of it requires a bit consideration but generally the performance is good. I'd originally used libJudy in a bunch of places for fast lookups but the init time for programs was being slowed by having to preload everything. Using an mmap/LMDB is a decent middle ground.
- jandrewrogers 6y agoFor storage engines that prioritize performance and scalability, mmap() is a poor choice. Not only is it slower and less scalable than alternatives but it also has many more edge cases and behaviors you have to consider. Compared to a good O_DIRECT/io_submit storage engine design, which is a common alternative, it isn't particularly close. And now we have io_uring as an alternative too. If your use case is quick-and-dirty happy path code then mmap() works fine. In more complex and rigorous environments, like database engines, mmap() is not well-behaved.
- aleden 6y agoThe boost.interprocess library presents the capability to keep data structures (std::list, std::vector, ...) in shared memory (i.e. a memory-mapped file)- "offset pointers" are key to that. I can think of no other programming language that can pull this off, with such grace.
- moldavi 6y agoThat sounds incredible. What kind of patterns does it enable? Have you seen it used well?
- AshamedCaptain 6y agoClaiming "mmap is faster than system calls" is dangerous. I once worked for a company where they also heard someone say "mmap is faster than read/write" and as a consequence rewrite their while( read() ) loop into the following monstrosity: 1. mmap a 4KB chunk of the file 2. memcpy it into the destination buffer 3. munmap the 4KB chunk 4. repeat until eof This is different from the claim in the article -- the above monstrosity is individually mmaping each 4KB block, while I presume the article's benchmark is mmaping the entire file in memory at once, which makes much more sense. After I claimed the "monstrosity" was absurdly stupid, someone pointed to a benchmark they made and found that the "monstrosity" version was actually faster. To me, this made no sense. The monstrosity has triple the syscall overhead vs the read() version, requires manipulating page tables for every 4KB block and as a consequence had several page faults for each 4KB block of the file. Yet it was true: their benchmarks showed the monstrosity version to be slightly faster. The idealist in me couldn't stand this and I reverted this change, using for my own (unrelated) experiments a binary which used the older, classic, read() loop instead of mmap. Eventually I noticed I was getting results much faster using my build on my single-socket Xeon than they were getting on their $$$ server farms. Despite what the benchmark said. Turns out, the "monstrosity" was indeed faster, but if you had several of these binaries running concurrently on the same machine, they would all slow down each other, as if the kernel was having scale issues with multiple processes constantly changing their page tables. The thing would slow down to single-core levels of performance. I still have no idea why the benchmark was apparently slightly faster, but obviously they were checking it either isolated or on machines where the other processes where running read() loops. I guess that by wasting more kernel CPU time on yourself you may starve other processes in the system leaving more user time for yourself. But once every process does it, the net result is still significantly lowered performance for everyone. Just yet another anecdote for the experience bag...
- searealist 6y agoYour anecdote doesn't follow your warning. Using mmap in an unusual way (to read chunks) on presumably legacy hardware doesn't generalize to using it in the obvious way (mmap entire files or at least larger windows) on modern hardware.
- baybal2 6y agoThere is a third option on the table! Using DMA controller. People say what's the difference in between copying memory by CPU, or by DMA controller? The difference is exactly that. You rarely need that, but in some cases you have: 1. You do very long copies, and want to have full CPU performance. 2. You do very long copies, and want to have caches to not be flushed during it. 3. You care about power consumption, as DMA controller may let the CPU core to enter low power mode quicker. 4. On some CPU architectures, you can get wildly spread out pages quicker than with CPU/software.
- beagle3 6y agoThe DMA needs to cooperate with the MMU which is on the CPU these days (and has been for almost 3 decades now). It's a lot of work to set up DMA correctly given physical<->logical memory mapping - so it's only worth it if you have a really big chunk to copy.
- GeorgeTirebiter 6y agoThis is quite interesting. This, to me, seems like a systems bug. In the Embedded World, it is exceedingly common to use DMA for all high-speed transfers -- it's effectively a specialized parallel hardware 'MOV' instruction. Also, I have never had an occasion on modern PC hw to need mmap; read() lseek() are clean and less complex overall. Maybe I lack imagination.
- astrange 6y agommap is being used by libraries under you; it's useful for files on internal drives, that won't be deleted, and want to be accessed randomly, and you don't want to allocate buffers to copy things out of them. For instance, anytime you call a shared library it's been mmapped by dyld/ld.so.
- __turbobrew__ 6y agoEvery time you run a dynamically linked program on linux the system linker mmaps your program + libraries into memory. You may not personally see mmap used, but it is used quite often in software you use.
- CodesInChaos 6y agoMemory mapped files are very tricky outside the happy path. In particular recovery from errors and concurrent modification leading to undefined behaviour. It's a good choice for certain use-cases, such as reading assets shipped with the application, where no untrusted process can write to the file and errors can be assumed to not happen. For high performance code I'd use io_uring.
- utopcell 6y agoLast year we were migrating part of YouTube's serving to a new system and we were observing unexplainable high tail latency. It was eventually attributed to mlock()ing some mmap()ed files, which ended up freezing the whole process for significant amounts of time. Be weary of powerful abstractions.
- lrossi 6y agoFrom the mouth of Linus: https://marc.info/?l=linux-kernel&m=95496636207616&w=2 https://marc.info/?l=linux-kernel&m=95496636207616&w=2 It’s a bit old, but it should still apply. I remember that in general he was annoyed when seeing people recommend mmap instead of read/write for basic I/O usecases. In general, it’s almost always better to use the specialized API (read, write etc.) instead of reinventing the wheel on your own.
- beagle3 6y agoLMDB (and its modern fork MDBX), and kdb+/shakti make incredibly good use of mmap - I suspect it is possible to get similar performance from read(), but probably at 10x the implementation complexity.
- nabla9 6y agoYes. If you do lots of sequential/local reads you can reduce the number of context switches dramatically if you do something like: /* to reduce context switch */ int bsize = 16*BUFSIZ; int fopen_bsize(char * filename) { int fp = fopen(filename, "r"); setvbuf(fp, NULL, _IOFBF, bsize); return fp; }
- justin_ 6y agoI'm not sure the conclusion that vector instructions are responsible for the speed-up is correct. Both implementations seem to use ERMS (using REP MOVSB instructions)[0]. Looking at the profiles, the syscall implementation spends time in the [xfs] driver (even in the long test), while the mmap implementation does not. It appears the real speed-up is related to how memory-mapped pages interact with the buffer cache. I might be misunderstanding things. What is really going on here? [0] Lines 56 and 180 here: http://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S;h=cf645dd7ffe8785b731cd1332e7f504ab8f5ba8b;hb=88b57b8ed41d5ecf2e1bdfc19556f9246a665ebb http://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_...
- petters 6y agoI thought this strange as well. The author even directly links to the source code where REP MOVSB is used.
- sltkr 6y agoThe copy hypothesis also doesn't explain the difference between warm and cold start performance, at least for the "random read" case. For warm start, the speed for read vs mmap is 3 GB/s vs 6 GB/s (i.e. mmap is 2x faster). For cold start, that changes to 0.05 GB/s vs 0.25 GB/s (i.e. mmap is 5x faster). If the only significant difference between read() and mmap() is how data is copied, then shouldn't the performance gap get much smaller in the cold start scenario? After all, you're adding the identical (and pretty large) overhead of reading from disk. Instead, the gap gets much larger. That's strange. (The sequential read case looks more reasonable.) So there must be something else going on besides more efficient copying with mmap.
- scaramanga 6y agoLong story short. Spectre mitigations make ret_from_syscall much slower. Much slower than taking a page-fault.
- layoutIfNeeded 6y ago>Further, since it is unsafe to directly dereference user-level pointers (what if they are null — that’ll crash the kernel!) the data referred to by these pointers must be copied into the kernel. False. If the file was opened with O_DIRECT, then the kernel uses the user-space buffer directly. From man write(2): O_DIRECT (Since Linux 2.4.10) Try to minimize cache effects of the I/O to and from this file. In general this will degrade performance, but it is useful in special situations, such as when applications do their own caching. File I/O is done directly to/from user-space buffers. The O_DIRECT flag on its own makes an effort to transfer data synchronously, but does not give the guarantees of the O_SYNC flag that data and necessary metadata are transferred. To guarantee synchronous I/O, O_SYNC must be used in addition to O_DIRECT. See NOTES below for further discussion.
- wtallis 6y agoI don't think O_DIRECT makes any guarantees about zero-copy operation. It merely disallows kernel-level caching of that data. But the kernel may make a private copy that isn't caching.
- layoutIfNeeded 6y agoWho said it was guaranteed to be zero-copy? The original article said that the data must be copied based on some bogus handwavy argument, and I’ve pointed out that the manpage of write(2) contradicts this when it says the following: >File I/O is done directly to/from user-space buffers.
- dmytroi 6y agoDid some research on the topic of high bandwidth/high IOPS file accesses, some of my conclusions could be wrong though, but as I discovered modern NVMe drives need to have some queue pressure on them to perform at advertised speeds, as in hardware level they are essentially just a separate CPU running in background that has command queue(s). They also need to have requests align with flash memory hierarchy to perform at advertised speeds. So that's puts a quite finicky limitation on your access patterns: 64-256kb aligned blocks, 8+ accesses in parallel. To see that just try CrystalDiskMark and put queue depth at 1-2, and/or block size on something small, like 4kb, and see how your random speed plummets. So given the limitations on the access pattern, if you just mmap your file and memcpy the pointer, you'll get like ~1 access request in flight if I understand right. And also as default page size is 4kb, that will be 4kb request size. And then your mmap relies on IRQ's to get completion notifications (instead of polling the device state), somewhat limiting your IOPS. Sure prefetching will help of course, but it is relying on a lot of heuristic machinery to get the correct access pattern, which sometimes fails. As 7+GB/s drives and 10+Gbe networks become more and more mainstream, the main point where people will realize these requirements are in file copying, for example Windows explorer struggles to copy files at rates 10-25GBe+ simply because how it's file access architecture is designed. And hopefully then we will be better equip to reason about "mmap" vs "read" (really should be pread here to avoid the offset sem in the kernel).
- foota 6y agoTypically reviews of drives publish rates at different queue depths, or at least specify the queue depths tested.
- wtallis 6y agoYep, mmap is really bad for performance on modern hardware because you can only fault on one page at a time (per thread), but SSDs require a high queue depth to deliver the advertised throughput. And you can't overcome that limitation by using more threads, because then you spend all your time on context switches. Hence, io_uring.
- astrange 6y ago
- einpoklum 6y agoThe author says that in userspace memcpy, AVX is used, but > The implementation of copy_user_enhanced_fast_string, on the other hand, is much more modest. Why is that? I mean, if you compiled your kernel for a wide range of machines, then fine, but if you compiled targeting your actual CPU, why would the kernel functions not use AVX?
- Matthias247 6y agoThe 4-16kB buffer sizes are all rather tiny and inefficient for high throughput use-cases, which makes those results not that relevant. Something between 64kB to 1MB seems more applicable.
- pjmlp 6y agoOn the context of Linux.
- amluto 6y agoThis is a poor explanation and poor benchmarking. Let’s see: copy_user_enhanced_fast_string uses a CPU feature that (supposedly) is very fast. Benchmarking it against AVX could be interesting, but it would need actual benchmarking instead of handwaving. It’s worth noting that using AVX at all carries overhead, and it’s not always the right choice even if it’s faster in a tight loop. Page faults, on x86_64, are much slower than syscalls. KPTI and other mitigations erode this difference to some extent. But surely the author should have compared the number of page faults to the number of syscalls. Perf can do this. Finally, munmap() is very, very expensive, as is discarding a mapped page. This is especially true on x86. Workloads that do a lot of munmapping need to be careful, especially in multithreaded programs.
- deleted 6y ago[deleted]
- qaq 6y agoThere is io_uring as an option
- deleted 6y ago[deleted]
- pengaru 6y agoThere isn't exactly a shortage of examples where mmap() was in the short-term quite convenient to get to market quickly with something Good Enough to appear production ready, that in the long-term proved to be rather problematic. Mongodb's original mmap storage engine (32-bit support anyone?), ultimately required replacement (wiredtiger was purchased in part for this reason IIRC). systemd-journald read side (journalctl, `systemctl status`, sd-journal API), performance/scalability definitely suffers from the decision to use mmap for all IO. mmap IO is extremely seductive in terms of programming style/convenience as it facilitates writing procedural, IO-naive code that assumes file-backed memory accesses Just Work. But this is transparently blocking, synchronous code. Since your program isn't explicitly scheduling IO and continuations for execution upon completion of IO, there's zero application-level IO parallelism. The only asynchronous IO possible is what the kernel can automagically perform using relatively dumb page fault heuristics and madvise() hints. When your process triggers a major fault on an mmap region, everything comes to a grinding hault; no speculative execution past that point, no potential for triggering other page faults in other files or independent regions of the same file, it all stops until the data is read from the backing store. Not without the use of threads, which brings a host of other issues.
- jjtheblunt 6y agoZero application level parallelism? What about multiple posix threads on a multicore machine manipulating the same mapped page at the same time?
- pengaru 6y ago> Not without the use of threads, which brings a host of other issues.
- wtallis 6y agoMultiple threads touching the same mapped page at the same time won't give you IO parallelism. You need different threads to be faulting on different pages at the same time in order to get multiple simultaneous IO requests sent to the storage device. But the IO queue depth is still limited by your thread count, and that limit can still be a serious problem even on the latest processors with the highest core counts.
- hendry 6y agoDevelopers using mmap broke the utility of strace for me. Using mmap makes it pretty much impossible to debug what a program is actually doing!
- scaramanga 6y agoI replicated the tests and discovered kernel/libc are both using rep/mov for the copy so copy-speed isn't the cause of the difference. The cause of the difference is spectre mitigations which now make system call overhead much higher than it should be. So much so that the cost of the additional pagefaults incurred by the inefficient mmap version get dwarfed. With mitigations=off they both perform roughly identically. You could smell a rat anyway because AVX copies may be faster, but not that much faster.
- sally1620 6y agoThere are different benefits and drawbacks of mmap. I only use mmap because it simplifies my code (I write experimental code). But the OP and commenters are mostly concerned about the extra copy of read(). For video memory, DMABUF has been used to avoid extra copying. The idea is that the hardware directly DMAs into a buffer mapped in the userspace instead of copying into a kernel buffer and then copying into userspace. A system call is still involved, but no extra copying of data. I wonder if anyone has actually implemented a disk I/O interface using DMABUF. There are a couple of low-level disk interfaces. But I don't know if any of them uses DMABUF.
- known 6y agoIt's like array access versus hash access;
- ngneer 6y agoOn a related tangent, why is mmap() needed to begin with? Is it not an example of a leaky abstraction? I know, I know, how else would a process communicate its needs to the memory management system. I have seen shellcode that marks writeable pages executable using mmap, so it is certainly useful.
- rahimiali 6y agoLike others, it's unclear to me that AVX is responsible for the speedup. copy_user_enhanced_fast_string seems to be a beast. To copy data from kernel memory to userland (code link below), for every byte it copies, it checks if it's crossing a page boundary and about to cause a page fault. I don't see how this whole function could get compiled down to just a REP MOVSB. [0] https://elixir.bootlin.com/linux/v4.2/source/arch/x86/lib/usercopy_64.c#L72 https://elixir.bootlin.com/linux/v4.2/source/arch/x86/lib/us...