17 ms·

How memory maps (mmap) deliver faster file access in Go

141 points by ingve 1y ago

nawgz 1y ago
Sounds interesting. Why wouldn’t the OS itself default to this behavior? Could it fall apart under load, or is it just not important enough to replace the legacy code relying on it?
- trenchpilgrim 1y ago
  1. mmap was added to Unix later by Sun, it wasn't in the original Unix 2. As the article points out mmap is very fast for reading huge amounts of data but is a lot slower at other file operations. For reading smallish files, which is the majority of calls most software will make to the filesystem, the regular file syscalls are better. 3. If you're on a modern Linux you might be better off with io_uring than mmap.
  - scottlamb 1y ago
    All true, and it's not just performance either. The API is just different. mmap data can change at any time. In fact, if the file shrinks, access to a formerly valid region of memory has behavior that is unspecified by the Single Unix Specification. (On Linux, it causes a SIGBUS if you access a page that is entirely invalid; bytes within the last page after the last valid byte probably are zeros or something? unsure.) In theory I suppose you could have a libc that mostly emulates read() and write() calls on files [1] with memcpy() on mmap()ed regions. But I don't think it'd be quite right. For one thing, that read() behavior after shrink would be a source of error. Higher-level APIs might be more free to do things with either mmap or read/write. [1] just on files; so it'd have to track which file descriptors are files as opposed to sockets/pipes/etc, maintaining the cached lengths and mmap()ed regions and such. libc doesn't normally do that, and it'd go badly if you bypass it with direct system calls.
    - nawgz 1y ago
      Interesting callouts! Thanks
  - deleted 1y ago
    [deleted]
- toast0 1y ago
  > Why wouldn’t the OS itself default to this behavior? Could it fall apart under load, or is it just not important enough to replace the legacy code relying on it? Mmap and read/write syscalls are both ways to interact with files, but they have different behaviors. You can't exactly swap one for the other without knowledge of the caller. What you likely do see is that OS utilities likely use mmap when it makes sense and a difference. You also have a lot of things that can work on files or pipes/etc and having a common interface is more useful than having more potential performance (sometimes the performance is enough to warrant writing it twice).
- kragen 1y ago
  Multics did default to this behavior, but Unix was written on the PDP-7 and later the PDP-11, neither of which supported virtual memory or paging, so the Unix system call interface necessarily used read() and write() calls instead. This permitted the use of the same system calls on files, on the teletype, on the paper tape reader and punch, on the magtape, on the line printer, and eventually on pipes. Even before pipes, the ability to "record" a program's terminal output in a file or "play back" simulated user input from a file made Unix especially convenient. But pipes, in turn, permitted entire programs on Unix to be used as the building blocks of a new, much more powerful programming language, one where you manipulated not just numbers or strings but potentially endless flows of data, and which could easily orchestrate computations that no single program in the PDP-11's 16-bit address space could manage. And that was how Unix users in the 01970s had an operating system with the entire printed manual available in the comprehensive online help system, a way to produce publication-quality documents on the phototypesetter, incremental recompilation, software version control, full-text search, WYSIWYG screen editors that could immediately jump to the definition of a function, networked email, interactive source-level debugging, a relational database, etc.—all on a 16-bit computer that struggled to run half a million instructions per second, which at most companies might have been relegated to controlling some motors and heaters in a chemical plant or something. It turns out that often what you can do matters even more than how fast you can do it.
- perbu 1y ago
  The point is that invoking the OS has a cost. Using mmap, for those situations where it makes sense, lets you avoid that cost.
nteon 1y ago
the downside is that the go runtime doesn't expect memory reads to page fault, so you may end up with stalls/latency/under-utilization if part of your dataset is paged out (like if you have a large cdb file w/ random access patterns). Using file IO, the Go runtime could be running a different goroutine if there is a disk read, but with mmap that thread is descheduled but holding an m & p. I'm also not sure if there would be increased stop the world pauses, or if the async preemption stuff would "just work". Section 3.2 of this paper has more details: https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf
- vlovich123 1y ago
  To me this indicates a limitation of the API. Cause you do want to maintain that the kernel can page out that memory under pressure while userspace accesses that memory asynchronously while allowing the thread to do other asynchronous things. There’s no good programming model/OS api that can accomplish this today.
  - im3w1l 1y ago
    There are apis that sort of let you do it: mincore, madvise, userfaultfd.
    - bcrl 1y ago
      None of those APIs are cheap enough to call in a fast path.
      - gpderetta 1y ago
        no syscall will be cheap to call in a fast path. You would need an hardware instruction that tells you if a load or store would fault.
        zozbot234 1y ago
        > You would need an hardware instruction that tells you if a load or store would fault. You have MADV_FREE pages/ranges. They get cleared when purged, so reading zeros tells you that the load would have faulted and needs to be populated from storage.
habibur 1y ago
Is mmap still faster than fread? That might have been true in the 90s but I was wondering about current improvements. If you have enough free memory, the file will be cached in memory anyway instead of residing on disk. Therefore both will be reading from memory, albeit through different API. Looking for recent benchmark or view from OS developers.
- stingraycharles 1y ago
  In our experience building a high performance database server: absolutely. If your line of thinking is “if you have enough free memory”, then these types of optimizations aren’t for you. one of the main benefits is eliminating an extra copy. additionally, mmap is heavily optimized for random access, so if that’s what you’re doing, then you’ll have a much better time with it than fread. (I hope a plug is not frowned upon here: if you like this kind of stuff, we’re a fully remote company and hiring C++ devs: https://apply.workable.com/quasar/j/436B0BEE43/ https://apply.workable.com/quasar/j/436B0BEE43/ )
  - YouAreWRONGtoo 1y ago
    [flagged]
    - vlovich123 1y ago
      What’s the false advertising?
      - deaddodo 1y ago
        Yeah, I took a look at the posting and it’s a bog standard job posting. I assume they’re referring to the no-salary aspect and (based on their speech style) are in the US. But, even in that case, it would only matter if the posting were targeted to one of the states that require salary information and the company operated or had a presence in said state. Since it’s an EU company, that’s almost definitely not the case.
buybackoff 1y ago
It looks suspicious at 25x. Even 2.5x would be suspicious unless reading very small records. I assume both cases have the file cached in RAM already fully, with a tiny size of 100MB. But the file read based version actually copies the data into a given buffer, which involves cache misses to get data from RAM to L1 for copying. The mmap version just returns the slice and it's discarded immediately, the actual data is not touched at all. Each record is 2 cache lines and with random indices is not prefetched. For the CPU AMD Ryzen 7 9800X3D mentioned in the repo, just reading 100 bytes from RAM to L1 should take ~100 nanos. The benchmark compares actually getting data vs getting data location. Single digit nanos is the scale of good hash tables lookups with data in CPU caches, not actual IO. For fairness, both should use/touch the data, eg copy it.
- Scaevolus 1y ago
  Yeah, 3.3ns is about 12 CPU cycles. You can indeed create a pointer to a memory location that fast!
- kragen 1y ago
  > For the CPU AMD Ryzen 7 9800X3D mentioned in the repo, just reading 100 bytes from RAM to L1 should take ~100 nanos. I think this is the wrong order of magnitude. One core of my Ryzen 5 3500U seems to be able to run memcpy() at 10 gigabytes per second (0.1 nanoseconds per byte) and memset() at 31 gigabytes per second (0.03 nanoseconds per byte). I'd expect a sequential read of 100 bytes to take about 3 nanoseconds, not 100 nanoseconds. However, I think random accesses do take close to 100 nanoseconds to transmit the starting row and column address and open the row. I haven't measured this on this hardware because I don't have a test I'm confident in.
  - bcrl 1y ago
    100 nanoseconds from RAM is correct. Latency != bandwidth. 3 nanoseconds would be from cache or so on a Ryzen. You ain't gonna get the benefits of prefetching on the first 100 bytes.
    - 1y ago
MayCXC 1y ago
wowie. mmap also dramatically improved perf for LLaMA: https://justine.lol/mmap/ https://justine.lol/mmap/
- kristjansson 1y ago
  uh. there was a bit more to the story than 'yup totally unalloyed free lunch'
mholt 1y ago
Just this month, I've learned the hard way that some file systems do not play well with mmap: https://github.com/mattn/go-sqlite3/issues/1355 https://github.com/mattn/go-sqlite3/issues/1355 In my case, it seems that Mac's ExFAT driver is incompatible with sqlite's WAL mode because the driver returned a memory address that is misaligned on ARM64. Most bizarre error I've encountered in years. So, uh, mind your file systems, kids!
- vlovich123 1y ago
  I would be very careful about that conclusion. Reading that thread it sounds like you’re relying on Claude to make this conclusion but you haven’t actually verified what the address being returned actually is. The reason I’m skeptical is three fold. The first is that it’s generally impossible for a filesystem to mmap return a pointer that’s not page boundary aligned. The second is that unaligned accesses are still fine on modern ARM is not a SIGBUS. The third is that Claude’s reasoning that the pointer must be 8-byte aligned and that indicates a misaligned read is flawed - how do you know that SQLite isn’t doing a 2-byte read at that address? If you really think it’s a bad alignment it should be trivial to reproduce - mmap the file explicitly and print the address or modify the SQLite source to print the mmap location it gets.
  - mholt 1y ago
    I'd love to be wrong, but the address it's referring to is the correct address from the error / stack trace. I honestly don't know anything about this. There's no search results for my error. ChatGPT and Claude and Grok all agreed one way or another, with various prompts. Would be happy to have some help verifying any of this. I just know that disabling WAL mode, and not using Mac's ExFAT driver, both fixed the error reliably.
    - achierius 1y ago
      But is that the address being returned by mmap? Furthermore, what instruction is this crashing on? You should be able to look up the specific alignment requirements of that instruction to verify. > ChatGPT and Claude and Grok all agreed one way or another, with various prompts. This means less than you'd think: they're all trained on a similar corpus, and Grok in particular is probably at least partially distilled from Claude. So they tend to come to similar conclusions given similar data.
commandersaki 1y ago
This is a good article but I'm wondering what is the relationship between this website/company and varnish-cache.org, since in the article they make claims of releasing Varnish Cache, and the article wasn't written by Poul-Henning Kamp.
- wmf 1y ago
  Varnish hasn't been a solo project for many years. Also PHK's version is now called Vinyl Cache while the corporate fork is called Varnish.
  - commandersaki 1y ago
    The article says "when we launched Varnish Cache back in 2006". Who is we? My memory was that around that time PHK released it to the world and was the sole developer at the time.
    - kragen 1y ago
      I was wondering about this too. He apparently worked at the company for a while? Did he found it?
      - perbu 1y ago
        Yes. When Varnish Cache launched, in 2006, I worked in a rather small OSS consultancy, which did the Linux port of Varnish Cache and provided maintenance and funding for the project.
        kragen 1y ago
        You say, "Yes. When Varnish Cache launched, in 2006, I worked in a rather small OSS consultancy, which did the Linux port of Varnish Cache and provided maintenance and funding for the project." But eventually phk left, and you came into conflict with him over the name, which was resolved by him choosing a different name for his version of Varnish?
Ingon 1y ago
When I adopted mmap in klevdb [1], I saw a dramatic performance improvements. So, even as klevdb completes a write segment, it will reopen, on demand, the segment for reading with mmap (segments are basically part of write only log). With this any random reads are super fast (but of course not as fast as sequential ones). [1] https://github.com/klev-dev/klevdb https://github.com/klev-dev/klevdb
gustavpaul 1y ago
The MmapReader is not copying the requested byte range into the buf argument, so if ever the underlying file descriptor is closed (or the file truncated out of band) any subsequent slice access will throw SIGBUS, which is really unpleasant. It also means the latency due to pagefaults is shifted from inside mmapReader.ReadRecord() (where it would be expected) to wherever in the application the bytes are first accessed, leading to spooky unpreditactable latency spikes in what are otherwise pure functions. That inevitably leads to wild arguments about how bad GC stalls are :-) An apples to apples comparison should be copying the bytes from the mmap buffer and returning the resulting slice.
- dapperdrake 1y ago
  It’s not accessible until it is in user space. (Virtual memory addresses mapped to physical RAM holding the data.) Good point.
- loeg 1y ago
  > so if ever the underlying file descriptor is closed Nit: Mmap mapping lifetimes are not attached to the underlying fd. The file truncation and latency concerns are valid, though.
- dahfizz 1y ago
  Being able to avoid an extra copy is actually a huge performance gain when you can safely do it. You shouldn't discount how useful mmap is just because its not useful in every scenario. You shouldn't replace every single file access with mmap. But when it makes sense, mmap is a big performance win.
liuliu 1y ago
mmap is a good crutch when you 1. don't have busy polling / async IO API available and want to do some quick & dirty preloading tricks; 2. don't want to manage the complexity of in-memory cache, especially cross-processes ones. Obviously if you have kernel-backed async IO APIs (io_uring) and willing to dig into the deeper end (for better managed cache), you can get better performance than mmap. But in many cases, mmap is "good-enough".
kragen 1y ago
The simple answer to "How do memory maps (mmap) deliver faster file access?" is "sometimes", but the blog post does give some more details. I was suspicious of the 25× speedup claim, but it's a lot more plausible than I thought. On this Ryzen 5 3500U running mostly at 3.667GHz (poorly controlled), reading data from an already-memory-mapped page is as fast as memcpy (about 10 gigabytes per second when not cached on one core of my laptop, which works out to 0.1 nanoseconds per byte, plus about 20 nanoseconds of overhead) while lseek+read is two system calls (590ns each) plus copying bytes into userspace (26–30ps per byte for small calls, 120ps per byte for a few megabytes). Small memcpy (from, as it happens, an mmapped page) also costs about 25ps per byte, plus about 2800ps per loop iteration, probably much of which is incrementing the loop counter and passing arguments to the memcpy function (GCC is emitting an actual call to memcpy, via the PLT). So mmap will always be faster than lseek+read on this machine, at least if it doesn't have a page fault, but the point at which memcpy from mmap would be 25× faster than lseek+read would be where 2×590 + .028n = 25×(2.8 + .025n) = 70 + .625n. Which is to say 1110 = .597n ∴ n = 1110/.597 = 1859 bytes. At that point, memcpy from mmap should be 49ns and lseek+read should be 1232ns, which is 25× as big. You can cut that size more than in half if you use pread() instead of lseek+read, and presumably io_uring would cut it even more. If we assume that we're also taking cache misses to bring in the data from main memory in both cases, we have 2×590 + .1n = 25×(2.8 + .1n) = 70 + 2.5n, so 1110 = 2.4n ∴ n = 1110/2.4 = 462 bytes. On the other hand, mmap will be slow if it's hitting a page fault, which sort of corresponds to the case where you could have cached the result of lseek+read in private RAM, which you could do on a smaller-than-pagesize granularity, which potentially means you could hit the slow path much less often for a given working set. And lseek+read has several possible ways to do make the I/O asynchronous, while the only way to make mmap page faults asynchronous is to hit the page faults in different threads, which is a pretty heavyweight mechanism. On the other hand, lseek+read with a software cache is sort of using twice as much memory (one copy is in the kernel's buffer cache and another copy is in the application's software cache) so mmap could still win. And, if there are other processes writing to the data being queried, you need some way to invalidate the software cache, which can be expensive. (On the gripping hand, if you're reading from shared memory while other processes are updating it, you're probably going to need some kind of locking or lock-free synchronization with those other processes.) So I think a reasonably architected lseek+read (or pread) approach to the problem might be a little faster or a little slower than the mmap approach, but the gap definitely won't be 25×. But very simple applications or libraries, or libraries where many processes might be simultaneously accessing the same data, could indeed get 25× or even 256× performance improvements by letting the kernel manage the cache instead of trying to do it themselves. Someone at a large user of Varnish told me they've mostly removed mmap from their Varnish fork for performance.
Animats 1y ago
I never knew that Linux memory mapped files were copy-on-write. I'd assumed they let you alter the page and wrote out dirty pages later.
- pengaru 1y ago
  MAP_PRIVATE vs. MAP_SHARED
charlietap 1y ago
This article is nonsensical. If you're reading this please don't start mmap'ing files just to read from them. It proposes an incredibly unrealistic scenario where the program is making thousands of random incredibly small unbuffered reads from a file. In reality 99 percent of programs will sequentially reading bytes into a buffer which makes orders of magnitude less syscalls. Mmap is useful in niche scenarios, it's not magic.
- karel-3d 1y ago
  That is not unrealistic if you are using the file to save binary data on given positions and don't need to read all data. For example if you have a big matrix of fixed size structs and you need to read only some of them.
- perbu 1y ago
  This is a niche scenario. The scenario outlined is reading CDB databases.
- icedchai 1y ago
  At a previous company, we had a custom "database" (I use that term very loosely) built on memory mapped files. All startup, all pages were read to ensure the data was hot, unlikely to be any page faults. It worked well for the application, but obviously because the whole thing fit in memory and was preloaded. We also had our own custom write-ahead-log. Today, I'd probably use sqlite.
philippta 1y ago
At computerenhance.com[0] Casey Muratori shows that memory mapped files actually perform worse at sequential reads, which is the common case for file access. That’s because the CPU won’t prefetch data as effectively and has to rely on page faults to know what to read next. With regular, sequential file reads, the CPU can be much smarter and prefetch the next page while the program is consuming the previous one. [0] https://www.computerenhance.com/p/memory-mapped-files https://www.computerenhance.com/p/memory-mapped-files
- vlovich123 1y ago
  Io_uring should be outperforming both - you can configure the read ahead optimally, there’s no page faults, and there’s no copies as there is with buffered I/O: [1] https://archive.is/vkdCo https://archive.is/vkdCo
- atombender 1y ago
  Does madvise(..., MADV_SEQUENTIAL) not help here?
gethly 1y ago
I have never used mmap, as I had no need, but I know BoltDB uses it and from what I remember, the mmap is good for when you are working with whole disk pages, which BoltDB does. Otherwise it seems to be wrong use case for it?
karel-3d 1y ago
mmap is fine when you know the file fits in memory, and you need random file reads/writes of only some parts of the file. It's not magic. It's also quite hard to debug in go, because mmaped files are not visible in pprof; whe you run out of memory, mmap starts behaving really suboptimally. And it's hard to see which file takes how much memory (again it doesn't show in pprof).
- perbu 1y ago
  random reads are ok. writes through a mmap are a disaster.
  - vlowther 1y ago
    Only if you are doing in-place updates. If append-only datastores are your jam, writes via mmap are Just Fine: $ go test -v === RUN TestChunkOps chunk_test.go:26: Checking basic persistence and Store expansion. chunk_test.go:74: Checking close and reopen read-only chunk_test.go:106: Checking that readonly blocks write ops chunk_test.go:116: Checking Clear chunk_test.go:175: Checking interrupted write --- PASS: TestChunkOps (0.06s) === RUN TestEncWriteSpeed chunk_test.go:246: Wrote 1443 MB/s chunk_test.go:264: Read 5525.418751 MB/s --- PASS: TestEncWriteSpeed (1.42s) === RUN TestPlaintextWriteSpeed chunk_test.go:301: Wrote 1693 MB/s chunk_test.go:319: Read 10528.744206 MB/s --- PASS: TestPlaintextWriteSpeed (1.36s) PASS
  - karel-3d 1y ago
    Interesting. We use it to mmap a big file that just-enough fits in memory and we mostly read randomly (around 1/1000 of the file) and sometimes sparringly write and it works great. I haven't tested how would it work more fast with Seek/Read/Write, but mmap is very ergonomic for this; it just acts as a slice and it's basically invisible; and you can then take the slice of bytes and unsafe.cast it as slice of something else. Very easy.
benjiro 1y ago
People are so focused on the mmap part, and the latency, that the usage is overlooked. > The last couple of weeks I've been working on an HTTP-backed filesystem. It feels like this is micro optimizations, that are going to get blocked anyway by the whole HTTP cycle anyway. There is also the benchmark issue: The enhanced CDB format seems to be focused on a read only benefits, as writes introduced a lot of latency, and issue with mmap. In other words, there is a need to freeze for the mmap, then unfreeze, write for updates, freeze for mmap ... This cycle introduces overhead, does it not? Has this been benchmarked? Because from what i am seeing, the benefits are mostly in the frozen state (aka read only). If the data is changed infrequently, why not just use json? No matter how slow it is, if your just going to do http requests for the directory listing, your overhead is not the actual file format. If this enhanced file format was used as file storage, and you want to be able to fast read files, that is a different matter. Then there are ways around it with keeping "part" files where files 1 ... 1000 are in file.01, 2 ... 2000 in file.02 (thus reducing overhead from the file system). And those are memory mapped for fast reading. And where updates are invalidated files/rewrites (as i do not see any delete/vacume ability in the file format). So, the actual benefits just for a file directory listing db escapes me.
- perbu 1y ago
  We need to support over 10M files in each folder. JSON wouldn't fare well as the lack of indices makes random access problematic. Composing a JSON file with many objects is, at least with the current JSON implementation, not feasible. CDB is only a transport medium. The data originates in PostgreSQL and upon request, stored in CDB and transferred. Writing/freezing to CDB is faster than encoding JSON. CDB also makes it possible to access it directly, with ranged HTTP requests. It isn't something I've implemented, but having the option to do so is nice.
  - benjiro 1y ago
    > CDB is only a transport medium. The data originates in PostgreSQL and upon request, stored in CDB and transferred. Writing/freezing to CDB is faster than encoding JSON. Might have been interesting to actually include this in the article, do you not think so? ;-) The way the article is written, made it seen that you used cdb on edge nodes to store metadata. With no information as to what your storing / access, how, why ... This is part of the reason we have these discussions here.
lzaf 1y ago
On a similar vein some time ago I had written a small toy lib that emulates the os.File interface for nmap backed files: https://github.com/zaf/yammap https://github.com/zaf/yammap, It even handled bus errors without panicking: https://github.com/zaf/yammap/blob/186f714343906bb9304ad5f30b19a7ad265a9686/yammap.go#L367 https://github.com/zaf/yammap/blob/186f714343906bb9304ad5f30... Read and write performance was usually better especially with larger write sizes. Compared to os.File: ~/src/yammap$ go test -benchtime=4s -bench . goos: linux goarch: amd64 pkg: github.com/zaf/yammap cpu: AMD Ryzen 9 5900X 12-Core Processor BenchmarkWrite-24 29085 164744 ns/op 25459.52 MB/s BenchmarkOSWrite-24 22204 215131 ns/op 19496.54 MB/s BenchmarkRead-24 29113 166820 ns/op 25142.72 MB/s BenchmarkOSRead-24 27451 172685 ns/op 24288.69 MB/s

141 points by ingve 1y ago

Sounds interesting. Why wouldn’t the OS itself default to this behavior? Could it fall apart under load, or is it just not important enough to replace the legacy code relying on it?

trenchpilgrim 1y ago
1. mmap was added to Unix later by Sun, it wasn't in the original Unix 2. As the article points out mmap is very fast for reading huge amounts of data but is a lot slower at other file operations. For reading smallish files, which is the majority of calls most software will make to the filesystem, the regular file syscalls are better. 3. If you're on a modern Linux you might be better off with io_uring than mmap.
- scottlamb 1y ago
  All true, and it's not just performance either. The API is just different. mmap data can change at any time. In fact, if the file shrinks, access to a formerly valid region of memory has behavior that is unspecified by the Single Unix Specification. (On Linux, it causes a SIGBUS if you access a page that is entirely invalid; bytes within the last page after the last valid byte probably are zeros or something? unsure.) In theory I suppose you could have a libc that mostly emulates read() and write() calls on files [1] with memcpy() on mmap()ed regions. But I don't think it'd be quite right. For one thing, that read() behavior after shrink would be a source of error. Higher-level APIs might be more free to do things with either mmap or read/write. [1] just on files; so it'd have to track which file descriptors are files as opposed to sockets/pipes/etc, maintaining the cached lengths and mmap()ed regions and such. libc doesn't normally do that, and it'd go badly if you bypass it with direct system calls.
  - nawgz 1y ago
    Interesting callouts! Thanks
- deleted 1y ago
  [deleted]
toast0 1y ago
> Why wouldn’t the OS itself default to this behavior? Could it fall apart under load, or is it just not important enough to replace the legacy code relying on it? Mmap and read/write syscalls are both ways to interact with files, but they have different behaviors. You can't exactly swap one for the other without knowledge of the caller. What you likely do see is that OS utilities likely use mmap when it makes sense and a difference. You also have a lot of things that can work on files or pipes/etc and having a common interface is more useful than having more potential performance (sometimes the performance is enough to warrant writing it twice).
kragen 1y ago
Multics did default to this behavior, but Unix was written on the PDP-7 and later the PDP-11, neither of which supported virtual memory or paging, so the Unix system call interface necessarily used read() and write() calls instead. This permitted the use of the same system calls on files, on the teletype, on the paper tape reader and punch, on the magtape, on the line printer, and eventually on pipes. Even before pipes, the ability to "record" a program's terminal output in a file or "play back" simulated user input from a file made Unix especially convenient. But pipes, in turn, permitted entire programs on Unix to be used as the building blocks of a new, much more powerful programming language, one where you manipulated not just numbers or strings but potentially endless flows of data, and which could easily orchestrate computations that no single program in the PDP-11's 16-bit address space could manage. And that was how Unix users in the 01970s had an operating system with the entire printed manual available in the comprehensive online help system, a way to produce publication-quality documents on the phototypesetter, incremental recompilation, software version control, full-text search, WYSIWYG screen editors that could immediately jump to the definition of a function, networked email, interactive source-level debugging, a relational database, etc.—all on a 16-bit computer that struggled to run half a million instructions per second, which at most companies might have been relegated to controlling some motors and heaters in a chemical plant or something. It turns out that often what you can do matters even more than how fast you can do it.
perbu 1y ago
The point is that invoking the OS has a cost. Using mmap, for those situations where it makes sense, lets you avoid that cost.

nteon 1y ago

the downside is that the go runtime doesn't expect memory reads to page fault, so you may end up with stalls/latency/under-utilization if part of your dataset is paged out (like if you have a large cdb file w/ random access patterns). Using file IO, the Go runtime could be running a different goroutine if there is a disk read, but with mmap that thread is descheduled but holding an m & p. I'm also not sure if there would be increased stop the world pauses, or if the async preemption stuff would "just work". Section 3.2 of this paper has more details: https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf

vlovich123 1y ago
To me this indicates a limitation of the API. Cause you do want to maintain that the kernel can page out that memory under pressure while userspace accesses that memory asynchronously while allowing the thread to do other asynchronous things. There’s no good programming model/OS api that can accomplish this today.
- im3w1l 1y ago
  There are apis that sort of let you do it: mincore, madvise, userfaultfd.
  - bcrl 1y ago
    None of those APIs are cheap enough to call in a fast path.
    - gpderetta 1y ago
      no syscall will be cheap to call in a fast path. You would need an hardware instruction that tells you if a load or store would fault.
      - zozbot234 1y ago
        > You would need an hardware instruction that tells you if a load or store would fault. You have MADV_FREE pages/ranges. They get cleared when purged, so reading zeros tells you that the load would have faulted and needs to be populated from storage.

habibur 1y ago

Is mmap still faster than fread? That might have been true in the 90s but I was wondering about current improvements. If you have enough free memory, the file will be cached in memory anyway instead of residing on disk. Therefore both will be reading from memory, albeit through different API. Looking for recent benchmark or view from OS developers.

stingraycharles 1y ago
In our experience building a high performance database server: absolutely. If your line of thinking is “if you have enough free memory”, then these types of optimizations aren’t for you. one of the main benefits is eliminating an extra copy. additionally, mmap is heavily optimized for random access, so if that’s what you’re doing, then you’ll have a much better time with it than fread. (I hope a plug is not frowned upon here: if you like this kind of stuff, we’re a fully remote company and hiring C++ devs: https://apply.workable.com/quasar/j/436B0BEE43/ https://apply.workable.com/quasar/j/436B0BEE43/ )
- YouAreWRONGtoo 1y ago
  [flagged]
  - vlovich123 1y ago
    What’s the false advertising?
    - deaddodo 1y ago
      Yeah, I took a look at the posting and it’s a bog standard job posting. I assume they’re referring to the no-salary aspect and (based on their speech style) are in the US. But, even in that case, it would only matter if the posting were targeted to one of the states that require salary information and the company operated or had a presence in said state. Since it’s an EU company, that’s almost definitely not the case.

buybackoff 1y ago

It looks suspicious at 25x. Even 2.5x would be suspicious unless reading very small records. I assume both cases have the file cached in RAM already fully, with a tiny size of 100MB. But the file read based version actually copies the data into a given buffer, which involves cache misses to get data from RAM to L1 for copying. The mmap version just returns the slice and it's discarded immediately, the actual data is not touched at all. Each record is 2 cache lines and with random indices is not prefetched. For the CPU AMD Ryzen 7 9800X3D mentioned in the repo, just reading 100 bytes from RAM to L1 should take ~100 nanos. The benchmark compares actually getting data vs getting data location. Single digit nanos is the scale of good hash tables lookups with data in CPU caches, not actual IO. For fairness, both should use/touch the data, eg copy it.

Scaevolus 1y ago
Yeah, 3.3ns is about 12 CPU cycles. You can indeed create a pointer to a memory location that fast!
kragen 1y ago
> For the CPU AMD Ryzen 7 9800X3D mentioned in the repo, just reading 100 bytes from RAM to L1 should take ~100 nanos. I think this is the wrong order of magnitude. One core of my Ryzen 5 3500U seems to be able to run memcpy() at 10 gigabytes per second (0.1 nanoseconds per byte) and memset() at 31 gigabytes per second (0.03 nanoseconds per byte). I'd expect a sequential read of 100 bytes to take about 3 nanoseconds, not 100 nanoseconds. However, I think random accesses do take close to 100 nanoseconds to transmit the starting row and column address and open the row. I haven't measured this on this hardware because I don't have a test I'm confident in.
- bcrl 1y ago
  100 nanoseconds from RAM is correct. Latency != bandwidth. 3 nanoseconds would be from cache or so on a Ryzen. You ain't gonna get the benefits of prefetching on the first 100 bytes.
  - 1y ago

MayCXC 1y ago

wowie. mmap also dramatically improved perf for LLaMA: https://justine.lol/mmap/ https://justine.lol/mmap/

kristjansson 1y ago
uh. there was a bit more to the story than 'yup totally unalloyed free lunch'

mholt 1y ago

Just this month, I've learned the hard way that some file systems do not play well with mmap: https://github.com/mattn/go-sqlite3/issues/1355 https://github.com/mattn/go-sqlite3/issues/1355 In my case, it seems that Mac's ExFAT driver is incompatible with sqlite's WAL mode because the driver returned a memory address that is misaligned on ARM64. Most bizarre error I've encountered in years. So, uh, mind your file systems, kids!

vlovich123 1y ago
I would be very careful about that conclusion. Reading that thread it sounds like you’re relying on Claude to make this conclusion but you haven’t actually verified what the address being returned actually is. The reason I’m skeptical is three fold. The first is that it’s generally impossible for a filesystem to mmap return a pointer that’s not page boundary aligned. The second is that unaligned accesses are still fine on modern ARM is not a SIGBUS. The third is that Claude’s reasoning that the pointer must be 8-byte aligned and that indicates a misaligned read is flawed - how do you know that SQLite isn’t doing a 2-byte read at that address? If you really think it’s a bad alignment it should be trivial to reproduce - mmap the file explicitly and print the address or modify the SQLite source to print the mmap location it gets.
- mholt 1y ago
  I'd love to be wrong, but the address it's referring to is the correct address from the error / stack trace. I honestly don't know anything about this. There's no search results for my error. ChatGPT and Claude and Grok all agreed one way or another, with various prompts. Would be happy to have some help verifying any of this. I just know that disabling WAL mode, and not using Mac's ExFAT driver, both fixed the error reliably.
  - achierius 1y ago
    But is that the address being returned by mmap? Furthermore, what instruction is this crashing on? You should be able to look up the specific alignment requirements of that instruction to verify. > ChatGPT and Claude and Grok all agreed one way or another, with various prompts. This means less than you'd think: they're all trained on a similar corpus, and Grok in particular is probably at least partially distilled from Claude. So they tend to come to similar conclusions given similar data.

commandersaki 1y ago

This is a good article but I'm wondering what is the relationship between this website/company and varnish-cache.org, since in the article they make claims of releasing Varnish Cache, and the article wasn't written by Poul-Henning Kamp.

wmf 1y ago
Varnish hasn't been a solo project for many years. Also PHK's version is now called Vinyl Cache while the corporate fork is called Varnish.
- commandersaki 1y ago
  The article says "when we launched Varnish Cache back in 2006". Who is we? My memory was that around that time PHK released it to the world and was the sole developer at the time.
  - kragen 1y ago
    I was wondering about this too. He apparently worked at the company for a while? Did he found it?
    - perbu 1y ago
      Yes. When Varnish Cache launched, in 2006, I worked in a rather small OSS consultancy, which did the Linux port of Varnish Cache and provided maintenance and funding for the project.
      - kragen 1y ago
        You say, "Yes. When Varnish Cache launched, in 2006, I worked in a rather small OSS consultancy, which did the Linux port of Varnish Cache and provided maintenance and funding for the project." But eventually phk left, and you came into conflict with him over the name, which was resolved by him choosing a different name for his version of Varnish?

Ingon 1y ago

When I adopted mmap in klevdb [1], I saw a dramatic performance improvements. So, even as klevdb completes a write segment, it will reopen, on demand, the segment for reading with mmap (segments are basically part of write only log). With this any random reads are super fast (but of course not as fast as sequential ones). [1] https://github.com/klev-dev/klevdb https://github.com/klev-dev/klevdb

gustavpaul 1y ago

The MmapReader is not copying the requested byte range into the buf argument, so if ever the underlying file descriptor is closed (or the file truncated out of band) any subsequent slice access will throw SIGBUS, which is really unpleasant. It also means the latency due to pagefaults is shifted from inside mmapReader.ReadRecord() (where it would be expected) to wherever in the application the bytes are first accessed, leading to spooky unpreditactable latency spikes in what are otherwise pure functions. That inevitably leads to wild arguments about how bad GC stalls are :-) An apples to apples comparison should be copying the bytes from the mmap buffer and returning the resulting slice.

dapperdrake 1y ago
It’s not accessible until it is in user space. (Virtual memory addresses mapped to physical RAM holding the data.) Good point.
loeg 1y ago
> so if ever the underlying file descriptor is closed Nit: Mmap mapping lifetimes are not attached to the underlying fd. The file truncation and latency concerns are valid, though.
dahfizz 1y ago
Being able to avoid an extra copy is actually a huge performance gain when you can safely do it. You shouldn't discount how useful mmap is just because its not useful in every scenario. You shouldn't replace every single file access with mmap. But when it makes sense, mmap is a big performance win.

liuliu 1y ago

mmap is a good crutch when you 1. don't have busy polling / async IO API available and want to do some quick & dirty preloading tricks; 2. don't want to manage the complexity of in-memory cache, especially cross-processes ones. Obviously if you have kernel-backed async IO APIs (io_uring) and willing to dig into the deeper end (for better managed cache), you can get better performance than mmap. But in many cases, mmap is "good-enough".

kragen 1y ago

The simple answer to "How do memory maps (mmap) deliver faster file access?" is "sometimes", but the blog post does give some more details. I was suspicious of the 25× speedup claim, but it's a lot more plausible than I thought. On this Ryzen 5 3500U running mostly at 3.667GHz (poorly controlled), reading data from an already-memory-mapped page is as fast as memcpy (about 10 gigabytes per second when not cached on one core of my laptop, which works out to 0.1 nanoseconds per byte, plus about 20 nanoseconds of overhead) while lseek+read is two system calls (590ns each) plus copying bytes into userspace (26–30ps per byte for small calls, 120ps per byte for a few megabytes). Small memcpy (from, as it happens, an mmapped page) also costs about 25ps per byte, plus about 2800ps per loop iteration, probably much of which is incrementing the loop counter and passing arguments to the memcpy function (GCC is emitting an actual call to memcpy, via the PLT). So mmap will always be faster than lseek+read on this machine, at least if it doesn't have a page fault, but the point at which memcpy from mmap would be 25× faster than lseek+read would be where 2×590 + .028n = 25×(2.8 + .025n) = 70 + .625n. Which is to say 1110 = .597n ∴ n = 1110/.597 = 1859 bytes. At that point, memcpy from mmap should be 49ns and lseek+read should be 1232ns, which is 25× as big. You can cut that size more than in half if you use pread() instead of lseek+read, and presumably io_uring would cut it even more. If we assume that we're also taking cache misses to bring in the data from main memory in both cases, we have 2×590 + .1n = 25×(2.8 + .1n) = 70 + 2.5n, so 1110 = 2.4n ∴ n = 1110/2.4 = 462 bytes. On the other hand, mmap will be slow if it's hitting a page fault, which sort of corresponds to the case where you could have cached the result of lseek+read in private RAM, which you could do on a smaller-than-pagesize granularity, which potentially means you could hit the slow path much less often for a given working set. And lseek+read has several possible ways to do make the I/O asynchronous, while the only way to make mmap page faults asynchronous is to hit the page faults in different threads, which is a pretty heavyweight mechanism. On the other hand, lseek+read with a software cache is sort of using twice as much memory (one copy is in the kernel's buffer cache and another copy is in the application's software cache) so mmap could still win. And, if there are other processes writing to the data being queried, you need some way to invalidate the software cache, which can be expensive. (On the gripping hand, if you're reading from shared memory while other processes are updating it, you're probably going to need some kind of locking or lock-free synchronization with those other processes.) So I think a reasonably architected lseek+read (or pread) approach to the problem might be a little faster or a little slower than the mmap approach, but the gap definitely won't be 25×. But very simple applications or libraries, or libraries where many processes might be simultaneously accessing the same data, could indeed get 25× or even 256× performance improvements by letting the kernel manage the cache instead of trying to do it themselves. Someone at a large user of Varnish told me they've mostly removed mmap from their Varnish fork for performance.

Animats 1y ago

I never knew that Linux memory mapped files were copy-on-write. I'd assumed they let you alter the page and wrote out dirty pages later.

pengaru 1y ago
MAP_PRIVATE vs. MAP_SHARED

charlietap 1y ago

This article is nonsensical. If you're reading this please don't start mmap'ing files just to read from them. It proposes an incredibly unrealistic scenario where the program is making thousands of random incredibly small unbuffered reads from a file. In reality 99 percent of programs will sequentially reading bytes into a buffer which makes orders of magnitude less syscalls. Mmap is useful in niche scenarios, it's not magic.

karel-3d 1y ago
That is not unrealistic if you are using the file to save binary data on given positions and don't need to read all data. For example if you have a big matrix of fixed size structs and you need to read only some of them.
perbu 1y ago
This is a niche scenario. The scenario outlined is reading CDB databases.
icedchai 1y ago
At a previous company, we had a custom "database" (I use that term very loosely) built on memory mapped files. All startup, all pages were read to ensure the data was hot, unlikely to be any page faults. It worked well for the application, but obviously because the whole thing fit in memory and was preloaded. We also had our own custom write-ahead-log. Today, I'd probably use sqlite.

philippta 1y ago

At computerenhance.com[0] Casey Muratori shows that memory mapped files actually perform worse at sequential reads, which is the common case for file access. That’s because the CPU won’t prefetch data as effectively and has to rely on page faults to know what to read next. With regular, sequential file reads, the CPU can be much smarter and prefetch the next page while the program is consuming the previous one. [0] https://www.computerenhance.com/p/memory-mapped-files https://www.computerenhance.com/p/memory-mapped-files

vlovich123 1y ago
Io_uring should be outperforming both - you can configure the read ahead optimally, there’s no page faults, and there’s no copies as there is with buffered I/O: [1] https://archive.is/vkdCo https://archive.is/vkdCo
atombender 1y ago
Does madvise(..., MADV_SEQUENTIAL) not help here?

gethly 1y ago

I have never used mmap, as I had no need, but I know BoltDB uses it and from what I remember, the mmap is good for when you are working with whole disk pages, which BoltDB does. Otherwise it seems to be wrong use case for it?

karel-3d 1y ago

mmap is fine when you know the file fits in memory, and you need random file reads/writes of only some parts of the file. It's not magic. It's also quite hard to debug in go, because mmaped files are not visible in pprof; whe you run out of memory, mmap starts behaving really suboptimally. And it's hard to see which file takes how much memory (again it doesn't show in pprof).

perbu 1y ago
random reads are ok. writes through a mmap are a disaster.
- vlowther 1y ago
  Only if you are doing in-place updates. If append-only datastores are your jam, writes via mmap are Just Fine: $ go test -v === RUN TestChunkOps chunk_test.go:26: Checking basic persistence and Store expansion. chunk_test.go:74: Checking close and reopen read-only chunk_test.go:106: Checking that readonly blocks write ops chunk_test.go:116: Checking Clear chunk_test.go:175: Checking interrupted write --- PASS: TestChunkOps (0.06s) === RUN TestEncWriteSpeed chunk_test.go:246: Wrote 1443 MB/s chunk_test.go:264: Read 5525.418751 MB/s --- PASS: TestEncWriteSpeed (1.42s) === RUN TestPlaintextWriteSpeed chunk_test.go:301: Wrote 1693 MB/s chunk_test.go:319: Read 10528.744206 MB/s --- PASS: TestPlaintextWriteSpeed (1.36s) PASS
- karel-3d 1y ago
  Interesting. We use it to mmap a big file that just-enough fits in memory and we mostly read randomly (around 1/1000 of the file) and sometimes sparringly write and it works great. I haven't tested how would it work more fast with Seek/Read/Write, but mmap is very ergonomic for this; it just acts as a slice and it's basically invisible; and you can then take the slice of bytes and unsafe.cast it as slice of something else. Very easy.

benjiro 1y ago

People are so focused on the mmap part, and the latency, that the usage is overlooked. > The last couple of weeks I've been working on an HTTP-backed filesystem. It feels like this is micro optimizations, that are going to get blocked anyway by the whole HTTP cycle anyway. There is also the benchmark issue: The enhanced CDB format seems to be focused on a read only benefits, as writes introduced a lot of latency, and issue with mmap. In other words, there is a need to freeze for the mmap, then unfreeze, write for updates, freeze for mmap ... This cycle introduces overhead, does it not? Has this been benchmarked? Because from what i am seeing, the benefits are mostly in the frozen state (aka read only). If the data is changed infrequently, why not just use json? No matter how slow it is, if your just going to do http requests for the directory listing, your overhead is not the actual file format. If this enhanced file format was used as file storage, and you want to be able to fast read files, that is a different matter. Then there are ways around it with keeping "part" files where files 1 ... 1000 are in file.01, 2 ... 2000 in file.02 (thus reducing overhead from the file system). And those are memory mapped for fast reading. And where updates are invalidated files/rewrites (as i do not see any delete/vacume ability in the file format). So, the actual benefits just for a file directory listing db escapes me.

perbu 1y ago
We need to support over 10M files in each folder. JSON wouldn't fare well as the lack of indices makes random access problematic. Composing a JSON file with many objects is, at least with the current JSON implementation, not feasible. CDB is only a transport medium. The data originates in PostgreSQL and upon request, stored in CDB and transferred. Writing/freezing to CDB is faster than encoding JSON. CDB also makes it possible to access it directly, with ranged HTTP requests. It isn't something I've implemented, but having the option to do so is nice.
- benjiro 1y ago
  > CDB is only a transport medium. The data originates in PostgreSQL and upon request, stored in CDB and transferred. Writing/freezing to CDB is faster than encoding JSON. Might have been interesting to actually include this in the article, do you not think so? ;-) The way the article is written, made it seen that you used cdb on edge nodes to store metadata. With no information as to what your storing / access, how, why ... This is part of the reason we have these discussions here.

lzaf 1y ago

On a similar vein some time ago I had written a small toy lib that emulates the os.File interface for nmap backed files: https://github.com/zaf/yammap https://github.com/zaf/yammap, It even handled bus errors without panicking: https://github.com/zaf/yammap/blob/186f714343906bb9304ad5f30b19a7ad265a9686/yammap.go#L367 https://github.com/zaf/yammap/blob/186f714343906bb9304ad5f30... Read and write performance was usually better especially with larger write sizes. Compared to os.File: ~/src/yammap$ go test -benchtime=4s -bench . goos: linux goarch: amd64 pkg: github.com/zaf/yammap cpu: AMD Ryzen 9 5900X 12-Core Processor BenchmarkWrite-24 29085 164744 ns/op 25459.52 MB/s BenchmarkOSWrite-24 22204 215131 ns/op 19496.54 MB/s BenchmarkRead-24 29113 166820 ns/op 25142.72 MB/s BenchmarkOSRead-24 27451 172685 ns/op 24288.69 MB/s