4 ms·
The problem with using mmap-based IO is you generally depend on the kernel's async readahead/readaround to keep things fast while writing blocking-style, obviou
by pengaru 3y ago
The problem with using mmap-based IO is you generally depend on the kernel's async readahead/readaround to keep things fast while writing blocking-style, obvious procedural style code, but it's totally naive, and sequential relative to the faulted addresses, much like a spinning disk prefers to be read.
That worked well when the storage system was made of spinning disks, and you were incentivized to use data structures laying data out appropriate for that access pattern. It was all complementary and cooperated pretty well.
Then SSDs became commonplace.
Today you want to exploit their fast random access, utilizing whatever data structures make the most sense in a world without seek times. Introducing mmap() into that stack kind of reintroduces spinning disk style performance characteristics, if you're relying on mmap()'s async readahead/readaround magic to keep your accesses from blocking on IO.
I'm not saying mmap() was ever a great choice for a database. But it's really awful for random access patterns until everything's hot in the page cache.
This happens to be a large part of why systemd-journald's journals can be slow to process w/`journalctl -u` or `systemctl status $service` type queries when cold. The access patterns are very random combined with tiny objects being read, while the kernel goes off prefetching heaps of ~unrelated nearby pages in the hopes of preventing blocking subsequent accesses.
- jnwatson 3y agoReadahead is configurable via madvise, and it is configurable in LMDB as well.
- jandrewrogers 3y agoWith the caveat that I have not looked at this in over a decade, readahead via madvise() is imprecise to the point of near uselessness for high-throughput database workloads. The biggest issue is that, as the name implies, Linux is free to ignore madvise() or do something that is not quite what you requested, and often does. Many types of finely choreographed locality optimizations require more determinism than it offers.
- hyc_symas 3y ago> That worked well when the storage system was made of spinning disks, and you were incentivized to use data structures laying data out appropriate for that access pattern. It was all complementary and cooperated pretty well. > Then SSDs became commonplace. You are laughably wrong. Random I/O is random I/O whether you do it using lseek/read or whether you access a page offset in a mmap. And we've got hundreds of hours of benchmarks that show that LMDB outperforms everything else, whether on spinning rust or solid state. http://www.lmdb.tech/bench/ http://www.lmdb.tech/bench/
- pengaru 3y agoMy comment isn't really targeted at LMDB specifically, it's more generally regarding mmap(). If you design the data format to work well on a spinning disk, it should work well with an mmap based access. That's what my comment implies, and it's completely consistent with what you're saying regarding the benchmarks.