6 ms·

Are you sure you want to use MMAP in your DBMS?

98 points by mfiguiere 3y ago

beoberha 3y ago
I really wish HYC was more articulate and less egomaniacal. He makes a lot of claims and doesn’t really flesh out great arguments. Would be really interesting to have him discuss with Andy Pavlo :)
- cmrdporcupine 3y ago
  Yeah the strident and somewhat impolite tone is really hard to tolerate. Esp when directed against respected researchers like Pavlo and Leis who, well, they know what they're talking about. Everything has compromises and trade-offs. LMDB looks great. But it's also, like a lot of similar systems, built to perform best at single-writer / multiple-reader workloads. That's not the only workload out there.
  - hyc_symas 3y ago
    If they knew what they were talking about they wouldn't have used a raw filesystem benchmarking tool to prove their arguments. They would have used a pair of actual, comparable DBMS implementations. Single-writer is all you need if each write is fast enough. BerkeleyDB, which we used before writing LMDB, is multi-writer. It's still slower than LMDB in every workload, because of all of the lock management that comes with being multi-writer.
- hyc_symas 3y ago
  Andy and I have had this conversation many times over. https://www.youtube.com/watch?v=tEa5sAh-kVk https://www.youtube.com/watch?v=tEa5sAh-kVk Feel free to point out specific arguments that you think need further fleshing out. Every aspect of LMDB's design and performance characteristics have been documented already, happy to link you to answers to any of your questions.
vlovich123 3y ago
Author is right that a read only mmap with writing occurring normally is a better design. It’s a weakness in the paper but in fairness few databases did what LMDB did sanely. Surprising that that’s the case but that is what people chose to do. Where they’re wrong is about the performance analysis: > In section 3.2 "I/O stalls" is again a non-issue; no matter how your DBMS handles I/O internally, synchronously or asynchronously, the calling application can't make any progress until the I/O completes, and if the data isn't already in memory then the application must wait. A TLB shoot down request to pull in a page or evict it can cause pauses not just in your own application but also many others because it’s basically a system wide mutex. There is a trade off obviously where the kernel can evict I/O buffers more efficiently. But to me all that says is that the interface for managing I/O buffers in user space is really immature and needs some better performing API so that we have something in between mmap and buffer management. Also an asynchronous application can switch to doing other work. If your synchronous memory access is blocking your thread, other unrelated work is being blocked. As for workload optimization, I think OP is overstating how much sharing resources mitigates this - when I’m doing a blind search through all data for some reason (maybe some kind of background GC) or a range lookup I can choose to not cache any of that data. Similarly, if I’m doing a point lookup I can do a pre read. However, mmap has no higher level concept about the workload that’s accessing the memory. This a very real problem the kernel tried to tackle back in the 90s / early 00s when disk indexing was starting out - a disk index would cause the cache for all other apps to be evicted resulting in waking up to a sluggish machine paging everything back in. The more high level context you can retain, the better your decision making about resource allocation is. When that high level context is lost/discarded you lose efficiency.
- inkyoto 3y ago
  > There is a trade off obviously where the kernel can evict I/O buffers more efficiently. But to me all that says is that the interface for managing I/O buffers in user space is really immature and needs some better performing API so that we have something in between mmap and buffer management. This was essentially the argument against adding the raw device support to Linux, which delayed the Oracle database Linux port for years, with Linus himself being adamant that operating system's own VMM and page cache management was superior to that of the database (except it was not), which was proven to be a fallacy as the transaction and error handling database requires completely different guarantees, and the VMM pursuing different design objectives that most of the time are incompatible with the database expectations. The raw devices have been used precisely for that reason: «let's bypass the VMM and the page cache for this device, and we will take it from there». That is what the Oracle database did on Solaris – it wanted an entire raw device that it would claim for the table space. Granted, Oracle has had its own VMM and page cache (so to speak) implementation within the DB, an OS mini-kernel if you like. The argument was eventually settled with adding the O_DIRECT flag to open(2) accomplishing the same outcome, and the raw devices have never made into Linux.
jitl 3y ago
It’s funny that the article holds up RavenDB’s rebuttal about MMAP and then discusses correctness, because Aphyr/Jepsen’s latest analysis on RavenDB showed huge correctness issues there: https://jepsen.io/analyses/ravendb-6.0.2 https://jepsen.io/analyses/ravendb-6.0.2 Absolutely brutal read on RavenDB.
- password4321 3y ago
  RavenDB 6.0.2 (A Jepsen Report) https://news.ycombinator.com/item?id=39204622 https://news.ycombinator.com/item?id=39204622 10 days ago
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]
Jakesjeff 3y ago
[dead]
gorset 3y ago
I struggle to use lmdb when using resource isolation (like docker, nomad exec driver). The hidden complexities really become apparent when you build largish systems using shared infrastructure. Anything is easy if you have a dedicated machine or have dataset smaller than your memory allocation. Mmap is so simple for the application/developer, but the price you pay is giving up clear ownership of memory and clean separation between kernel page cache and application space. The interaction between the mmu, mmap, tlb and the linux kernel IO layer is complicated. You don't want to touch that complexity at all if you care about ultra low latency. The article rephrases the question into "do you really want to reimplement everything the OS already does for you?" but I would rephrase it again into questioning why fairly straight forward concepts like a buffer cache can't be implemented directly in the application?
- natmaka 3y ago
  > You don't want to touch that complexity at all if you care about ultra low latency. This is less about a given criterion (such as ultra low latency) than about an adequate global performance (multiple requirements: max latency, average latency, perceived interactive latency, throughput...) on a high-yield machine (more often than not the implication being that it simultaneously has many different and changing missions).
ComputerGuru 3y ago
Something I ~never see mentioned in mmap discussions is that it completely bypasses the expensive syscall transition for read calls, which has only grown rapidly in cost thanks to all the speculative execution mitigations. In benchmarking I found an mmap approach to be integral to achieving maximum performance in IO-heavy workloads on modern OSes [0]. (Note that it still is a win even with mitigations disabled.) [0]: https://neosmart.net/blog/using-simd-acceleration-in-rust-to-create-the-worlds-fastest-tac/ https://neosmart.net/blog/using-simd-acceleration-in-rust-to...
- lvlabguy 3y ago
  Then you have page faults to deal with.
  - deleted 3y ago
    [deleted]
- hnlmorg 3y ago
  Unless I’m completely missing your point, and to be fair I might be because it’s late in Europe and I should have gone to bed hours ago, read speeds due to bypassing expensive syscall overhead is an often cited part of the mmap discussion.
  - ComputerGuru 3y ago
    Sorry, I meant specifically the post-mitigations aspect. It really does change the calculus.
- lathiat 3y ago
  You can likely be solve that with io_uring
  - hinkley 3y ago
    If it's not solved by io_uring, then what is io_uring for?
pengaru 3y ago
The problem with using mmap-based IO is you generally depend on the kernel's async readahead/readaround to keep things fast while writing blocking-style, obvious procedural style code, but it's totally naive, and sequential relative to the faulted addresses, much like a spinning disk prefers to be read. That worked well when the storage system was made of spinning disks, and you were incentivized to use data structures laying data out appropriate for that access pattern. It was all complementary and cooperated pretty well. Then SSDs became commonplace. Today you want to exploit their fast random access, utilizing whatever data structures make the most sense in a world without seek times. Introducing mmap() into that stack kind of reintroduces spinning disk style performance characteristics, if you're relying on mmap()'s async readahead/readaround magic to keep your accesses from blocking on IO. I'm not saying mmap() was ever a great choice for a database. But it's really awful for random access patterns until everything's hot in the page cache. This happens to be a large part of why systemd-journald's journals can be slow to process w/`journalctl -u` or `systemctl status $service` type queries when cold. The access patterns are very random combined with tiny objects being read, while the kernel goes off prefetching heaps of ~unrelated nearby pages in the hopes of preventing blocking subsequent accesses.
- jnwatson 3y ago
  Readahead is configurable via madvise, and it is configurable in LMDB as well.
  - jandrewrogers 3y ago
    With the caveat that I have not looked at this in over a decade, readahead via madvise() is imprecise to the point of near uselessness for high-throughput database workloads. The biggest issue is that, as the name implies, Linux is free to ignore madvise() or do something that is not quite what you requested, and often does. Many types of finely choreographed locality optimizations require more determinism than it offers.
- hyc_symas 3y ago
  > That worked well when the storage system was made of spinning disks, and you were incentivized to use data structures laying data out appropriate for that access pattern. It was all complementary and cooperated pretty well. > Then SSDs became commonplace. You are laughably wrong. Random I/O is random I/O whether you do it using lseek/read or whether you access a page offset in a mmap. And we've got hundreds of hours of benchmarks that show that LMDB outperforms everything else, whether on spinning rust or solid state. http://www.lmdb.tech/bench/ http://www.lmdb.tech/bench/
dwattttt 3y ago
> The ones who got it wrong are irrelevant, because correct solutions clearly exist for all the potential problems. This is a pretty weak argument. Correct solutions exist for doing extremely unsafe things, but that doesn't mean using safer approaches is bad. If a bunch of different projects get something wrong, yes that _is_ a signal that you should think twice before using it yourself.
- deleted 3y ago
  [deleted]
dboreham 3y ago
You need to understand (well) what the kernel is going to do. That's more difficult than if you call read/write yourself (because their semantics are well defined). Otoh the problem is significantly less than it was back in the day because there's really only one kernel now.
- hyc_symas 3y ago
  Welp, back in college I wrote a virtual memory demand pager for IBM 370 mainframe. I also read the entire 4.3BSD kernel source, as well as the Minix kernel source, and hacked on both of them. Also ported NFS server and client to the mainframe. After college, wrote VMEbus device drivers for Sun4 SPARC/Solaris, and a couple drivers for Linux and Windows as well. I'm pretty familiar with what any given kernel does.
raccoonone 3y ago
I wrote redb (https://github.com/cberner/redb https://github.com/cberner/redb) using mmap, initially. However, I later removed it and switched to read()/write() with my own user space cache. I'm sure it's not as good as the OS page cache, but the difference was only 1.2-1.5x performance on the benchmarks I cared about, and the cache is less than 500 lines of code. Also, by removing mmap() I was able to remove all the unsafe code associated with it, so now redb is memory-safe.
- hyc_symas 3y ago
  500 lines of code is still 500 lines of added complexity. For LMDB that'd be a 7% increase in LOCs, which would also introduce the need for manual cache configuration and tuning (further increasing complexity for the end user) and a 50% performance loss? Doesn't seem like a good tradeoff to me. But at least you thought about the decision. Nice benchmark, by the way. https://mastodon.social/@hyc/111887577620902329 https://mastodon.social/@hyc/111887577620902329
xyzzy_plugh 3y ago
In the days of yore, everyone eschewed mmap because it required platform specific tuning. Then everyone switched to one platform and continued to eschew mmap because they didn't want to read the docs. Sure, you can manifest destiny and write it all yourself, and many do, but if you learn how to make mmap really fly, then you can do that again and again without having to even so much as smell a DBMS. The Linux kernel defaults are pretty horrendous for anything resembling high performance hardware, let alone disks that act like memory, but it can most comfortably accommodate all of those things handedly if you just configure the damn thing properly.
cperciva 3y ago
Yes and no. The authors are very correct in drawing a distinction between read-only mmaps and writable mmaps. Read-only is pretty safe -- the biggest concern is probably sanity-checking data being read from disk, but for some applications on-disk corruption isn't a big concern anyway -- while writable mmaps need very careful use of msync to ensure crash safety and it's very easy to have subtle bugs in that sort of code. On the other hand, they hand-wave away I/O stalls saying that the application is going to stall regardless of whether disk reads are implicit or explicit; this fails to account for the possible advantages of narrowing down when a stall might occur. A DBMS might be servicing many requests at once; using explicit I/O, blocking reads could be shunted off to a different context (dedicated I/O threads, I/O request rings, etc) allowing unblocked operations to complete without waiting. The authors also make the odd claim that "[LMDB's] B+tree design is naturally optimal with an LRU cache", which is highly suspect in principle (LRU is a standard not-horribly-bad cache eviction scheme, but it's hard to imagine it being optimal for anything but a trivial workload) but also ignores the fact that a database inherently knows which pages it doesn't need any more. This can be communicated to the kernel with madvise, but that adds more syscalls with inherent overhead. Finally, there's an issue of paging granularity: If you have a database with a large amount of data in core, you really want to be using superpages in order to reduce your TLB miss costs -- but you can't mmap data with superpages if you want the kernel to be able to evict unused pages. In short: Using mmap in a database absolutely can work, especially if (like the authors) you're only doing it read-only; but the case for using mmap is nowhere near as strong as the authors make it out to be.
- hyc_symas 3y ago
  > hand-wave away I/O stalls > A DBMS might be servicing many requests at once; Yes, a DBMS may be servicing many requests at once, which is why individual stalls don't matter. Other threads will be making progress. > a database inherently knows which pages it doesn't need any more. DBMS engineers always make this BS claim and that's what it is, BS. You may know exactly what information you need for a single given query. But you don't know what the next query will be. You don't know if the next query will need the same indices or something else entirely. If you throw something away because you're done with it on one query, and it's referenced again on the next, you'll have pessimized your performance for no good reason. TLB miss costs - this is totally irrelevant. Go check out any benchmark you can find of LMDB against any other DBMS. Whether conducted by us or any 3rd party. Find any system that gets anywhere close to LMDB's read performance and scalability under heavy concurrent load, and prove that TLB misses are costing LMDB efficiency.
IamLoading 3y ago
https://www.youtube.com/watch?v=1BRGU_AS25c https://www.youtube.com/watch?v=1BRGU_AS25c - video regarding the paper findings by the authors.
hyc_symas 3y ago
Before we go any further here, you should all read the original LMDB design paper, to get a grasp of what we traded away first. https://openldap.org/pub/ https://openldap.org/pub/ You can also watch the presentation at CMU https://www.youtube.com/watch?v=tEa5sAh-kVk https://www.youtube.com/watch?v=tEa5sAh-kVk (slides are here http://www.lmdb.tech/SymasDocs/20151008-CMU-LMDB.pdf http://www.lmdb.tech/SymasDocs/20151008-CMU-LMDB.pdf )
leonardb2 3y ago
I love lmdb and mmap, but I wish there was prefix compression for keys. I'm on the fence between rocksdb and lmdb, both using mmap and will likely support both. My database file was so much smaller with rocksdb because it compresses segments, but lmdb is so much more concise and will never do background jobs like rocksdb. One can use different subtables in lmdb, but the amount must be set up front with mdb_env_set_maxdbs and it says that there is a linear search involved.
- hyc_symas 3y ago
  Prefix compression adds a lot of complexity. If you insert a new record that makes an existing prefix non-unique, you then have to expand and rewrite every key in the page, and that will ripple up to every parent page as well. And the expansion can then cause the records to no longer fit in their current page, so then you need to do a bunch of page splits. It's gross and ugly and easy to get wrong. (The btree.c that LMDB was based on has prefix compression. We ripped it out after finding bugs in it.) If your data values are small enough, you can use DUPSORT records instead, and manage your prefixes manually. There is a linear search involved to open a DBI handle, yes. But you only do that once, and keep it open for the life of the program. Not a big deal.
BohuTANG 3y ago
These articles all miss the point. If you are a DBMS kernel developer, the most urgent need is how to control memory more precisely to prevent out-of-memory (OOM) situations. Clearly, MMAP cannot achieve this, which is also why most databases still use their own memory pool + direct-IO.
- hyc_symas 3y ago
  > Clearly, MMAP cannot achieve this Clearly you have no idea what you're talking about. With a read-only mmap you don't need to do anything, OOM won't happen*. All of the pages of the map are clean; whenever there's memory pressure the OS can just reclaim those pages from the page cache and use them to meet other demands. That page re-use is zero cost; since the pages are clean they don't require a flush or pageout before they can be reused. Using a read-only mmap means the DB will never be the cause of an OOM situation, and will never be the cause of swap usage. *on Linux you must lower /proc/sys/vm/swappiness from its default of 60, otherwise the kernel won't reclaim FS cache pages.
  - BohuTANG 3y ago
    Imagine you're using a EC2 to run several DBMS instances at the same time, each for a different customer. How do you make sure that each instance only uses a certain amount of memory and doesn't affect the others?
    - inkyoto 3y ago
      In Linux, you use cgroups to confine a process to a fixed memory limit that won't cause an OOM in adjacent processes. You can create a separate memory cgroup with a different allowance for each customer depending how much memory is allotted to them per the contract, for example. You can also use ulimit with a hard memory limit, but cgroups are more fine grained and flexible. Also do note that the point about the read-only mmap still holds – it can't cause an OOM, it is mmap'd pages pointing to dirty pages that have not been written out to the file system yet that can – on the condition their number is growing uncontrollably and/or fast. P.S. But allowing a database to cause an OOM is never a good idea to start off with whether memory limits have been set or not – the data loss is unacceptable for a database, and databases have been optimised to operate within pre-configured space constraints that do not even require ulimit/cgroup in the first place. Perhaps fixing the database configuration is a better starting point rather than going down the rabbit hole with mmap(2).
pengaru 3y ago
> memcpy() (ie "read()" in this case) is _always_ going to be faster in many > cases, just because it avoids all the extra complexity. While mmap() is > going to be faster in other cases. > - Linus Torvalds [0] At least a long-running db process can amortize the page table costs across many operations, unlike something ephemeral like journalctl/systemctl where it really hurts. [0] https://yarchive.net/comp/mmap.html https://yarchive.net/comp/mmap.html
dnc 3y ago
Previous discussions: - https://news.ycombinator.com/item?id=36563187 https://news.ycombinator.com/item?id=36563187 - https://news.ycombinator.com/item?id=29936104 https://news.ycombinator.com/item?id=29936104 - https://news.ycombinator.com/item?id=31504052 https://news.ycombinator.com/item?id=31504052
monsecchris 3y ago
On disk errors you get sigbus and your application exits. This is totally unacceptable in most circumstances I am familiar with.
- icedchai 3y ago
  If this happens regularly, you should probably fix your hardware.
icedchai 3y ago
About 20 years ago, I worked on a server application that used mmap for its database. At startup, it read all the pages into memory, so all data was "hot", ready to go. It also built its in-memory indexes at the same time. It worked incredibly well, handling 1000's of simultaneous clients. This was the main enterprise application responsible for 100's of millions in revenue for a large organization. It wasn't a general purpose solution, though. The system also fed more standard relational databases for reporting purposes, using a custom change data capture process.
- deleted 3y ago
  [deleted]

98 points by mfiguiere 3y ago

I really wish HYC was more articulate and less egomaniacal. He makes a lot of claims and doesn’t really flesh out great arguments. Would be really interesting to have him discuss with Andy Pavlo :)

cmrdporcupine 3y ago
Yeah the strident and somewhat impolite tone is really hard to tolerate. Esp when directed against respected researchers like Pavlo and Leis who, well, they know what they're talking about. Everything has compromises and trade-offs. LMDB looks great. But it's also, like a lot of similar systems, built to perform best at single-writer / multiple-reader workloads. That's not the only workload out there.
- hyc_symas 3y ago
  If they knew what they were talking about they wouldn't have used a raw filesystem benchmarking tool to prove their arguments. They would have used a pair of actual, comparable DBMS implementations. Single-writer is all you need if each write is fast enough. BerkeleyDB, which we used before writing LMDB, is multi-writer. It's still slower than LMDB in every workload, because of all of the lock management that comes with being multi-writer.
hyc_symas 3y ago
Andy and I have had this conversation many times over. https://www.youtube.com/watch?v=tEa5sAh-kVk https://www.youtube.com/watch?v=tEa5sAh-kVk Feel free to point out specific arguments that you think need further fleshing out. Every aspect of LMDB's design and performance characteristics have been documented already, happy to link you to answers to any of your questions.

vlovich123 3y ago

Author is right that a read only mmap with writing occurring normally is a better design. It’s a weakness in the paper but in fairness few databases did what LMDB did sanely. Surprising that that’s the case but that is what people chose to do. Where they’re wrong is about the performance analysis: > In section 3.2 "I/O stalls" is again a non-issue; no matter how your DBMS handles I/O internally, synchronously or asynchronously, the calling application can't make any progress until the I/O completes, and if the data isn't already in memory then the application must wait. A TLB shoot down request to pull in a page or evict it can cause pauses not just in your own application but also many others because it’s basically a system wide mutex. There is a trade off obviously where the kernel can evict I/O buffers more efficiently. But to me all that says is that the interface for managing I/O buffers in user space is really immature and needs some better performing API so that we have something in between mmap and buffer management. Also an asynchronous application can switch to doing other work. If your synchronous memory access is blocking your thread, other unrelated work is being blocked. As for workload optimization, I think OP is overstating how much sharing resources mitigates this - when I’m doing a blind search through all data for some reason (maybe some kind of background GC) or a range lookup I can choose to not cache any of that data. Similarly, if I’m doing a point lookup I can do a pre read. However, mmap has no higher level concept about the workload that’s accessing the memory. This a very real problem the kernel tried to tackle back in the 90s / early 00s when disk indexing was starting out - a disk index would cause the cache for all other apps to be evicted resulting in waking up to a sluggish machine paging everything back in. The more high level context you can retain, the better your decision making about resource allocation is. When that high level context is lost/discarded you lose efficiency.

inkyoto 3y ago
> There is a trade off obviously where the kernel can evict I/O buffers more efficiently. But to me all that says is that the interface for managing I/O buffers in user space is really immature and needs some better performing API so that we have something in between mmap and buffer management. This was essentially the argument against adding the raw device support to Linux, which delayed the Oracle database Linux port for years, with Linus himself being adamant that operating system's own VMM and page cache management was superior to that of the database (except it was not), which was proven to be a fallacy as the transaction and error handling database requires completely different guarantees, and the VMM pursuing different design objectives that most of the time are incompatible with the database expectations. The raw devices have been used precisely for that reason: «let's bypass the VMM and the page cache for this device, and we will take it from there». That is what the Oracle database did on Solaris – it wanted an entire raw device that it would claim for the table space. Granted, Oracle has had its own VMM and page cache (so to speak) implementation within the DB, an OS mini-kernel if you like. The argument was eventually settled with adding the O_DIRECT flag to open(2) accomplishing the same outcome, and the raw devices have never made into Linux.

jitl 3y ago

It’s funny that the article holds up RavenDB’s rebuttal about MMAP and then discusses correctness, because Aphyr/Jepsen’s latest analysis on RavenDB showed huge correctness issues there: https://jepsen.io/analyses/ravendb-6.0.2 https://jepsen.io/analyses/ravendb-6.0.2 Absolutely brutal read on RavenDB.

password4321 3y ago
RavenDB 6.0.2 (A Jepsen Report) https://news.ycombinator.com/item?id=39204622 https://news.ycombinator.com/item?id=39204622 10 days ago
deleted 3y ago
[deleted]

deleted 3y ago

[deleted]

Jakesjeff 3y ago

[dead]

gorset 3y ago

I struggle to use lmdb when using resource isolation (like docker, nomad exec driver). The hidden complexities really become apparent when you build largish systems using shared infrastructure. Anything is easy if you have a dedicated machine or have dataset smaller than your memory allocation. Mmap is so simple for the application/developer, but the price you pay is giving up clear ownership of memory and clean separation between kernel page cache and application space. The interaction between the mmu, mmap, tlb and the linux kernel IO layer is complicated. You don't want to touch that complexity at all if you care about ultra low latency. The article rephrases the question into "do you really want to reimplement everything the OS already does for you?" but I would rephrase it again into questioning why fairly straight forward concepts like a buffer cache can't be implemented directly in the application?

natmaka 3y ago
> You don't want to touch that complexity at all if you care about ultra low latency. This is less about a given criterion (such as ultra low latency) than about an adequate global performance (multiple requirements: max latency, average latency, perceived interactive latency, throughput...) on a high-yield machine (more often than not the implication being that it simultaneously has many different and changing missions).

ComputerGuru 3y ago

Something I ~never see mentioned in mmap discussions is that it completely bypasses the expensive syscall transition for read calls, which has only grown rapidly in cost thanks to all the speculative execution mitigations. In benchmarking I found an mmap approach to be integral to achieving maximum performance in IO-heavy workloads on modern OSes [0]. (Note that it still is a win even with mitigations disabled.) [0]: https://neosmart.net/blog/using-simd-acceleration-in-rust-to-create-the-worlds-fastest-tac/ https://neosmart.net/blog/using-simd-acceleration-in-rust-to...

lvlabguy 3y ago
Then you have page faults to deal with.
- deleted 3y ago
  [deleted]
hnlmorg 3y ago
Unless I’m completely missing your point, and to be fair I might be because it’s late in Europe and I should have gone to bed hours ago, read speeds due to bypassing expensive syscall overhead is an often cited part of the mmap discussion.
- ComputerGuru 3y ago
  Sorry, I meant specifically the post-mitigations aspect. It really does change the calculus.
lathiat 3y ago
You can likely be solve that with io_uring
- hinkley 3y ago
  If it's not solved by io_uring, then what is io_uring for?

pengaru 3y ago

The problem with using mmap-based IO is you generally depend on the kernel's async readahead/readaround to keep things fast while writing blocking-style, obvious procedural style code, but it's totally naive, and sequential relative to the faulted addresses, much like a spinning disk prefers to be read. That worked well when the storage system was made of spinning disks, and you were incentivized to use data structures laying data out appropriate for that access pattern. It was all complementary and cooperated pretty well. Then SSDs became commonplace. Today you want to exploit their fast random access, utilizing whatever data structures make the most sense in a world without seek times. Introducing mmap() into that stack kind of reintroduces spinning disk style performance characteristics, if you're relying on mmap()'s async readahead/readaround magic to keep your accesses from blocking on IO. I'm not saying mmap() was ever a great choice for a database. But it's really awful for random access patterns until everything's hot in the page cache. This happens to be a large part of why systemd-journald's journals can be slow to process w/`journalctl -u` or `systemctl status $service` type queries when cold. The access patterns are very random combined with tiny objects being read, while the kernel goes off prefetching heaps of ~unrelated nearby pages in the hopes of preventing blocking subsequent accesses.

jnwatson 3y ago
Readahead is configurable via madvise, and it is configurable in LMDB as well.
- jandrewrogers 3y ago
  With the caveat that I have not looked at this in over a decade, readahead via madvise() is imprecise to the point of near uselessness for high-throughput database workloads. The biggest issue is that, as the name implies, Linux is free to ignore madvise() or do something that is not quite what you requested, and often does. Many types of finely choreographed locality optimizations require more determinism than it offers.
hyc_symas 3y ago
> That worked well when the storage system was made of spinning disks, and you were incentivized to use data structures laying data out appropriate for that access pattern. It was all complementary and cooperated pretty well. > Then SSDs became commonplace. You are laughably wrong. Random I/O is random I/O whether you do it using lseek/read or whether you access a page offset in a mmap. And we've got hundreds of hours of benchmarks that show that LMDB outperforms everything else, whether on spinning rust or solid state. http://www.lmdb.tech/bench/ http://www.lmdb.tech/bench/

dwattttt 3y ago

> The ones who got it wrong are irrelevant, because correct solutions clearly exist for all the potential problems. This is a pretty weak argument. Correct solutions exist for doing extremely unsafe things, but that doesn't mean using safer approaches is bad. If a bunch of different projects get something wrong, yes that _is_ a signal that you should think twice before using it yourself.

deleted 3y ago
[deleted]

dboreham 3y ago

You need to understand (well) what the kernel is going to do. That's more difficult than if you call read/write yourself (because their semantics are well defined). Otoh the problem is significantly less than it was back in the day because there's really only one kernel now.

hyc_symas 3y ago
Welp, back in college I wrote a virtual memory demand pager for IBM 370 mainframe. I also read the entire 4.3BSD kernel source, as well as the Minix kernel source, and hacked on both of them. Also ported NFS server and client to the mainframe. After college, wrote VMEbus device drivers for Sun4 SPARC/Solaris, and a couple drivers for Linux and Windows as well. I'm pretty familiar with what any given kernel does.

raccoonone 3y ago

I wrote redb (https://github.com/cberner/redb https://github.com/cberner/redb) using mmap, initially. However, I later removed it and switched to read()/write() with my own user space cache. I'm sure it's not as good as the OS page cache, but the difference was only 1.2-1.5x performance on the benchmarks I cared about, and the cache is less than 500 lines of code. Also, by removing mmap() I was able to remove all the unsafe code associated with it, so now redb is memory-safe.

hyc_symas 3y ago
500 lines of code is still 500 lines of added complexity. For LMDB that'd be a 7% increase in LOCs, which would also introduce the need for manual cache configuration and tuning (further increasing complexity for the end user) and a 50% performance loss? Doesn't seem like a good tradeoff to me. But at least you thought about the decision. Nice benchmark, by the way. https://mastodon.social/@hyc/111887577620902329 https://mastodon.social/@hyc/111887577620902329

xyzzy_plugh 3y ago

In the days of yore, everyone eschewed mmap because it required platform specific tuning. Then everyone switched to one platform and continued to eschew mmap because they didn't want to read the docs. Sure, you can manifest destiny and write it all yourself, and many do, but if you learn how to make mmap really fly, then you can do that again and again without having to even so much as smell a DBMS. The Linux kernel defaults are pretty horrendous for anything resembling high performance hardware, let alone disks that act like memory, but it can most comfortably accommodate all of those things handedly if you just configure the damn thing properly.

cperciva 3y ago

Yes and no. The authors are very correct in drawing a distinction between read-only mmaps and writable mmaps. Read-only is pretty safe -- the biggest concern is probably sanity-checking data being read from disk, but for some applications on-disk corruption isn't a big concern anyway -- while writable mmaps need very careful use of msync to ensure crash safety and it's very easy to have subtle bugs in that sort of code. On the other hand, they hand-wave away I/O stalls saying that the application is going to stall regardless of whether disk reads are implicit or explicit; this fails to account for the possible advantages of narrowing down when a stall might occur. A DBMS might be servicing many requests at once; using explicit I/O, blocking reads could be shunted off to a different context (dedicated I/O threads, I/O request rings, etc) allowing unblocked operations to complete without waiting. The authors also make the odd claim that "[LMDB's] B+tree design is naturally optimal with an LRU cache", which is highly suspect in principle (LRU is a standard not-horribly-bad cache eviction scheme, but it's hard to imagine it being optimal for anything but a trivial workload) but also ignores the fact that a database inherently knows which pages it doesn't need any more. This can be communicated to the kernel with madvise, but that adds more syscalls with inherent overhead. Finally, there's an issue of paging granularity: If you have a database with a large amount of data in core, you really want to be using superpages in order to reduce your TLB miss costs -- but you can't mmap data with superpages if you want the kernel to be able to evict unused pages. In short: Using mmap in a database absolutely can work, especially if (like the authors) you're only doing it read-only; but the case for using mmap is nowhere near as strong as the authors make it out to be.

hyc_symas 3y ago
> hand-wave away I/O stalls > A DBMS might be servicing many requests at once; Yes, a DBMS may be servicing many requests at once, which is why individual stalls don't matter. Other threads will be making progress. > a database inherently knows which pages it doesn't need any more. DBMS engineers always make this BS claim and that's what it is, BS. You may know exactly what information you need for a single given query. But you don't know what the next query will be. You don't know if the next query will need the same indices or something else entirely. If you throw something away because you're done with it on one query, and it's referenced again on the next, you'll have pessimized your performance for no good reason. TLB miss costs - this is totally irrelevant. Go check out any benchmark you can find of LMDB against any other DBMS. Whether conducted by us or any 3rd party. Find any system that gets anywhere close to LMDB's read performance and scalability under heavy concurrent load, and prove that TLB misses are costing LMDB efficiency.

IamLoading 3y ago

https://www.youtube.com/watch?v=1BRGU_AS25c https://www.youtube.com/watch?v=1BRGU_AS25c - video regarding the paper findings by the authors.

hyc_symas 3y ago

Before we go any further here, you should all read the original LMDB design paper, to get a grasp of what we traded away first. https://openldap.org/pub/ https://openldap.org/pub/ You can also watch the presentation at CMU https://www.youtube.com/watch?v=tEa5sAh-kVk https://www.youtube.com/watch?v=tEa5sAh-kVk (slides are here http://www.lmdb.tech/SymasDocs/20151008-CMU-LMDB.pdf http://www.lmdb.tech/SymasDocs/20151008-CMU-LMDB.pdf )

leonardb2 3y ago

I love lmdb and mmap, but I wish there was prefix compression for keys. I'm on the fence between rocksdb and lmdb, both using mmap and will likely support both. My database file was so much smaller with rocksdb because it compresses segments, but lmdb is so much more concise and will never do background jobs like rocksdb. One can use different subtables in lmdb, but the amount must be set up front with mdb_env_set_maxdbs and it says that there is a linear search involved.

hyc_symas 3y ago
Prefix compression adds a lot of complexity. If you insert a new record that makes an existing prefix non-unique, you then have to expand and rewrite every key in the page, and that will ripple up to every parent page as well. And the expansion can then cause the records to no longer fit in their current page, so then you need to do a bunch of page splits. It's gross and ugly and easy to get wrong. (The btree.c that LMDB was based on has prefix compression. We ripped it out after finding bugs in it.) If your data values are small enough, you can use DUPSORT records instead, and manage your prefixes manually. There is a linear search involved to open a DBI handle, yes. But you only do that once, and keep it open for the life of the program. Not a big deal.

BohuTANG 3y ago

These articles all miss the point. If you are a DBMS kernel developer, the most urgent need is how to control memory more precisely to prevent out-of-memory (OOM) situations. Clearly, MMAP cannot achieve this, which is also why most databases still use their own memory pool + direct-IO.

hyc_symas 3y ago
> Clearly, MMAP cannot achieve this Clearly you have no idea what you're talking about. With a read-only mmap you don't need to do anything, OOM won't happen*. All of the pages of the map are clean; whenever there's memory pressure the OS can just reclaim those pages from the page cache and use them to meet other demands. That page re-use is zero cost; since the pages are clean they don't require a flush or pageout before they can be reused. Using a read-only mmap means the DB will never be the cause of an OOM situation, and will never be the cause of swap usage. *on Linux you must lower /proc/sys/vm/swappiness from its default of 60, otherwise the kernel won't reclaim FS cache pages.
- BohuTANG 3y ago
  Imagine you're using a EC2 to run several DBMS instances at the same time, each for a different customer. How do you make sure that each instance only uses a certain amount of memory and doesn't affect the others?
  - inkyoto 3y ago
    In Linux, you use cgroups to confine a process to a fixed memory limit that won't cause an OOM in adjacent processes. You can create a separate memory cgroup with a different allowance for each customer depending how much memory is allotted to them per the contract, for example. You can also use ulimit with a hard memory limit, but cgroups are more fine grained and flexible. Also do note that the point about the read-only mmap still holds – it can't cause an OOM, it is mmap'd pages pointing to dirty pages that have not been written out to the file system yet that can – on the condition their number is growing uncontrollably and/or fast. P.S. But allowing a database to cause an OOM is never a good idea to start off with whether memory limits have been set or not – the data loss is unacceptable for a database, and databases have been optimised to operate within pre-configured space constraints that do not even require ulimit/cgroup in the first place. Perhaps fixing the database configuration is a better starting point rather than going down the rabbit hole with mmap(2).

pengaru 3y ago

> memcpy() (ie "read()" in this case) is _always_ going to be faster in many > cases, just because it avoids all the extra complexity. While mmap() is > going to be faster in other cases. > - Linus Torvalds [0] At least a long-running db process can amortize the page table costs across many operations, unlike something ephemeral like journalctl/systemctl where it really hurts. [0] https://yarchive.net/comp/mmap.html https://yarchive.net/comp/mmap.html

dnc 3y ago

Previous discussions: - https://news.ycombinator.com/item?id=36563187 https://news.ycombinator.com/item?id=36563187 - https://news.ycombinator.com/item?id=29936104 https://news.ycombinator.com/item?id=29936104 - https://news.ycombinator.com/item?id=31504052 https://news.ycombinator.com/item?id=31504052

monsecchris 3y ago

On disk errors you get sigbus and your application exits. This is totally unacceptable in most circumstances I am familiar with.

icedchai 3y ago
If this happens regularly, you should probably fix your hardware.

icedchai 3y ago

About 20 years ago, I worked on a server application that used mmap for its database. At startup, it read all the pages into memory, so all data was "hot", ready to go. It also built its in-memory indexes at the same time. It worked incredibly well, handling 1000's of simultaneous clients. This was the main enterprise application responsible for 100's of millions in revenue for a large organization. It wasn't a general purpose solution, though. The system also fed more standard relational databases for reporting purposes, using a custom change data capture process.

deleted 3y ago
[deleted]