15 ms·
Are you sure you want to use MMAP in your database management system? [pdf]
- tptacek 5y agoInteresting parallels in this work to Tanenbaum's "RPC Considered Harmful"†; in both cases, you've got an abstraction that papers over a huge amount of complexity, and it ends up burning you because a lot of that complexity turns out to be pretty important and the abstraction has cost you control over it. † https://www.cs.vu.nl/~ast/Publications/Papers/euteco-1988.pdf https://www.cs.vu.nl/~ast/Publications/Papers/euteco-1988.pd...
- ncmncm 5y agoYes. Whenever you need better performance and reliability, identify an abstraction beloved of CS professors, and bypass it. When I last checked, libtorrent was utterly failing to use O_DIRECT semantics. I started making a patch, but there are several places that do file ops, and the main one was more complicated than I could afford to dive into at the time.
- groby_b 5y agoSo... don't bypass abstractions, unless you actually have time to do a better job, no? We have abstractions for a reason. We have lower-level primitives for a reason. Understanding the differences, reasoning about all trade-off angles, and making the right choice in each project is a majority of the software engineering job.
- yellowapple 5y ago> So... don't bypass abstractions, unless you actually have time to do a better job, no? And unless there's a clear benefit to it. If you haven't identified the abstraction as a significant bottleneck, then is it really worthwhile to go through the trouble of bypassing it?
- hinkley 5y agoThe problem is that people like to carve out territories in their data architecture before they have become subject matter experts. Once you split two things it's so difficult to add certain kinds of features that most people just give up and deal with higher fanout. What you often get is the sum being less than the whole of its parts, and trying to offset that by achieving greater 'parts' through coherence and conceptual integrity in isolation. There is such a thing as 'Coherent but wrong'.
- pengaru 5y agoChoosing mmap() gets you something that works sooner than later. But then you have a pile of blocking-style synchronous code likely exploiting problematic assumptions to rewrite when you realize you want something that doesn't just work, but works well.
- germandiago 5y agommap + C++ coroutines can be combined.
- pengaru 5y agoAnd? So you're going to take a syscall (mincore()?) hit before every file-backed mapping access to test if it'd incur a page fault, and try switch to another coroutine if it would? Syscalls aren't free, especially today, and mmap() is already bringing significant overhead to the party. If you've brought coroutines into the picture, you might as well schedule them using async IO completions and kick mmap() to the curb.
- jandrewrogers 5y agoWhat problem does this solve? Use of mmap() tends to imply blocking calls.
- apavlo 5y agoThis link should be to this page that includes more info and the accompanying video: https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/
- dsiegel2275 5y agoThank you for sharing your DB course(s) videos on the YouTube. I'm a CMU staff member (Open Learning Initiative) that would never be able to enroll on-site, given likely my lower priority for getting a seat, but watching your videos online has been fantastic.
- lmwnshn 5y agoSince you have a CMU ID, AFAIK you should be able to enroll in Piazza in addition to following along with the assignments/projects (if you wanted to).
- PaulHoule 5y agoMost of the times I used mmap I wasn't happy in the end. I went through a phase when I thought it was fun to do extreme random access on image files, archives and things like that. At some point I think "I want to do this for a file I fetch over the network" and that needs a rewrite.
- icedchai 5y agoI worked at a company that developed its own proprietary database for a financial application. The entire database, several gigabytes, not large by today's standards, was mmap'd and read at startup to warm the page cache. We also built in-memory "indexes" at startup. This was back in the early 2000's, when having 4 gigabytes of RAM would be considered large. The "database server" was single threaded, and all changes were logged to a WAL-ish file before updating the mmap'd database. It was fun stuff to work on. It worked well, but it wasn't a general purpose DB.
- jnwatson 5y agoI have a great deal of experience in running very large memory-mapped databases using LMDB. The default Linux settings dealing with memory mapped files are pretty horrible. The observed poor performance is directly related to not configuring several very important kernel parameters.
- jzwinck 5y agoCan you share a couple examples of those parameters?
- jnwatson 5y agoThis describes the relevant kernel variables: https://synapse.docs.vertex.link/en/latest/synapse/devguides/devops_general.html#tips-for-better-performance https://synapse.docs.vertex.link/en/latest/synapse/devguides...
- sprachspiel 5y agoThese settings control writing back modified pages. The experiments in the paper are read-only. With writes the situation is even worse than shown in the paper (though tuning these settings may help a bit).
- cryptonector 5y agoI am convinced. Great video.
- eqvinox 5y agoAnyone have any clue why there's a 3-phase sine wave showing up in mmap performance? (Figures 2a/2b)
- mpweiher 5y agoYes, I do. Really. And if current mmap() implementations aren't up to the task, can we fix mmap()? Please.
- assface 5y agoRavenDB's response to this paper: https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to-use-mmap-in-your-database-management-system https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to...
- tptacek 5y agoFrom that article: the whole fsyncgate thing seems like a pretty strong counterargument to "mmap adds more complexity than it removes": https://danluu.com/fsyncgate/ https://danluu.com/fsyncgate/
- dwenzek 5y agoThank you for these counter-arguments. It's good to have them to make up your own mind, especially when recognized experts use a mocking tone "you will no dare think the contrary".
- tyingq 5y agoThe really key part seems to be this: "If you aren’t using mmap, on the other hand, you still need to handle of all those issues" Which seems like a reasonable statement. Is it less work to make your own top-to-bottom buffer pool, and would that necessarily avoid similar issues? Or is it less work to use mmap(), but address the issues?
- dboreham 5y agoWhen I worked on/with BerkeleyDB in the late 90s we came to the conclusion that the various OS mmap() implementations had been tweaked/fixed to the point where they worked for the popular high profile applications (in those days: Oracle). So it can appear like everything is fine, but that probably means your code behaves the same way as <popular database du jour>.
- dang 5y agoUrl changed from https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/, which has the abstract and a link to this video: https://www.youtube.com/watch?v=1BRGU_AS25c https://www.youtube.com/watch?v=1BRGU_AS25c and this code: https://github.com/viktorleis/mmapbench https://github.com/viktorleis/mmapbench
- gnabgib 5y agoI personally prefer the abstract to jumping straight into a full paper, especially since it's quite rich (not one of those two line entries like some arXiv paper abstracts). After reading the abstract I did end up opening the PDF.. but I'm hesitant to pay the PDF tax early. Is this one of those "original source" type decisions?
- dang 5y agoYes. I hear you about the downside, but the downside of the more superficial-accessible 'home page' is that people will not read any further, and instead simply respond generically.
- yellowapple 5y agoThe current URL just redirects back to that "superficial-accessible 'home page'" anyway (probably as a substitute for 404 handling, I'd guess); if the intent is to link directly to the paper/PDF, you probably want https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf But I agree with the other person; if people really won't read any further from the homepage, then I highly doubt they'd read any further than the headline and maybe abstract of the original paper anyway, so there ain't really much upside to linking directly to the paper - whereas there's quite a bit of downside for anyone who might feel inclined to watch the video instead (which essentially covers the same information as the paper, just at a higher level / without the same level of detail) or review the benchmark code - neither of which are accessible from the paper.
- 5y ago
- moonbug 5y agowhy settle for errno when you can have a segfault.
- kabdib 5y agoYup. Why use the operating system's async I/O system when you can simply burn a thread and do blocking I/O? </snark> Been down that primrose path, have the road rash to prove it. mmap() is great until you realize that pretty much all you've avoided is some buffer management that you probably need to do anyway. The OS just doesn't have the information it needs to do a great (or even correct) job of caching database pages.
- randbox 5y agoA user process doesn't have the information it needs to do a good job of coordinating updates from multiple writers to database pages and indices. With MMAP, writers have access to shared atomics which they can update using compare-exchange operations to prevent data races which would be common when using read() and write() without locks.
- jstimpfle 5y agoAre you saying that without mmap() there will be data races??
- randbox 5y agoThere can be a data race any time a processor loads a value, modifies it, and writes it back. Without an atomic update operation like compare_exchange() generally you need to lock the database file against other processes and threads. The typical solution is to only have one process update the file, only have one thread perform the writes, and combine it with a TCP server. Suppose you have a big data file and want to mark which pages are occupied and which pages are free. Suppose a writer wants to read a bit from an index page to the stack to check whether a data page is occupied, modify the page bit in the stack to claim the data page if another process hasn't claimed it, and write the updated value back to memory to claim the data page to store the data value if another process hasn't claimed it. If each process read()s the index bits, they can both see that page 2 bit in the index is unset and try to claim it, then write() back the updated index value. The updates to the index will collide, both writers will think the claimed page 2 when only one should have, and one of the data values written to that page will get lost.
- deleted 5y ago[deleted]
- dboreham 5y agoDo they mean: mmap() ?
- deleted 5y ago[deleted]
- jasone 5y agoCheck out the creative use of emoji in the running header!
- jandrewrogers 5y agoThe pragmatic consideration that usually influences the decision to use mmap() is the large discontinuity in skill and expertise required to replace it. Writing your own alternative to mmap() can be significantly superior in terms of performance and functionality, and often lends itself to a cleaner database architecture. However, this presumes a sufficiently sophisticated design for an mmap() replacement. The learning curve is steep and the critical nuances of sophisticated and practical designs are poorly explored in readily available literature, providing little in the way of "how-to" guides that you can lean on. As a consequence, early attempts to replace mmap() are often quite poor. You don't know what you don't know, and details of the implementation that are often glossed over turn out to be critical in practice. For example, most people eventually figure out that LRU cache replacement is a bad idea, but many of the academic alternatives cause CPU cache thrashing in real systems, replacing one problem with another. There are clever and non-obvious design elements that can greatly mitigate this but they are treated as implementation details in most discussions of cache replacement and largely not discoverable if you are writing one for the first time. While mmap() is a mediocre facility for a database, I think we also have to be cognizant that replacing it competently is not a trivial ask for most software engineers. If their learning curve is anything like mine, I went from mmap() to designing obvious alternatives with many poorly handled edge cases, and eventually figuring out how to design non-obvious alternatives that could smoothly handled very diverse workloads. That period of "poor alternatives" in the middle doesn't produce great databases but it almost feels necessary to properly grok the design problem. Most people would rather spend their time working on other parts of a database.
- tmountain 5y agoThe original version of MongoDB used mmap, and I worked at a company that had a ton of issues with cache warmup and the cache getting trashed by competing processes. Granted this was a long time ago, but the main issue was the operating system's willingness to reallocate large swaths of memory from the address space to whatever process was asking for memory right now. Once the working set got trashed, performance would go through the floor, and our app would slow to a crawl while the cache went through the warmup cycle. Long story short, with that model, Mongo couldn't "own" the memory it was using, and this lead to chronic problems. Wiredtiger fixed this completely, but I still think this is a cautionary tale for anyone considering building a DB without a dedicated memory manager.
- pizlonator 5y agoThis is a great write-up! Makes me wonder if there is an alternative universe in which there is a syscall with semantics similar to mmap that avoids these pitfalls. It's not like mmap's semantics are the only semantics that we could have for memory-mapped IO.
- sprachspiel 5y agoThis would be exactly the kind of innovation we would need in computer science. Instead we often get stuck in local minima (in this case a 40-year old POSIX interface) without realizing how much pain this causes.
- rectang 5y agoOne possible advantage of using mmap over a buffer pool can be programmer ergonomics. Reading data into a buffer pool in process RAM takes time to warm up, and the pool can only be accessed by a single process. In contrast, for an mmap-backed data structure, assuming that files are static once written (which can be the case for an multi-version concurrency control (MVCC) architecture), you open an mmap read-only connection from any process and the so long as the data is already in the OS cache, you get instant fast reads. This makes managing database connections much easier, since connections are cheap and the programmer can just open as many as they want whenever and wherever they want. It is true that cache eviction strategy used by the OS is likely to be suboptimal. So if you're in a position to only run a single database process, you might decide to make different tradeoffs.
- timgarmstrong 5y agoThis is true, but in the case where files are read only, just reading directly from the files with fread()/read()/etc works pretty well. You do have to pay the cost of a system call and a copy from the OS buffer cache into your user-space buffer, but OTOH when the page isn't in the buffer cache, the cost of reading the required data from storage is more predictable than the cost of faulting in all the 4kb pages you're reading.
- randbox 5y agoHow do you implement lockless atomic updates for multiple writers across multiple threads & processes without mmap? With mmap it is straight forward for processes to open persistent arrays of atomics as a file, and use compare and exchange operations to prevent data races when multiple threads or processes update the same page without any file locks, advisory locks, or mutexes. With manual read() and write() calls, the data may be overwritten by another writer before the update is committed.
- jstimpfle 5y agoWhy do you need lockless atomic updates to a file-backed memory area? Genuinely curious.
- CyberDildonics 5y agoBecause it allows you to do lock free memory based interprocess communication, which can be extremely fast.
- jstimpfle 5y agoThere is no need for file-backed memory to do that.
- CyberDildonics 5y agoSounds good, what is your solution and why didn't you explain it in your first reply?
- jstimpfle 5y agoWhat is my solution to what? To database I/O? I guess that's what the article is about...
- CyberDildonics 5y ago