4 ms·
We don't do anything super special here. Most databases have some notion of "node" (or in mongo, "bucket", to distinguish from a machine as a node in a cluster)
by leif 14y ago
We don't do anything super special here. Most databases have some notion of "node" (or in mongo, "bucket", to distinguish from a machine as a node in a cluster), and keep track of which nodes in memory are clean or dirty. We log all operations and by default, fsync that on commit, so we're completely durable, there's no data loss (unless you didn't commit something before you pulled the plug, of course). To trim the log, you have to know that everything dirtied by a transaction has been written to disk, and if you don't trim the log often enough, it gets big and your crash recovery takes a long time (innodb sometimes exhibits this). We and others have a notion of "checkpointing" which says "all the nodes as of this point in the log has been written to disk and marked clean", and that allows us to trim the log. Note that this doesn't have to mean "stop the world and write out everything in main memory", you can do a lot better (but that's a long discussion).
Because a given operation, even if it involves lots of random writes in the key space, still only dirties a few nodes at the top of the tree, we actually checkpoint faster and by writing less data than the same b-tree would (because it would have to checkpoint a bunch of leaves, with random I/O, for every one of our nodes), so our log is often much smaller and our recovery times are much faster, usually on the order of seconds or minutes, rather than hours. By default in our mysql product, we checkpoint every...I think 60 seconds, but it could be 90. It's configurable but that's the ballpark.
But most of my comparisons here are with innodb, to be fair. Mongodb's storage system doesn't support transactions and we're still learning about its durability and recovery model, so I can't make any totally fair claims comparing us to them.
(I love talking about this stuff, so keep up the curiosity)