5 ms·

…I don't quite understand why you rebutted my points about Berkeley DB with points about LMDB. The only point you actually rebutted without referring to LMBD w

…I don't quite understand why you rebutted my points about Berkeley DB with points about LMDB. The only point you actually rebutted without referring to LMBD was regarding documentation. Yes, some parts of BDB are copiously documented. Others are not. How cursors interact (or, don't) with partitioning is not documented at all. Some settings are underdocumented (e.g. the cache size setting doesn't specify whether "gigabytes" means 10^9 or 2^30). What the cache priority levels mean is barely even discernible from the source code. The documentation for how to set parameters like the total number of mutexes and active transactions to allocate boils down to "guess and check". Why enabling MVCC causes transactions to linger for ages isn't explained anywhere, beside maybe one mailing list post from 2008. I'm glad to see that my experience with BDB is corroborated by that of the developer of BDB's poster child (despite your unsubstantiated claim that I'm "wrong on so many points"). That said, since you felt the need to claim it, I feel the need to ask how you can rectify the claim that "LMDB is so much faster than every other data management mechanism in existence, nothing else even comes close" with the fact that LMDB is Btree-based and thus must face significant performance challenges with both sequential synchronous random writes, and with random writes and deletes to/from databases whose working set does not fit in memory.

hyc_symas 11y ago
Many of your points were about embedded stores, categorically. LMDB is existence proof that as a category, embedded data stores are perfectly suitable for just about any purpose. Cache size: "All sizes are in powers-of-two" http://docs.oracle.com/cd/E17076_04/html/api_reference/C/set_cachesize_parameter.html http://docs.oracle.com/cd/E17076_04/html/api_reference/C/set... Cache priority levels: "The lowest priority: pages are the most likely to be discarded." http://docs.oracle.com/cd/E17076_04/html/api_reference/C/mempset_priority.html http://docs.oracle.com/cd/E17076_04/html/api_reference/C/mem... Come on.
- colanderman 11y ago
  Not a single sentence of mine has a subject other than "Berkeley DB" or "it". I'm completely baffled how you read anything categorical from my post. "More likely" is not documentation, when you need guarantees that certain pages are not evicted in favor of others. (If my memory of the source code serves me, these levels map to internal values that have some nonlinear relationship with some LRU cache parameter, with the highest priority effectively locking pages into cache, except for certain cases which I forget.) Also, the powers-of-two phrase is a recent addition: http://web.stanford.edu/class/cs276a/projects/docs/berkeleydb/api_c/env_set_cachesize.html http://web.stanford.edu/class/cs276a/projects/docs/berkeleyd...
  - hyc_symas 11y ago
    "Its embedded nature means that any "client" of a shared DB is free to deadlock the whole thing (if the client dies) or corrupt the database's structure (if the client has a memory bug). By the time you have a working robust multi-client system, you have completely reinvented half of what an RDBMS provides, since you will have pushed most functionality into a central server to avoid the lack of robustness its embedded nature brings," Sounds very much to me like you are claiming that embedded designs by their nature lack robustness. BDB's lack of robustness is due to its very specific design choices, not solely because it is an embedded data store.
    - colanderman 11y ago
      OK, except you concede even this point in LMDB's documentation, when running it in read-write mode: "Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database" I don't doubt LMDB is more robust than Berkeley DB even in that mode (it's hard not to be, since BDB will gladly corrupt itself if you mess up following its Byzantine C++ allocation patterns), but I think this one generalization I make is at worst not unarguably falsified.
      - hyc_symas 11y ago
        Hmmm... You have a point. I have to say though that the read-write mode was not part of LMDB's original design; it was added in response to a request from a RedHat engineer. So in its purest original form, LMDB was completely incorruptible. In practice we've also found that writable mmap is only a win when your entire DB resides in RAM. When the DB is larger than RAM it's actually slower than just using write(). (When you try to write to an mmap page, if it's not already resident the OS must page it in. This is a wasted page-in since we're simply going to overwrite the entire page.)
hyc_symas 11y ago
As for performance challenges "sequential synchronous random writes" doesn't make sense to me, what exactly do you mean? LMDB is read-optimized; for reads nothing else can touch it. For writes, there are other engines that may be faster some of the time. For sequential writes LMDB cheats. The cheat is described in my LMDB Internals presentations. Nothing else can touch its sequential write speed. For random writes - LMDB generally loses to other designs when records are smaller than 1/2 a DB page. For larger, LMDB wins, and the margin grows as record sizes grow.
- colanderman 11y ago
  "Sequential synchronous random writes" means that you issue writes with random keys in sequence, waiting for each to be synchronously written to durable storage before writing the next. This is an important metric in certain domains, and it is generally bound by the underlying hardware regardless of database architecture (though certain designs can make it much worse). How does LMDB outperform log-structured databases when writing page-sized objects with random keys when the working set is larger than RAM? I'm genuinely curious. Any other Btree-based database will perform at disk read seek rates, at best.
  - hyc_symas 11y ago
    OK, for your sequential synchronous random write case, LMDB is fair-to-middling. In fully-synchronous mode (which is the default) it takes 2 fsyncs per transaction, so at least 2 IOPS per write. Many other DBs (including SQLite, Berkeley, and Kyoto) take even more than 2 fsyncs per transaction, and are slower than LMDB. Some others take only 1, and are much faster in this workload. LMDB outperforms all other engines on larger-than-RAM DBs with page-sized objects because its zero-copy architecture doesn't waste any RAM. For other DBs you must dedicate some fraction of RAM to write buffers and page caches in order to obtain their maximum performance. This means the largest working set size they can keep in RAM is much smaller than the actual physical RAM. With LMDB there are no separate write buffers or caches, all of available RAM can be utilized for your working set. This is precisely the scenario tested here http://symas.com/mdb/hyperdex/ http://symas.com/mdb/hyperdex/ I'll note that in that test, LMDB performed at exactly the disk read seek rate; HyperLevelDB's LSM performed at much worse than that rate.

hyc_symas 11y ago

Many of your points were about embedded stores, categorically. LMDB is existence proof that as a category, embedded data stores are perfectly suitable for just about any purpose. Cache size: "All sizes are in powers-of-two" http://docs.oracle.com/cd/E17076_04/html/api_reference/C/set_cachesize_parameter.html http://docs.oracle.com/cd/E17076_04/html/api_reference/C/set... Cache priority levels: "The lowest priority: pages are the most likely to be discarded." http://docs.oracle.com/cd/E17076_04/html/api_reference/C/mempset_priority.html http://docs.oracle.com/cd/E17076_04/html/api_reference/C/mem... Come on.

colanderman 11y ago
Not a single sentence of mine has a subject other than "Berkeley DB" or "it". I'm completely baffled how you read anything categorical from my post. "More likely" is not documentation, when you need guarantees that certain pages are not evicted in favor of others. (If my memory of the source code serves me, these levels map to internal values that have some nonlinear relationship with some LRU cache parameter, with the highest priority effectively locking pages into cache, except for certain cases which I forget.) Also, the powers-of-two phrase is a recent addition: http://web.stanford.edu/class/cs276a/projects/docs/berkeleydb/api_c/env_set_cachesize.html http://web.stanford.edu/class/cs276a/projects/docs/berkeleyd...
- hyc_symas 11y ago
  "Its embedded nature means that any "client" of a shared DB is free to deadlock the whole thing (if the client dies) or corrupt the database's structure (if the client has a memory bug). By the time you have a working robust multi-client system, you have completely reinvented half of what an RDBMS provides, since you will have pushed most functionality into a central server to avoid the lack of robustness its embedded nature brings," Sounds very much to me like you are claiming that embedded designs by their nature lack robustness. BDB's lack of robustness is due to its very specific design choices, not solely because it is an embedded data store.
  - colanderman 11y ago
    OK, except you concede even this point in LMDB's documentation, when running it in read-write mode: "Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database" I don't doubt LMDB is more robust than Berkeley DB even in that mode (it's hard not to be, since BDB will gladly corrupt itself if you mess up following its Byzantine C++ allocation patterns), but I think this one generalization I make is at worst not unarguably falsified.
    - hyc_symas 11y ago
      Hmmm... You have a point. I have to say though that the read-write mode was not part of LMDB's original design; it was added in response to a request from a RedHat engineer. So in its purest original form, LMDB was completely incorruptible. In practice we've also found that writable mmap is only a win when your entire DB resides in RAM. When the DB is larger than RAM it's actually slower than just using write(). (When you try to write to an mmap page, if it's not already resident the OS must page it in. This is a wasted page-in since we're simply going to overwrite the entire page.)

hyc_symas 11y ago

As for performance challenges "sequential synchronous random writes" doesn't make sense to me, what exactly do you mean? LMDB is read-optimized; for reads nothing else can touch it. For writes, there are other engines that may be faster some of the time. For sequential writes LMDB cheats. The cheat is described in my LMDB Internals presentations. Nothing else can touch its sequential write speed. For random writes - LMDB generally loses to other designs when records are smaller than 1/2 a DB page. For larger, LMDB wins, and the margin grows as record sizes grow.

colanderman 11y ago
"Sequential synchronous random writes" means that you issue writes with random keys in sequence, waiting for each to be synchronously written to durable storage before writing the next. This is an important metric in certain domains, and it is generally bound by the underlying hardware regardless of database architecture (though certain designs can make it much worse). How does LMDB outperform log-structured databases when writing page-sized objects with random keys when the working set is larger than RAM? I'm genuinely curious. Any other Btree-based database will perform at disk read seek rates, at best.
- hyc_symas 11y ago
  OK, for your sequential synchronous random write case, LMDB is fair-to-middling. In fully-synchronous mode (which is the default) it takes 2 fsyncs per transaction, so at least 2 IOPS per write. Many other DBs (including SQLite, Berkeley, and Kyoto) take even more than 2 fsyncs per transaction, and are slower than LMDB. Some others take only 1, and are much faster in this workload. LMDB outperforms all other engines on larger-than-RAM DBs with page-sized objects because its zero-copy architecture doesn't waste any RAM. For other DBs you must dedicate some fraction of RAM to write buffers and page caches in order to obtain their maximum performance. This means the largest working set size they can keep in RAM is much smaller than the actual physical RAM. With LMDB there are no separate write buffers or caches, all of available RAM can be utilized for your working set. This is precisely the scenario tested here http://symas.com/mdb/hyperdex/ http://symas.com/mdb/hyperdex/ I'll note that in that test, LMDB performed at exactly the disk read seek rate; HyperLevelDB's LSM performed at much worse than that rate.