7 ms·

Eh, you're right on some and wrong on many other points. An embedded key-value store can be a good foundation for an RDBMS - or many other data models. The Open

Eh, you're right on some and wrong on many other points. An embedded key-value store can be a good foundation for an RDBMS - or many other data models. The OpenLDAP Project used BerkeleyDB for more than 15 years. BerkeleyDB architects still point to OpenLDAP as a reference standard for transactional BDB code. http://www.bozemanpass.com/services/bdb.html http://www.bozemanpass.com/services/bdb.html As the primary author of OpenLDAP's BDB support, and of the LMDB library which now supersedes BDB, I've got quite a deep perspective into this topic. BDB is deadlock-prone, no argument there. But that doesn't mean the embedded engine approach can't work. LMDB is deadlock-proof, and serves perfectly well as the engine for an RDBMS as well as the engine for a hierarchical DB (X.500/LDAP directory), or arbitrary graphs (http://sph.io/content/2faf http://sph.io/content/2faf , https://github.com/pietermartin/thundergraph https://github.com/pietermartin/thundergraph , etc...) etc. When you start with a model-less key/value store you can implement any other data model on top. LMDB's robustness is, in a word, flawless. It's crash-proof by design and multiple independent researchers have borne out the integrity of that design. (https://www.usenix.org/conference/osdi14/technical-sessions/presentation/pillai https://www.usenix.org/conference/osdi14/technical-sessions/... https://www.usenix.org/conference/osdi14/technical-sessions/presentation/zheng_mai https://www.usenix.org/conference/osdi14/technical-sessions/... ) Your assertion that you can't build a robust multi-client system in a lightweight embedded system is flat wrong. In fact, this approach is the only way to get maximum performance from a software system. LMDB is so much faster than every other data management mechanism in existence, nothing else even comes close. http://symas.com/mdb/#bench http://symas.com/mdb/#bench Crap tools, crap locking - these are certainly weaknesses in BDB's locking design. LMDB's locking design doesn't have these weaknesses. Querying the size of the DB is an O(1) operation, simply reading a few words out of the DB header. Underdocumentation - I have no idea what you're talking about here. To me, BDB was copiously documented, and I largely owe my understanding of transactional systems to the BDB documentation. Yes, BDB is complicated and has too many moving parts. LMDB doesn't have this flaw either. But I can't take anyone seriously who says BDB was underdocumented. C++ interface - I don't use C++, so no comment here. C >> C++. SQLite on BDB - yes, it's a joke. I've run it, and the performance is pretty awful. But we do far better. https://github.com/LMDB/sqlightning https://github.com/LMDB/sqlightning There are many positive and negative lessons in software design to be learned from Berkeley DB. It really started life as a test bed, for learning. In that, I think it succeeded admirably. Its authors were able to experiment with extensible hashing, B+trees, ARIES logging, and a multitude of other important software architecture and data management concepts. LMDB owes much of its design to lessons learned from BDB. LMDB's use of a read-only memory map by default means you can safely allow multi-client access without fearing corruption due to memory bugs. LMDB's exclusive use of mmap instead of complex application-level caching means you avoid all of the horrors of BDB's cache tuning, bloat driving into swap space, and other admin/tuning nightmares. LMDB's crash-proof persistent on-disk format means you don't need to worry about error-prone and inherently unreliable transaction logging mechanisms. All of these features are part of the LMDB design because of our experience working with BDB over the past 15+ years.

colanderman 11y ago
…I don't quite understand why you rebutted my points about Berkeley DB with points about LMDB. The only point you actually rebutted without referring to LMBD was regarding documentation. Yes, some parts of BDB are copiously documented. Others are not. How cursors interact (or, don't) with partitioning is not documented at all. Some settings are underdocumented (e.g. the cache size setting doesn't specify whether "gigabytes" means 10^9 or 2^30). What the cache priority levels mean is barely even discernible from the source code. The documentation for how to set parameters like the total number of mutexes and active transactions to allocate boils down to "guess and check". Why enabling MVCC causes transactions to linger for ages isn't explained anywhere, beside maybe one mailing list post from 2008. I'm glad to see that my experience with BDB is corroborated by that of the developer of BDB's poster child (despite your unsubstantiated claim that I'm "wrong on so many points"). That said, since you felt the need to claim it, I feel the need to ask how you can rectify the claim that "LMDB is so much faster than every other data management mechanism in existence, nothing else even comes close" with the fact that LMDB is Btree-based and thus must face significant performance challenges with both sequential synchronous random writes, and with random writes and deletes to/from databases whose working set does not fit in memory.
- hyc_symas 11y ago
  Many of your points were about embedded stores, categorically. LMDB is existence proof that as a category, embedded data stores are perfectly suitable for just about any purpose. Cache size: "All sizes are in powers-of-two" http://docs.oracle.com/cd/E17076_04/html/api_reference/C/set_cachesize_parameter.html http://docs.oracle.com/cd/E17076_04/html/api_reference/C/set... Cache priority levels: "The lowest priority: pages are the most likely to be discarded." http://docs.oracle.com/cd/E17076_04/html/api_reference/C/mempset_priority.html http://docs.oracle.com/cd/E17076_04/html/api_reference/C/mem... Come on.
  - colanderman 11y ago
    Not a single sentence of mine has a subject other than "Berkeley DB" or "it". I'm completely baffled how you read anything categorical from my post. "More likely" is not documentation, when you need guarantees that certain pages are not evicted in favor of others. (If my memory of the source code serves me, these levels map to internal values that have some nonlinear relationship with some LRU cache parameter, with the highest priority effectively locking pages into cache, except for certain cases which I forget.) Also, the powers-of-two phrase is a recent addition: http://web.stanford.edu/class/cs276a/projects/docs/berkeleydb/api_c/env_set_cachesize.html http://web.stanford.edu/class/cs276a/projects/docs/berkeleyd...

colanderman 11y ago

…I don't quite understand why you rebutted my points about Berkeley DB with points about LMDB. The only point you actually rebutted without referring to LMBD was regarding documentation. Yes, some parts of BDB are copiously documented. Others are not. How cursors interact (or, don't) with partitioning is not documented at all. Some settings are underdocumented (e.g. the cache size setting doesn't specify whether "gigabytes" means 10^9 or 2^30). What the cache priority levels mean is barely even discernible from the source code. The documentation for how to set parameters like the total number of mutexes and active transactions to allocate boils down to "guess and check". Why enabling MVCC causes transactions to linger for ages isn't explained anywhere, beside maybe one mailing list post from 2008. I'm glad to see that my experience with BDB is corroborated by that of the developer of BDB's poster child (despite your unsubstantiated claim that I'm "wrong on so many points"). That said, since you felt the need to claim it, I feel the need to ask how you can rectify the claim that "LMDB is so much faster than every other data management mechanism in existence, nothing else even comes close" with the fact that LMDB is Btree-based and thus must face significant performance challenges with both sequential synchronous random writes, and with random writes and deletes to/from databases whose working set does not fit in memory.

hyc_symas 11y ago
Many of your points were about embedded stores, categorically. LMDB is existence proof that as a category, embedded data stores are perfectly suitable for just about any purpose. Cache size: "All sizes are in powers-of-two" http://docs.oracle.com/cd/E17076_04/html/api_reference/C/set_cachesize_parameter.html http://docs.oracle.com/cd/E17076_04/html/api_reference/C/set... Cache priority levels: "The lowest priority: pages are the most likely to be discarded." http://docs.oracle.com/cd/E17076_04/html/api_reference/C/mempset_priority.html http://docs.oracle.com/cd/E17076_04/html/api_reference/C/mem... Come on.
- colanderman 11y ago
  Not a single sentence of mine has a subject other than "Berkeley DB" or "it". I'm completely baffled how you read anything categorical from my post. "More likely" is not documentation, when you need guarantees that certain pages are not evicted in favor of others. (If my memory of the source code serves me, these levels map to internal values that have some nonlinear relationship with some LRU cache parameter, with the highest priority effectively locking pages into cache, except for certain cases which I forget.) Also, the powers-of-two phrase is a recent addition: http://web.stanford.edu/class/cs276a/projects/docs/berkeleydb/api_c/env_set_cachesize.html http://web.stanford.edu/class/cs276a/projects/docs/berkeleyd...