3 ms·

Many of your points were about embedded stores, categorically. LMDB is existence proof that as a category, embedded data stores are perfectly suitable for just

Many of your points were about embedded stores, categorically. LMDB is existence proof that as a category, embedded data stores are perfectly suitable for just about any purpose. Cache size: "All sizes are in powers-of-two" http://docs.oracle.com/cd/E17076_04/html/api_reference/C/set_cachesize_parameter.html http://docs.oracle.com/cd/E17076_04/html/api_reference/C/set... Cache priority levels: "The lowest priority: pages are the most likely to be discarded." http://docs.oracle.com/cd/E17076_04/html/api_reference/C/mempset_priority.html http://docs.oracle.com/cd/E17076_04/html/api_reference/C/mem... Come on.

colanderman 11y ago
Not a single sentence of mine has a subject other than "Berkeley DB" or "it". I'm completely baffled how you read anything categorical from my post. "More likely" is not documentation, when you need guarantees that certain pages are not evicted in favor of others. (If my memory of the source code serves me, these levels map to internal values that have some nonlinear relationship with some LRU cache parameter, with the highest priority effectively locking pages into cache, except for certain cases which I forget.) Also, the powers-of-two phrase is a recent addition: http://web.stanford.edu/class/cs276a/projects/docs/berkeleydb/api_c/env_set_cachesize.html http://web.stanford.edu/class/cs276a/projects/docs/berkeleyd...
- hyc_symas 11y ago
  "Its embedded nature means that any "client" of a shared DB is free to deadlock the whole thing (if the client dies) or corrupt the database's structure (if the client has a memory bug). By the time you have a working robust multi-client system, you have completely reinvented half of what an RDBMS provides, since you will have pushed most functionality into a central server to avoid the lack of robustness its embedded nature brings," Sounds very much to me like you are claiming that embedded designs by their nature lack robustness. BDB's lack of robustness is due to its very specific design choices, not solely because it is an embedded data store.
  - colanderman 11y ago
    OK, except you concede even this point in LMDB's documentation, when running it in read-write mode: "Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database" I don't doubt LMDB is more robust than Berkeley DB even in that mode (it's hard not to be, since BDB will gladly corrupt itself if you mess up following its Byzantine C++ allocation patterns), but I think this one generalization I make is at worst not unarguably falsified.
    - hyc_symas 11y ago
      Hmmm... You have a point. I have to say though that the read-write mode was not part of LMDB's original design; it was added in response to a request from a RedHat engineer. So in its purest original form, LMDB was completely incorruptible. In practice we've also found that writable mmap is only a win when your entire DB resides in RAM. When the DB is larger than RAM it's actually slower than just using write(). (When you try to write to an mmap page, if it's not already resident the OS must page it in. This is a wasted page-in since we're simply going to overwrite the entire page.)

colanderman 11y ago

Not a single sentence of mine has a subject other than "Berkeley DB" or "it". I'm completely baffled how you read anything categorical from my post. "More likely" is not documentation, when you need guarantees that certain pages are not evicted in favor of others. (If my memory of the source code serves me, these levels map to internal values that have some nonlinear relationship with some LRU cache parameter, with the highest priority effectively locking pages into cache, except for certain cases which I forget.) Also, the powers-of-two phrase is a recent addition: http://web.stanford.edu/class/cs276a/projects/docs/berkeleydb/api_c/env_set_cachesize.html http://web.stanford.edu/class/cs276a/projects/docs/berkeleyd...

hyc_symas 11y ago
"Its embedded nature means that any "client" of a shared DB is free to deadlock the whole thing (if the client dies) or corrupt the database's structure (if the client has a memory bug). By the time you have a working robust multi-client system, you have completely reinvented half of what an RDBMS provides, since you will have pushed most functionality into a central server to avoid the lack of robustness its embedded nature brings," Sounds very much to me like you are claiming that embedded designs by their nature lack robustness. BDB's lack of robustness is due to its very specific design choices, not solely because it is an embedded data store.
- colanderman 11y ago
  OK, except you concede even this point in LMDB's documentation, when running it in read-write mode: "Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database" I don't doubt LMDB is more robust than Berkeley DB even in that mode (it's hard not to be, since BDB will gladly corrupt itself if you mess up following its Byzantine C++ allocation patterns), but I think this one generalization I make is at worst not unarguably falsified.
  - hyc_symas 11y ago
    Hmmm... You have a point. I have to say though that the read-write mode was not part of LMDB's original design; it was added in response to a request from a RedHat engineer. So in its purest original form, LMDB was completely incorruptible. In practice we've also found that writable mmap is only a win when your entire DB resides in RAM. When the DB is larger than RAM it's actually slower than just using write(). (When you try to write to an mmap page, if it's not already resident the OS must page it in. This is a wasted page-in since we're simply going to overwrite the entire page.)