6 ms·

MDBM guy here. Care to elaborate on what we got wrong?

by luckydude 12y ago

hyc_symas 12y ago
See below. https://news.ycombinator.com/item?id=8734356 https://news.ycombinator.com/item?id=8734356
- cbsmith 12y ago
  Umm... the lack of transactional integrity is part of the mdbm design. So it's only "careless" in the strictest sense of the term (the designers explicitly wanted to exploit not having to care about it). mdbm is certainly not without limitations, but is careful about its use of mmap to an extent that comparisons with MongoDB are laughable.
  - hyc_symas 12y ago
    I have already admitted my obvious biases, but seriously - when you design such a trivially corruptible system, you can't call it a persistent database; it's at most a cache. It won't survive a system crash at all, it probably won't survive an application crash intact either. To call it persistent is laughable.
    - cbsmith 12y ago
      > I have already admitted my obvious biases, but seriously - when you design such a trivially corruptible system, you can't call it a persistent database; it's at most a cache. It won't survive a system crash at all, it probably won't survive an application crash intact either. To call it persistent is laughable. There are a number of use cases where in the event of a node failure it is better to rebuild from a replica or a log. Statistically, the RAM on another host is actually more reliable than local storage. Additionally, the database does have sync'ing primitives that allow for a variety of persistence strategies... just not the traditional ACID strategy. In practice, there are lots of cases where the freedom to ignore transactional integrity is very handy, and yes, a cache would definitely be one of them.
      - hyc_symas 12y ago
        If DB updates are not atomically visible, then no sync'ing strategy will protect you from corruption during a crash. So in other words, you're saying "MDBM is not a persistent database." Glad that's clear. Totally agree, there are probably lots of use cases for it. But persistent data store isn't one of them.
        luckydude 12y ago
        MDBM is a hash, a pretty scalable one and fairly light weight. It is so fast that a common usage is to log updates to a log file and just rebuild the hash from that after a reboot. We've been doing that for years, works well for us. I'm guessing you are one of the people behind some other technology. Goody for you but do you really think you make your case by dissing anything else? If you have a solution that works for you, great. SGI, Yahoo, and other companies have found a use for MDBM. SGI was using it ~20 years ago and at the time there was nothing that came close to the same performance.
        nieve 12y ago
        It doesn't mean he's wrong, but hyc_symas is the CTO of Symas who produce the Lightning memory-mapped database, so part of his business is in direct competition. He does dance around this a bit below, but I think how aggressively he's trashing MDBM with broad & unsupported statements in this thread doesn't really speak well of his company or his personaly integrity. You can have a technical disagreement without looking like you're spreading FUD in place of valid technical arguments. This isn't how you do that.
        hyc_symas 12y ago
        I stated up front that I am totally biased. I provided multiple links backing up my statements. Nothing I've said here is unfounded. The only FUD here is advertising a piece of software as a high performance embedded database when in fact it's not suitable for such use on its own. The most viable use cases the authors have presented is when using MDBM as part of a larger distributed system such that the loss of a single DB instance isn't fatal. The above comment talks about restoring from a log, but the actual log mechanism isn't part of MDBM. I.e., MDBM is incomplete on its own and you must provide additional pieces in order to use it effectively. I'm not solely interested in promoting my own DB. If you read the On-Disk microbenchmarks I linked you'll see that there's a broad range of use cases where LMDB gets trounced by LevelDB and other LSMs. I'm interested in facts. From the original link: "On clean shutdown of the machine, all of the MDBM data will be flushed to disk. However, in cases like power-failure and hardware problems, it’s possible for data to be lost, and the resulting DB to be corrupted. MDBM includes a tool to check DB consistency. However, you should always have contingencies. One way or another this is some form of redundancy…" The fact is, this is a system that can lose data on a crash and it doesn't include its own recovery mechanism. Without such a mechanism you can't call it a persistent data store because MDBM by itself is not persistent.

MDBM guy here. Care to elaborate on what we got wrong?

hyc_symas 12y ago

See below. https://news.ycombinator.com/item?id=8734356 https://news.ycombinator.com/item?id=8734356

cbsmith 12y ago
Umm... the lack of transactional integrity is part of the mdbm design. So it's only "careless" in the strictest sense of the term (the designers explicitly wanted to exploit not having to care about it). mdbm is certainly not without limitations, but is careful about its use of mmap to an extent that comparisons with MongoDB are laughable.
- hyc_symas 12y ago
  I have already admitted my obvious biases, but seriously - when you design such a trivially corruptible system, you can't call it a persistent database; it's at most a cache. It won't survive a system crash at all, it probably won't survive an application crash intact either. To call it persistent is laughable.
  - cbsmith 12y ago
    > I have already admitted my obvious biases, but seriously - when you design such a trivially corruptible system, you can't call it a persistent database; it's at most a cache. It won't survive a system crash at all, it probably won't survive an application crash intact either. To call it persistent is laughable. There are a number of use cases where in the event of a node failure it is better to rebuild from a replica or a log. Statistically, the RAM on another host is actually more reliable than local storage. Additionally, the database does have sync'ing primitives that allow for a variety of persistence strategies... just not the traditional ACID strategy. In practice, there are lots of cases where the freedom to ignore transactional integrity is very handy, and yes, a cache would definitely be one of them.
    - hyc_symas 12y ago
      If DB updates are not atomically visible, then no sync'ing strategy will protect you from corruption during a crash. So in other words, you're saying "MDBM is not a persistent database." Glad that's clear. Totally agree, there are probably lots of use cases for it. But persistent data store isn't one of them.
      - luckydude 12y ago
        MDBM is a hash, a pretty scalable one and fairly light weight. It is so fast that a common usage is to log updates to a log file and just rebuild the hash from that after a reboot. We've been doing that for years, works well for us. I'm guessing you are one of the people behind some other technology. Goody for you but do you really think you make your case by dissing anything else? If you have a solution that works for you, great. SGI, Yahoo, and other companies have found a use for MDBM. SGI was using it ~20 years ago and at the time there was nothing that came close to the same performance.
        nieve 12y ago
        It doesn't mean he's wrong, but hyc_symas is the CTO of Symas who produce the Lightning memory-mapped database, so part of his business is in direct competition. He does dance around this a bit below, but I think how aggressively he's trashing MDBM with broad & unsupported statements in this thread doesn't really speak well of his company or his personaly integrity. You can have a technical disagreement without looking like you're spreading FUD in place of valid technical arguments. This isn't how you do that.
        hyc_symas 12y ago
        I stated up front that I am totally biased. I provided multiple links backing up my statements. Nothing I've said here is unfounded. The only FUD here is advertising a piece of software as a high performance embedded database when in fact it's not suitable for such use on its own. The most viable use cases the authors have presented is when using MDBM as part of a larger distributed system such that the loss of a single DB instance isn't fatal. The above comment talks about restoring from a log, but the actual log mechanism isn't part of MDBM. I.e., MDBM is incomplete on its own and you must provide additional pieces in order to use it effectively. I'm not solely interested in promoting my own DB. If you read the On-Disk microbenchmarks I linked you'll see that there's a broad range of use cases where LMDB gets trounced by LevelDB and other LSMs. I'm interested in facts. From the original link: "On clean shutdown of the machine, all of the MDBM data will be flushed to disk. However, in cases like power-failure and hardware problems, it’s possible for data to be lost, and the resulting DB to be corrupted. MDBM includes a tool to check DB consistency. However, you should always have contingencies. One way or another this is some form of redundancy…" The fact is, this is a system that can lose data on a crash and it doesn't include its own recovery mechanism. Without such a mechanism you can't call it a persistent data store because MDBM by itself is not persistent.