12 ms·

I have already admitted my obvious biases, but seriously - when you design such a trivially corruptible system, you can't call it a persistent database; it's at

I have already admitted my obvious biases, but seriously - when you design such a trivially corruptible system, you can't call it a persistent database; it's at most a cache. It won't survive a system crash at all, it probably won't survive an application crash intact either. To call it persistent is laughable.

cbsmith 12y ago
> I have already admitted my obvious biases, but seriously - when you design such a trivially corruptible system, you can't call it a persistent database; it's at most a cache. It won't survive a system crash at all, it probably won't survive an application crash intact either. To call it persistent is laughable. There are a number of use cases where in the event of a node failure it is better to rebuild from a replica or a log. Statistically, the RAM on another host is actually more reliable than local storage. Additionally, the database does have sync'ing primitives that allow for a variety of persistence strategies... just not the traditional ACID strategy. In practice, there are lots of cases where the freedom to ignore transactional integrity is very handy, and yes, a cache would definitely be one of them.
- hyc_symas 12y ago
  If DB updates are not atomically visible, then no sync'ing strategy will protect you from corruption during a crash. So in other words, you're saying "MDBM is not a persistent database." Glad that's clear. Totally agree, there are probably lots of use cases for it. But persistent data store isn't one of them.
  - luckydude 12y ago
    MDBM is a hash, a pretty scalable one and fairly light weight. It is so fast that a common usage is to log updates to a log file and just rebuild the hash from that after a reboot. We've been doing that for years, works well for us. I'm guessing you are one of the people behind some other technology. Goody for you but do you really think you make your case by dissing anything else? If you have a solution that works for you, great. SGI, Yahoo, and other companies have found a use for MDBM. SGI was using it ~20 years ago and at the time there was nothing that came close to the same performance.
    - nieve 12y ago
      It doesn't mean he's wrong, but hyc_symas is the CTO of Symas who produce the Lightning memory-mapped database, so part of his business is in direct competition. He does dance around this a bit below, but I think how aggressively he's trashing MDBM with broad & unsupported statements in this thread doesn't really speak well of his company or his personaly integrity. You can have a technical disagreement without looking like you're spreading FUD in place of valid technical arguments. This isn't how you do that.
      - hyc_symas 12y ago
        I stated up front that I am totally biased. I provided multiple links backing up my statements. Nothing I've said here is unfounded. The only FUD here is advertising a piece of software as a high performance embedded database when in fact it's not suitable for such use on its own. The most viable use cases the authors have presented is when using MDBM as part of a larger distributed system such that the loss of a single DB instance isn't fatal. The above comment talks about restoring from a log, but the actual log mechanism isn't part of MDBM. I.e., MDBM is incomplete on its own and you must provide additional pieces in order to use it effectively. I'm not solely interested in promoting my own DB. If you read the On-Disk microbenchmarks I linked you'll see that there's a broad range of use cases where LMDB gets trounced by LevelDB and other LSMs. I'm interested in facts. From the original link: "On clean shutdown of the machine, all of the MDBM data will be flushed to disk. However, in cases like power-failure and hardware problems, it’s possible for data to be lost, and the resulting DB to be corrupted. MDBM includes a tool to check DB consistency. However, you should always have contingencies. One way or another this is some form of redundancy…" The fact is, this is a system that can lose data on a crash and it doesn't include its own recovery mechanism. Without such a mechanism you can't call it a persistent data store because MDBM by itself is not persistent.
        cbsmith 12y ago
        > The fact is, this is a system that can lose data on a crash and it doesn't include its own recovery mechanism. Without such a mechanism you can't call it a persistent data store because MDBM by itself is not persistent. Persistence != ACID By your definition, ext2 is not persistent. Give it a break.
  - cbsmith 12y ago
    > If DB updates are not atomically visible, then no sync'ing strategy will protect you from corruption during a crash. Who said anything about them not being atomically visible? Not being ACID doesn't mean "none of the above", and atomicity in an embedded database is practically a footnote in a distributed system anyway, because you're looking for atomicity on a much grander scale. You can impose whatever locking strategy you want around the memory to ensure updates are atomically visible. It's an embedded DB so consistency is in the hands of the application logic. Traditional ACID terms for isolation and durability aren't there, but for say a distributed system you'd do that in the network layer. If you have multiple replicas, if one node dies you don't try to recover its data (oddly, storage corruption and node failure have a high co-occurrence rate ;-), you just wipe and pull from the other nodes. If you can't get a clean, consistent state from a node that hasn't failed, you've got bigger problems than your embedded database. As I pointed out, pulling from other nodes is often faster and more reliable than pulling from disk anyway. In general, with a high performance network application, it is often preferable address these issues somewhere other than in your embedded database. So what is the corruption you are envisioning?

cbsmith 12y ago

> I have already admitted my obvious biases, but seriously - when you design such a trivially corruptible system, you can't call it a persistent database; it's at most a cache. It won't survive a system crash at all, it probably won't survive an application crash intact either. To call it persistent is laughable. There are a number of use cases where in the event of a node failure it is better to rebuild from a replica or a log. Statistically, the RAM on another host is actually more reliable than local storage. Additionally, the database does have sync'ing primitives that allow for a variety of persistence strategies... just not the traditional ACID strategy. In practice, there are lots of cases where the freedom to ignore transactional integrity is very handy, and yes, a cache would definitely be one of them.

hyc_symas 12y ago
If DB updates are not atomically visible, then no sync'ing strategy will protect you from corruption during a crash. So in other words, you're saying "MDBM is not a persistent database." Glad that's clear. Totally agree, there are probably lots of use cases for it. But persistent data store isn't one of them.
- luckydude 12y ago
  MDBM is a hash, a pretty scalable one and fairly light weight. It is so fast that a common usage is to log updates to a log file and just rebuild the hash from that after a reboot. We've been doing that for years, works well for us. I'm guessing you are one of the people behind some other technology. Goody for you but do you really think you make your case by dissing anything else? If you have a solution that works for you, great. SGI, Yahoo, and other companies have found a use for MDBM. SGI was using it ~20 years ago and at the time there was nothing that came close to the same performance.
  - nieve 12y ago
    It doesn't mean he's wrong, but hyc_symas is the CTO of Symas who produce the Lightning memory-mapped database, so part of his business is in direct competition. He does dance around this a bit below, but I think how aggressively he's trashing MDBM with broad & unsupported statements in this thread doesn't really speak well of his company or his personaly integrity. You can have a technical disagreement without looking like you're spreading FUD in place of valid technical arguments. This isn't how you do that.
    - hyc_symas 12y ago
      I stated up front that I am totally biased. I provided multiple links backing up my statements. Nothing I've said here is unfounded. The only FUD here is advertising a piece of software as a high performance embedded database when in fact it's not suitable for such use on its own. The most viable use cases the authors have presented is when using MDBM as part of a larger distributed system such that the loss of a single DB instance isn't fatal. The above comment talks about restoring from a log, but the actual log mechanism isn't part of MDBM. I.e., MDBM is incomplete on its own and you must provide additional pieces in order to use it effectively. I'm not solely interested in promoting my own DB. If you read the On-Disk microbenchmarks I linked you'll see that there's a broad range of use cases where LMDB gets trounced by LevelDB and other LSMs. I'm interested in facts. From the original link: "On clean shutdown of the machine, all of the MDBM data will be flushed to disk. However, in cases like power-failure and hardware problems, it’s possible for data to be lost, and the resulting DB to be corrupted. MDBM includes a tool to check DB consistency. However, you should always have contingencies. One way or another this is some form of redundancy…" The fact is, this is a system that can lose data on a crash and it doesn't include its own recovery mechanism. Without such a mechanism you can't call it a persistent data store because MDBM by itself is not persistent.
      - cbsmith 12y ago
        > The fact is, this is a system that can lose data on a crash and it doesn't include its own recovery mechanism. Without such a mechanism you can't call it a persistent data store because MDBM by itself is not persistent. Persistence != ACID By your definition, ext2 is not persistent. Give it a break.
- cbsmith 12y ago
  > If DB updates are not atomically visible, then no sync'ing strategy will protect you from corruption during a crash. Who said anything about them not being atomically visible? Not being ACID doesn't mean "none of the above", and atomicity in an embedded database is practically a footnote in a distributed system anyway, because you're looking for atomicity on a much grander scale. You can impose whatever locking strategy you want around the memory to ensure updates are atomically visible. It's an embedded DB so consistency is in the hands of the application logic. Traditional ACID terms for isolation and durability aren't there, but for say a distributed system you'd do that in the network layer. If you have multiple replicas, if one node dies you don't try to recover its data (oddly, storage corruption and node failure have a high co-occurrence rate ;-), you just wipe and pull from the other nodes. If you can't get a clean, consistent state from a node that hasn't failed, you've got bigger problems than your embedded database. As I pointed out, pulling from other nodes is often faster and more reliable than pulling from disk anyway. In general, with a high performance network application, it is often preferable address these issues somewhere other than in your embedded database. So what is the corruption you are envisioning?