5 ms·

> The RocksDB engine developed by Facebook engineers is one of the fastest, most compact and write-optimized storage engines available. Will it fix MongoDB's d

by tfb 11y ago

> The RocksDB engine developed by Facebook engineers is one of the fastest, most compact and write-optimized storage engines available. Will it fix MongoDB's data loss issues? I'm hoping that is what "write-optimized" is partly implying.

spimmy 11y ago
Which data loss issues? You're gonna have to be more specific. ;) It doesn't fix the election rollback issue, because that's handled way above the storage layer. It does solve a whole slew of storage engine related write issues though. No more "we flush to disk every 100ms and call it good".
jbooth 11y ago
The 'compact' and 'write-optimized' are probably to differentiate RocksDB from LMDB, which pretty thoroughly smokes it for read loads and has an arguably more useful transaction model (which it pays for by single-threading writes and having a little more write-amplification for small records).
- hyc_symas 11y ago
  RocksDB is also single-writer. http://rocksdb.org/blog/521/lock/ http://rocksdb.org/blog/521/lock/ "write-optimized" means they take great pains to turn all writes into sequential writes, to avoid random I/O seeks and get maximum I/O throughput to the storage device. Of course structuring data as they do makes their reads much slower. LMDB is read-optimized, and foregoes major effort at those types of write optimizations because, quite frankly, rotating storage media is going the way of the magnetic tape drive. Solid state storage is ubiquitous, and storage seek time just isn't an issue any more. (Literally and figuratively - HDDs are not going extinct; because of their capacity/$$ advantage they're still used for archival purposes. But everyone doing performance/latency-sensitive work has moved on to SSDs, Flash-based or otherwise.) "compact" doesn't make much sense. There's nothing compact about the RocksDB codebase. Over 121,000 lines of source code https://www.openhub.net/p/rocksdb https://www.openhub.net/p/rocksdb and over 20MB of object code. Compared to e.g. 7,000 lines of source for LMDB and 64KB of object code. https://www.openhub.net/p/MDB https://www.openhub.net/p/MDB
  - jbooth 11y ago
    I hear you RE: compact source code, and that as much as the benchmarks are why I use LMDB (thanks) and not Rocks when I have a need. I was under the impression that Rocks manages more compact storage, probably as another consequence of all those sequential writes being packed right next to each other, rather than LMDB's freelist-of-4k-pages model. Is that the case or was I misreading whatever mailing list I got that from? Don't get me wrong, I value not having compactions more than slightly less write amplification, just checking my understanding here.
    - hyc_symas 11y ago
      RocksDB is more compact storage-wise when records are small. Notice here http://symas.com/mdb/ondisk/ http://symas.com/mdb/ondisk/ that RocksDB space is smaller using 24 byte values, but same or larger at 96 byte values. By the time you get to 768 byte values, LMDB is smallest.
      - jbooth 11y ago
        Cool, thanks for the response and for writing lmdb!

by tfb 11y ago

spimmy 11y ago

Which data loss issues? You're gonna have to be more specific. ;) It doesn't fix the election rollback issue, because that's handled way above the storage layer. It does solve a whole slew of storage engine related write issues though. No more "we flush to disk every 100ms and call it good".

jbooth 11y ago

The 'compact' and 'write-optimized' are probably to differentiate RocksDB from LMDB, which pretty thoroughly smokes it for read loads and has an arguably more useful transaction model (which it pays for by single-threading writes and having a little more write-amplification for small records).

hyc_symas 11y ago
RocksDB is also single-writer. http://rocksdb.org/blog/521/lock/ http://rocksdb.org/blog/521/lock/ "write-optimized" means they take great pains to turn all writes into sequential writes, to avoid random I/O seeks and get maximum I/O throughput to the storage device. Of course structuring data as they do makes their reads much slower. LMDB is read-optimized, and foregoes major effort at those types of write optimizations because, quite frankly, rotating storage media is going the way of the magnetic tape drive. Solid state storage is ubiquitous, and storage seek time just isn't an issue any more. (Literally and figuratively - HDDs are not going extinct; because of their capacity/$$ advantage they're still used for archival purposes. But everyone doing performance/latency-sensitive work has moved on to SSDs, Flash-based or otherwise.) "compact" doesn't make much sense. There's nothing compact about the RocksDB codebase. Over 121,000 lines of source code https://www.openhub.net/p/rocksdb https://www.openhub.net/p/rocksdb and over 20MB of object code. Compared to e.g. 7,000 lines of source for LMDB and 64KB of object code. https://www.openhub.net/p/MDB https://www.openhub.net/p/MDB
- jbooth 11y ago
  I hear you RE: compact source code, and that as much as the benchmarks are why I use LMDB (thanks) and not Rocks when I have a need. I was under the impression that Rocks manages more compact storage, probably as another consequence of all those sequential writes being packed right next to each other, rather than LMDB's freelist-of-4k-pages model. Is that the case or was I misreading whatever mailing list I got that from? Don't get me wrong, I value not having compactions more than slightly less write amplification, just checking my understanding here.
  - hyc_symas 11y ago
    RocksDB is more compact storage-wise when records are small. Notice here http://symas.com/mdb/ondisk/ http://symas.com/mdb/ondisk/ that RocksDB space is smaller using 24 byte values, but same or larger at 96 byte values. By the time you get to 768 byte values, LMDB is smallest.
    - jbooth 11y ago
      Cool, thanks for the response and for writing lmdb!