4 ms·

We have only ever compared LMDB in synchronous mode to other DBs in synchronous mode, and LMDB in asynch mode to other DBs in asynch mode. Come on, that's too o

We have only ever compared LMDB in synchronous mode to other DBs in synchronous mode, and LMDB in asynch mode to other DBs in asynch mode. Come on, that's too obvious. And LMDB beats the crap out of SQLite, in any mode. http://www.lmdb.tech/bench/microbench/ http://www.lmdb.tech/bench/microbench/ Replacing SQLite's Btree engine with LMDB makes the SQLite footprint smaller, faster, and more reliable too. https://github.com/LMDB/sqlightning https://github.com/LMDB/sqlightning

jimis 8y ago
It's not that simple. What you mean by saying "(a)synchronous mode" is very different from database to database. See my above comment. Is SQLite synchronous or asynchronous if you configure it as journal_mode=WAL and PRAGMA synchronous=NORMAL? For my purposes as an application developer, I care about comparing databases operating in safe mode i.e. a system crash should never cause total data loss. According to my experience SQLite's safe mode is many times faster than LMDB's safe mode, with a write workload. While LMDB is thrashing the disk and achieves only a handful of write transactions per second.
- hyc_symas 8y ago
  SQLite's safe mode is not comparable to LMDB's; SQLite is vulnerable to silent data loss in a crash. https://wisdom.cs.wisc.edu/workshops/spring-14/talks/Thanu.pdf https://wisdom.cs.wisc.edu/workshops/spring-14/talks/Thanu.p... LMDB is not.
  - jimis 8y ago
    Isn't this from the paper "all filesystems are not created equal"? [1] If you search the tables for "sqlite-wal" you will see that it shows zero vulnerabilities. [1] https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-pillai.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14...
    - hyc_symas 8y ago
      Ah, the earlier result was for SQLite-rollback, their default mode. IMO any DB should default to its safest mode.
jimis 8y ago
> And LMDB beats the crap out of SQLite, in any mode. http://www.lmdb.tech/bench/microbench/ http://www.lmdb.tech/bench/microbench/ Alright, I went through this page to see why my experience is different. While it mentions that they enabled SQLite's WAL journal, it also mentions that the synchronous writes were performed with PRAGMA synchronous=FULL. I believe that if they set PRAGMA synchronous=NORMAL, they will get an ACI-reliable database that is way faster than LMDB with write-to-disk workloads. Most of the measurements on that page are not really useful to me; they do not persist the data (db on tmpfs) or they do not care about reliability, using dangerous settings. Only few measurements are both persisting the data and writing safely, but the SQLite configuration is sub-optimal for them.

jimis 8y ago

It's not that simple. What you mean by saying "(a)synchronous mode" is very different from database to database. See my above comment. Is SQLite synchronous or asynchronous if you configure it as journal_mode=WAL and PRAGMA synchronous=NORMAL? For my purposes as an application developer, I care about comparing databases operating in safe mode i.e. a system crash should never cause total data loss. According to my experience SQLite's safe mode is many times faster than LMDB's safe mode, with a write workload. While LMDB is thrashing the disk and achieves only a handful of write transactions per second.

hyc_symas 8y ago
SQLite's safe mode is not comparable to LMDB's; SQLite is vulnerable to silent data loss in a crash. https://wisdom.cs.wisc.edu/workshops/spring-14/talks/Thanu.pdf https://wisdom.cs.wisc.edu/workshops/spring-14/talks/Thanu.p... LMDB is not.
- jimis 8y ago
  Isn't this from the paper "all filesystems are not created equal"? [1] If you search the tables for "sqlite-wal" you will see that it shows zero vulnerabilities. [1] https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-pillai.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14...
  - hyc_symas 8y ago
    Ah, the earlier result was for SQLite-rollback, their default mode. IMO any DB should default to its safest mode.

jimis 8y ago

> And LMDB beats the crap out of SQLite, in any mode. http://www.lmdb.tech/bench/microbench/ http://www.lmdb.tech/bench/microbench/ Alright, I went through this page to see why my experience is different. While it mentions that they enabled SQLite's WAL journal, it also mentions that the synchronous writes were performed with PRAGMA synchronous=FULL. I believe that if they set PRAGMA synchronous=NORMAL, they will get an ACI-reliable database that is way faster than LMDB with write-to-disk workloads. Most of the measurements on that page are not really useful to me; they do not persist the data (db on tmpfs) or they do not care about reliability, using dangerous settings. Only few measurements are both persisting the data and writing safely, but the SQLite configuration is sub-optimal for them.