4 ms·

BoringDB: A high-performance key-value store built on SQLite

4 points by vincent_s 3y ago

throwawayapples 3y ago
Here's what it's focused on, targeting Themelio cryptocurrency's requirements: > In a blockchain, the behavior we almost always want is for all transactions on the database to be serializable and for database integrity to be guaranteed — that is, a transaction either happens completely or not all, and crashes/power failures cannot cause inconsistency — but we do not need strong durability. We don’t need to ensure that a information saved to disk is still there if the system crashes immediately, since generally Themelio will tolerate “traveling back in time” a few seconds. > The combination of extremely robust serializability and integrity with eventual durability is hard to find. SQLite in WAL mode with PRAGMA synchronous = NORMAL is close, but directly using SQLite still has relatively poor performance with the extremely fragmented reads and writes Themelio nodes create. LMDB and friends simply give no option besides either synchronizing every transaction or leaving database integrity up to chance every time a power failure might happen. > ... We actively encourage use of BoringDB outside of Themelio; we think it’s the perfect choice whenever you need a single-process, embedded key-value database with extreme reliability but also reasonable write performance. It's written in Rust with a permissive license.
Rochus 3y ago
This article is from 2021. They indeed use an SQL database and SQL queries to implement a key-value store. The data is stored in a regular table (CREATE TABLE example (key BLOB PRIMARY KEY, value BLOB NOT NULL). It's very unlikely that this really achieves high-performance, given that the SQLite backend is actually a key-value store and the SQL interpreter runs on top of it as an intermediate layer. To avoid some performance-drawbacks caused by this architecture they added caching. They have even considered RocksDB and LMDB, but have rejected them with reasons that I can not relate. I have implemented a transactional key-value store myself for my project and information management tools fifteen years ago (before the other mentioned stores became available) based on the aforementioned SQLite backend (see https://github.com/rochus-keller/Udb https://github.com/rochus-keller/Udb and https://github.com/rochus-keller/Sdb https://github.com/rochus-keller/Sdb) which supports both robust integrity and high performance. Doing the same with the full SQLite just adds complexity and lowers performance with no added value.

4 points by vincent_s 3y ago

Here's what it's focused on, targeting Themelio cryptocurrency's requirements: > In a blockchain, the behavior we almost always want is for all transactions on the database to be serializable and for database integrity to be guaranteed — that is, a transaction either happens completely or not all, and crashes/power failures cannot cause inconsistency — but we do not need strong durability. We don’t need to ensure that a information saved to disk is still there if the system crashes immediately, since generally Themelio will tolerate “traveling back in time” a few seconds. > The combination of extremely robust serializability and integrity with eventual durability is hard to find. SQLite in WAL mode with PRAGMA synchronous = NORMAL is close, but directly using SQLite still has relatively poor performance with the extremely fragmented reads and writes Themelio nodes create. LMDB and friends simply give no option besides either synchronizing every transaction or leaving database integrity up to chance every time a power failure might happen. > ... We actively encourage use of BoringDB outside of Themelio; we think it’s the perfect choice whenever you need a single-process, embedded key-value database with extreme reliability but also reasonable write performance. It's written in Rust with a permissive license.

Rochus 3y ago

This article is from 2021. They indeed use an SQL database and SQL queries to implement a key-value store. The data is stored in a regular table (CREATE TABLE example (key BLOB PRIMARY KEY, value BLOB NOT NULL). It's very unlikely that this really achieves high-performance, given that the SQLite backend is actually a key-value store and the SQL interpreter runs on top of it as an intermediate layer. To avoid some performance-drawbacks caused by this architecture they added caching. They have even considered RocksDB and LMDB, but have rejected them with reasons that I can not relate. I have implemented a transactional key-value store myself for my project and information management tools fifteen years ago (before the other mentioned stores became available) based on the aforementioned SQLite backend (see https://github.com/rochus-keller/Udb https://github.com/rochus-keller/Udb and https://github.com/rochus-keller/Sdb https://github.com/rochus-keller/Sdb) which supports both robust integrity and high performance. Doing the same with the full SQLite just adds complexity and lowers performance with no added value.