3 ms·

Great article! One cool thing about RocksDB it's actually even used in other KV databases such as Redis on Flash https://redis.com/blog/hood-redis-enterprise-fl

Great article! One cool thing about RocksDB it's actually even used in other KV databases such as Redis on Flash https://redis.com/blog/hood-redis-enterprise-flash-database-architecture/ https://redis.com/blog/hood-redis-enterprise-flash-database-...

eatonphil 4y ago
Yup, FB's ZippyDB [0] is another example mentioned in the article. [0] https://engineering.fb.com/2021/08/06/core-data/zippydb/ https://engineering.fb.com/2021/08/06/core-data/zippydb/ Edit: I've added Redis Enterprise Flash to the list now. Thanks!
dboreham 4y ago
The article misses the point. All data storage and query systems end up architected in layers. Upper layers deal with higher abstractions (objects, rows, whatever). Lower layers deal with simpler functions, closer to the hardware. The upper layers are consumers of the lower layers. This is where "embedded KV stores" like LevelDB, RocksDB, etc come from. They began as the embedded storage layer for some bigger thing. Every product you think of as a database or document store is built like this, including MySQL and PostgreSQL and Oracle. Such a storage layer, shipped as an independent library, is how you (or anyone) builds your own database-ish thing. That's what the article should say. The list of examples are odd. For instance MongoRocks is cited for using RocksDB, but actual stock MongoDB uses Wired Tiger, which isn't mentioned. Disclosure: I played a part in the late-beginning of this space when Netscape funded Sleepycat to develop BerkeleyDB. dbm and ndbm existed beforehand, but BerkeleyDB used in LDAP servers is I think the genesis point for this pattern as it exists today.
- eatonphil 4y ago
  If there's a difference between what you wrote and what I wrote I'm missing it. But you're also welcome to write your own post. :)
  - morelisp 4y ago
    I do feel like there's a historical perspective missing from the article which the GP touches on. Embedded KV stores aren't new (although some of the algorithms behind the current crop certainly are). They used to dominate "backend" software development until their popularity waned as the world got obsessed with "model the domain, damn the computation cost" (because all resources were doubling or more yearly) followed by "we'll just distribute it". The need for parallelism killed the first approach and the cost of increasingly complex reduce steps killed the second. Now we're back to "how much can we fit in RAM on a local machine" and it turns out, if you can still bang bits for smart key formats, a hell of a lot.
- galaxyLogic 4y ago
  > Upper layers deal with higher abstractions (objects, rows, whatever) Right, I'm waiting for standard for a level above relational databases which is Object-databases. I know there are several ones already and there are Object-Relational mapping layers. I think the key point there is that Object databases are a level ABOVE relational databases. They are not "better" but they deal with the higher level of objects rather than "tables", just like relational databases can be seen to be are a level above key-value -stores. I would like Object databases to become better and easier to use and more standardized. I think there is value in being able to see both level, the objects, and the relational data that makes up the objects.
  - morelisp 4y ago
    Neither objects nor relations are "above" the other. You can map them in a vacuous mathematical sense, but it's a massively leaky abstraction in either direction.
    - eatonphil 4y ago
      Some concrete examples: 1. Yugabyte's relational query layer sits on top of a document store (DocDB): https://www.yugabyte.com/blog/how-we-built-a-high-performance-document-store-on-rocksdb/ https://www.yugabyte.com/blog/how-we-built-a-high-performanc.... 2. You can put documents in a PostgreSQL JSON(B) column.
    - galaxyLogic 4y ago
      When I use the word "above" I mean "layers" of code. So if an Object-database was implemented by using a relational database, it would be "above" the layer of the RDBMs. I think that is what object-to-relational mappers like Hibernate do. I think it would seem quite natural to implement objects on top of, with the help of an RDBMS. But not sure if the opposite is true.
- nicholasjarnold 4y ago
  > They began as the embedded storage layer for some bigger thing. I immediately thought of Kafka's streaming query stuff when I read the headline (ksqlDB). I'm not sure if that's the origin story of RocksDB, but it's the storage engine underlying that streaming query tooling in Kafka's ecosystem.

eatonphil 4y ago

Yup, FB's ZippyDB [0] is another example mentioned in the article. [0] https://engineering.fb.com/2021/08/06/core-data/zippydb/ https://engineering.fb.com/2021/08/06/core-data/zippydb/ Edit: I've added Redis Enterprise Flash to the list now. Thanks!

dboreham 4y ago

The article misses the point. All data storage and query systems end up architected in layers. Upper layers deal with higher abstractions (objects, rows, whatever). Lower layers deal with simpler functions, closer to the hardware. The upper layers are consumers of the lower layers. This is where "embedded KV stores" like LevelDB, RocksDB, etc come from. They began as the embedded storage layer for some bigger thing. Every product you think of as a database or document store is built like this, including MySQL and PostgreSQL and Oracle. Such a storage layer, shipped as an independent library, is how you (or anyone) builds your own database-ish thing. That's what the article should say. The list of examples are odd. For instance MongoRocks is cited for using RocksDB, but actual stock MongoDB uses Wired Tiger, which isn't mentioned. Disclosure: I played a part in the late-beginning of this space when Netscape funded Sleepycat to develop BerkeleyDB. dbm and ndbm existed beforehand, but BerkeleyDB used in LDAP servers is I think the genesis point for this pattern as it exists today.

eatonphil 4y ago
If there's a difference between what you wrote and what I wrote I'm missing it. But you're also welcome to write your own post. :)
- morelisp 4y ago
  I do feel like there's a historical perspective missing from the article which the GP touches on. Embedded KV stores aren't new (although some of the algorithms behind the current crop certainly are). They used to dominate "backend" software development until their popularity waned as the world got obsessed with "model the domain, damn the computation cost" (because all resources were doubling or more yearly) followed by "we'll just distribute it". The need for parallelism killed the first approach and the cost of increasingly complex reduce steps killed the second. Now we're back to "how much can we fit in RAM on a local machine" and it turns out, if you can still bang bits for smart key formats, a hell of a lot.
galaxyLogic 4y ago
> Upper layers deal with higher abstractions (objects, rows, whatever) Right, I'm waiting for standard for a level above relational databases which is Object-databases. I know there are several ones already and there are Object-Relational mapping layers. I think the key point there is that Object databases are a level ABOVE relational databases. They are not "better" but they deal with the higher level of objects rather than "tables", just like relational databases can be seen to be are a level above key-value -stores. I would like Object databases to become better and easier to use and more standardized. I think there is value in being able to see both level, the objects, and the relational data that makes up the objects.
- morelisp 4y ago
  Neither objects nor relations are "above" the other. You can map them in a vacuous mathematical sense, but it's a massively leaky abstraction in either direction.
  - eatonphil 4y ago
    Some concrete examples: 1. Yugabyte's relational query layer sits on top of a document store (DocDB): https://www.yugabyte.com/blog/how-we-built-a-high-performance-document-store-on-rocksdb/ https://www.yugabyte.com/blog/how-we-built-a-high-performanc.... 2. You can put documents in a PostgreSQL JSON(B) column.
  - galaxyLogic 4y ago
    When I use the word "above" I mean "layers" of code. So if an Object-database was implemented by using a relational database, it would be "above" the layer of the RDBMs. I think that is what object-to-relational mappers like Hibernate do. I think it would seem quite natural to implement objects on top of, with the help of an RDBMS. But not sure if the opposite is true.
nicholasjarnold 4y ago
> They began as the embedded storage layer for some bigger thing. I immediately thought of Kafka's streaming query stuff when I read the headline (ksqlDB). I'm not sure if that's the origin story of RocksDB, but it's the storage engine underlying that streaming query tooling in Kafka's ecosystem.