3 ms·
How does it compare to BadgerDB/RocksDB/LevelDB? I see that it's using Arrow and Parquet of course, but the Sparse Index sounds very similar to LSM Tree like s
by caust1c 4y ago
How does it compare to BadgerDB/RocksDB/LevelDB? I see that it's using Arrow and Parquet of course, but the Sparse Index sounds very similar to LSM Tree like storage engines, except using something like a K-Way Merge algorithm and a heap structure to manage that somehow?
I'm more of an operator and user of these systems, so as an operator I care more about the usability than what's underneath, but also am reasonably skeptical of new databases since theres literally hundreds being written every year.
So what benefits does this structure and data format provide over classical LSM-like databases which are currently dominating the high-write-throughput embedded DB space?
- brancz 4y agoIt's closer to DuckDB rather than Badger/RocksDB/LevelDB, but similar in the sense that it is an embeddable database, not one that is operated standalone. It's not unlike an LSM tree, but the difference is that the leafs in the tree are not individual keys, but rather describe a range of values that are all read at once if read. So this allows high write throughput _and_ high read throughput, trading off mutability (which is ok for the Observability use cases we aim for it to fulfil). I think it's reasonable to be skeptical about new databases, if it helps, we worked on the Prometheus and Thanos project storage layers before we started the work on this, which now powers hundreds of thousands of monitoring stacks out there.
- caust1c 4y agoIt was my understanding that those databases do have sparse indexes, but admittedly I may have applied that assumption based on my majority experience with Clickhouse where it uses LSM Tree engines and also has a sparse index. Would it be correct to say this is like an embeddable clickhouse engine, minus the SQL interface and using Arrow and Parquet as the storage format?
- brancz 4y agoYes, that's correct! Plus the dynamic column feature, which we think is crucial for Observability-type workloads (from what we know only InfluxDB IOx supports a similar feature).