4 ms·

Why not boltdb?

renke1 9y ago
I was wondering too, but Badger doesn't offer transactions and stuff like that (on purpose). It seems to be more low-level in some regards. https://github.com/boltdb/bolt https://github.com/boltdb/bolt
libeclipse 9y ago
They're based on different technologies. Bolt uses B-Trees and Badger/Rocks/Leveldb use LSM trees. Bolt also uses insane amounts of RAM and writes get slower and slower as the size of the database increases. (Personal experience, don't have benchmarks. Take this at face value.)
- nemo1618 9y ago
  We've experienced these downsides as well, although it seems that boltdb's memory usage can be deceptive due to mmap'ing the db file.
zimbatm 9y ago
InfluxDB went from RocksDB (LSM) -> BoltDB (B+Tree) - > Custom (LSM again). Here is a pretty good writeup: https://docs.influxdata.com/influxdb/v0.9/concepts/storage_engine/ https://docs.influxdata.com/influxdb/v0.9/concepts/storage_e...
rakoo 9y ago
There are basically two usecases for LSM trees vs B+trees: - Either you have a lot of random new writes and not so many updates, in which case LSM trees will ingest new data as fast as the disk can store them - Or you care more about read performance, and LMDB and BoltDB will have more predictable (and arguably better) performance InfluxDB is a timeseries database, so they write a lot of stuff and don't even read all of it. As data gets older it can be pruned efficiently with an LSM-based design, not so easily with B+tree. Dgraph on the other hand seems to sit right in the middle as it wants to be a general purpose database so there's no easy winner here. Hopefully the choice was correct for most use cases.
mrjn 9y ago
(Badger author here) BoltDB is super slow. Doesn't even come close to the performance we need, which is why we decided to write Badger from scratch.
- piotrkaminski 9y ago
  I'm curious if you have any more details here, or comparable benchmarks to share. BoltDB uses B+ trees and you say that a B+ tree approach is worth investigating due to improvements in SSD random write performance, so does BoltDB falsify that hypothesis or do you think it's just not well implemented and the idea still has potential?
  - mrjn 9y ago
    We have tried with BoltDB. Its performance is really bad. It is just badly implemented, acquires a global mutex lock across all reads and writes. We wouldn't have written Badger if BoltDB worked for us. RocksDB performs much better and is the most popular and efficient KV store in the market, being used at both Google (Leveldb) and Facebook. Therefore, the benchmarks are against that. Without spending time generating benchmarks, I'll bet that Badger would beat BoltDB any day. The idea of using B+-trees with value log has potential. One would need to do some obvious optimizations to decrease the frequency of writes to disk. Because SSDs have to run garbage collection cycles, which can affect write performance in a long running task. But, I think it would make a great research project. And if it comes out to be better than our current approach of LSM tree in read-write performance, I'd switch in a heartbeat; because I think B+-tree might be a simpler design.
    - piotrkaminski 9y ago
      Very interesting, thanks for the answer! For the application and data structure I have in mind, I expect to need lots of range iteration over small values (<32 bytes?) and relatively infrequent writes, so Badger might not be a good fit. At a minimum sounds like I'd need to set up my own benchmarks.
      - 9y ago

Why not boltdb?

I was wondering too, but Badger doesn't offer transactions and stuff like that (on purpose). It seems to be more low-level in some regards. https://github.com/boltdb/bolt https://github.com/boltdb/bolt

libeclipse 9y ago

They're based on different technologies. Bolt uses B-Trees and Badger/Rocks/Leveldb use LSM trees. Bolt also uses insane amounts of RAM and writes get slower and slower as the size of the database increases. (Personal experience, don't have benchmarks. Take this at face value.)

nemo1618 9y ago
We've experienced these downsides as well, although it seems that boltdb's memory usage can be deceptive due to mmap'ing the db file.

zimbatm 9y ago

InfluxDB went from RocksDB (LSM) -> BoltDB (B+Tree) - > Custom (LSM again). Here is a pretty good writeup: https://docs.influxdata.com/influxdb/v0.9/concepts/storage_engine/ https://docs.influxdata.com/influxdb/v0.9/concepts/storage_e...

rakoo 9y ago

There are basically two usecases for LSM trees vs B+trees: - Either you have a lot of random new writes and not so many updates, in which case LSM trees will ingest new data as fast as the disk can store them - Or you care more about read performance, and LMDB and BoltDB will have more predictable (and arguably better) performance InfluxDB is a timeseries database, so they write a lot of stuff and don't even read all of it. As data gets older it can be pruned efficiently with an LSM-based design, not so easily with B+tree. Dgraph on the other hand seems to sit right in the middle as it wants to be a general purpose database so there's no easy winner here. Hopefully the choice was correct for most use cases.

mrjn 9y ago

(Badger author here) BoltDB is super slow. Doesn't even come close to the performance we need, which is why we decided to write Badger from scratch.

piotrkaminski 9y ago
I'm curious if you have any more details here, or comparable benchmarks to share. BoltDB uses B+ trees and you say that a B+ tree approach is worth investigating due to improvements in SSD random write performance, so does BoltDB falsify that hypothesis or do you think it's just not well implemented and the idea still has potential?
- mrjn 9y ago
  We have tried with BoltDB. Its performance is really bad. It is just badly implemented, acquires a global mutex lock across all reads and writes. We wouldn't have written Badger if BoltDB worked for us. RocksDB performs much better and is the most popular and efficient KV store in the market, being used at both Google (Leveldb) and Facebook. Therefore, the benchmarks are against that. Without spending time generating benchmarks, I'll bet that Badger would beat BoltDB any day. The idea of using B+-trees with value log has potential. One would need to do some obvious optimizations to decrease the frequency of writes to disk. Because SSDs have to run garbage collection cycles, which can affect write performance in a long running task. But, I think it would make a great research project. And if it comes out to be better than our current approach of LSM tree in read-write performance, I'd switch in a heartbeat; because I think B+-tree might be a simpler design.
  - piotrkaminski 9y ago
    Very interesting, thanks for the answer! For the application and data structure I have in mind, I expect to need lots of range iteration over small values (<32 bytes?) and relatively infrequent writes, so Badger might not be a good fit. At a minimum sounds like I'd need to set up my own benchmarks.
    - 9y ago