3 ms·

LevelDB support! Would love to see how it compares to Bitcask in terms of speed.

Ixiaus 15y ago
I imagine it won't be as fast? The cool thing about Bitcask is that all of the keys are in memory - I imagine that would also be beneficial with secondary indexes now supported... LevelDB seems mostly well suited for data that becomes (in terms of key size and number of keys) bigger than your RAM...
- dsl 15y ago
  I think it will be a welcome change for anyone who runs a decent sized Riak deployment. We are currently adding machines simply to increase available RAM in the cluster.
  - nirvana 15y ago
    Why not bring a node down, and then replace it with a node that has more RAM? Are you exceeding the size of a node you can supply (in terms of RAM) for your cluster? I'd be very curious to know a bit about the character of your data, the size of your cluster, etc. (I've only run test clusters at this point, so hearing from someone doing production work would be informative.)
    - dsl 15y ago
      Replacement vs. addition is a situational trade-off, but ultimately the problem remains that you need to bring more RAM to the party. My biggest RAM consumer stores historical data for a goods trading platform. Each trade is a unique key, with all the trade data being the value. Access speed is important, but not as critical as the other goodies I get from Riak (replication and automated rebalancing). Metadata is stored separately, but I hope to change that with Riak 1.0 secondary indexes.
- varworld 15y ago
  Secondary indexes are currently only supported on levelDB.
  - Ixiaus 15y ago
    Interesting - well, that would get me to choose leveldb then!
willbmoss 15y ago
Bitcask can guarantee one disk seek, whereas LevelDB will do one disk seek per level, so at least from that perspective, it can't be better. Level also has to look down the entire tree if a key is missing. This means inserts end up being more expensive than reads or updates (which are all just a hash lookup in Bitcask).
- nosequel 15y ago
  LevelDB is there as the replacement for those who are currently using Innostore as their backend and not for those who have a dataset that fits bitcask.
- fizx 15y ago
  "Bitcask can guarantee one disk seek, whereas LevelDB will do one disk seek per level, so at least from that perspective, it can't be better." Yep, this is a standard tradeoff. When you want your data to be iterable, you have to take the hit. In practice (I oversee a large cassandra cluster), this hit happens about ~1% of the time, which is either a lot, or a little, depending on your constraints. "Level also has to look down the entire tree if a key is missing." This is why Cassandra has a bloom filter on top of a very similar data store.

I imagine it won't be as fast? The cool thing about Bitcask is that all of the keys are in memory - I imagine that would also be beneficial with secondary indexes now supported... LevelDB seems mostly well suited for data that becomes (in terms of key size and number of keys) bigger than your RAM...

dsl 15y ago
I think it will be a welcome change for anyone who runs a decent sized Riak deployment. We are currently adding machines simply to increase available RAM in the cluster.
- nirvana 15y ago
  Why not bring a node down, and then replace it with a node that has more RAM? Are you exceeding the size of a node you can supply (in terms of RAM) for your cluster? I'd be very curious to know a bit about the character of your data, the size of your cluster, etc. (I've only run test clusters at this point, so hearing from someone doing production work would be informative.)
  - dsl 15y ago
    Replacement vs. addition is a situational trade-off, but ultimately the problem remains that you need to bring more RAM to the party. My biggest RAM consumer stores historical data for a goods trading platform. Each trade is a unique key, with all the trade data being the value. Access speed is important, but not as critical as the other goodies I get from Riak (replication and automated rebalancing). Metadata is stored separately, but I hope to change that with Riak 1.0 secondary indexes.
varworld 15y ago
Secondary indexes are currently only supported on levelDB.
- Ixiaus 15y ago
  Interesting - well, that would get me to choose leveldb then!

willbmoss 15y ago

Bitcask can guarantee one disk seek, whereas LevelDB will do one disk seek per level, so at least from that perspective, it can't be better. Level also has to look down the entire tree if a key is missing. This means inserts end up being more expensive than reads or updates (which are all just a hash lookup in Bitcask).

nosequel 15y ago
LevelDB is there as the replacement for those who are currently using Innostore as their backend and not for those who have a dataset that fits bitcask.
fizx 15y ago
"Bitcask can guarantee one disk seek, whereas LevelDB will do one disk seek per level, so at least from that perspective, it can't be better." Yep, this is a standard tradeoff. When you want your data to be iterable, you have to take the hit. In practice (I oversee a large cassandra cluster), this hit happens about ~1% of the time, which is either a lot, or a little, depending on your constraints. "Level also has to look down the entire tree if a key is missing." This is why Cassandra has a bloom filter on top of a very similar data store.