4 ms·

Did you guys investigate any other choices such as DynamoDB, or HBase? I know that Facebook (inventors of Cassandra) have moved off of Cassandra over to HBase f

Did you guys investigate any other choices such as DynamoDB, or HBase? I know that Facebook (inventors of Cassandra) have moved off of Cassandra over to HBase for its back-end messaging services due to its inherent consistency problems.

jbellis 14y ago
"Consistency problems" is a red herring; Cassandra can be easily asked to sacrifice availability for consistency [on a per-request basis], if that's what you want: http://www.datastax.com/docs/1.1/dml/data_consistency http://www.datastax.com/docs/1.1/dml/data_consistency Facebook never used modern Apache Cassandra, so when they tasked a team of experienced Haddop/HDFS engineers to build Messaging, it was natural for them to choose HBase. But the things they've had to do to deal with the problems in that architecture (e.g., sharding into "cells" to deal with namenode SPOF) [1] make me think that they would have done better with Cassandra. [1] http://www.slideshare.net/brizzzdotcom/facebook-messages-hbase/23 http://www.slideshare.net/brizzzdotcom/facebook-messages-hba...
- dhruba 14y ago
  If you have three replicas in casandra and you want your reads to be consistent, you would have to read from at least two replicas . This doubles the number of iops to the disk subsystem. Most disk based storage systems are bottlenecked by random iops, and a solution that requires double-the-iops is a non-starter.
  - 3amOpsGuy 14y ago
    I don't think that's strictly true. With cassandra you choose where the extra reads (or writes) to ensure consistency occur as best suits your use case: * At write time (write quorum) like a traditional replicated datastore * At read time (read quorum) * In the background (read one, with a non-zero chance of read repair) This is quite nice to be able to bend Cassandra on a use case by use case basis (NB: these are not cluster wide approaches, for different columns / circumstances i can choose different patterns).
sbtourist 14y ago
We already have an HBase cluster in-house we use together with Hadoop for social analytics. By the way, we ruled it out as our operational experience with it was a little bit of a pain, in particular around version updates. Talking about DynamoDB, it is certainly attractive as it scales, is managed by Amazon and all the yadda-yadda, but we didn't want to be lock-in to Amazon, and costs were another concern.
sbtourist 14y ago
And talking about consistency, as already pointed out by Jonathan, Cassandra actually provides a way to allow for strong consistency: also, with its tunable consistency levels, it provides greater freedom to implement different access patterns depending on the use case, as described on the blog post.
- monstrado 14y ago
  As someone has mentioned, Cassandra is able to be strongly consistent at the cost of a pretty significant performance cut. I am curious just how significant this performance cut would be to get consistency at the level of something like DynamoDB or HBase which is advertised as strongly consistent. Afterwards I'd be interested to see what the performance numbers would look like for all three. I agree with you though, locking into Amazon is kind of a drawback :-(
  - sbtourist 14y ago
    I disagree here. First, in order to achieve strong consistency in a distributed system with replication enabled, you will always have to write to multiple replicas: provided the number of replicas is the same, whether you're using a consistent-hashing based replication (Cassandra) or a distributed file system (HBase) it really doesn't matter much, as I/O latencies (both network and disk related) will always play a large role here. That said, you can have different performances if you: 1) Change the replication factor: this is something you can do regardless of the actual system you're using, so it doesn't count. 2) Change the number of "acknowledged" writes/reads as a fraction of the replication factor: strongly consistent systems will always provide at least as much acknowledges as Cassandra's QUORUM, so no big win here. 3) Change the point where you acknowledge writes, either memory, disk or physical sync: this is about durability, and if you use different settings for different systems (i.e. Mongo VS Cassandra) it is apples and oranges. 4) Change your physical storage structure (provided same level of durability as explained above): different databases use different structures, Cassandra uses Memtables+SSTables, which I don't expect to be slower than HBase HDFS-based ones. So in the end, Cassandra is at least as fast as other distributed databases, even when working in strongly consistent fashion.

jbellis 14y ago

"Consistency problems" is a red herring; Cassandra can be easily asked to sacrifice availability for consistency [on a per-request basis], if that's what you want: http://www.datastax.com/docs/1.1/dml/data_consistency http://www.datastax.com/docs/1.1/dml/data_consistency Facebook never used modern Apache Cassandra, so when they tasked a team of experienced Haddop/HDFS engineers to build Messaging, it was natural for them to choose HBase. But the things they've had to do to deal with the problems in that architecture (e.g., sharding into "cells" to deal with namenode SPOF) [1] make me think that they would have done better with Cassandra. [1] http://www.slideshare.net/brizzzdotcom/facebook-messages-hbase/23 http://www.slideshare.net/brizzzdotcom/facebook-messages-hba...

dhruba 14y ago
If you have three replicas in casandra and you want your reads to be consistent, you would have to read from at least two replicas . This doubles the number of iops to the disk subsystem. Most disk based storage systems are bottlenecked by random iops, and a solution that requires double-the-iops is a non-starter.
- 3amOpsGuy 14y ago
  I don't think that's strictly true. With cassandra you choose where the extra reads (or writes) to ensure consistency occur as best suits your use case: * At write time (write quorum) like a traditional replicated datastore * At read time (read quorum) * In the background (read one, with a non-zero chance of read repair) This is quite nice to be able to bend Cassandra on a use case by use case basis (NB: these are not cluster wide approaches, for different columns / circumstances i can choose different patterns).

sbtourist 14y ago

We already have an HBase cluster in-house we use together with Hadoop for social analytics. By the way, we ruled it out as our operational experience with it was a little bit of a pain, in particular around version updates. Talking about DynamoDB, it is certainly attractive as it scales, is managed by Amazon and all the yadda-yadda, but we didn't want to be lock-in to Amazon, and costs were another concern.

sbtourist 14y ago

And talking about consistency, as already pointed out by Jonathan, Cassandra actually provides a way to allow for strong consistency: also, with its tunable consistency levels, it provides greater freedom to implement different access patterns depending on the use case, as described on the blog post.

monstrado 14y ago
As someone has mentioned, Cassandra is able to be strongly consistent at the cost of a pretty significant performance cut. I am curious just how significant this performance cut would be to get consistency at the level of something like DynamoDB or HBase which is advertised as strongly consistent. Afterwards I'd be interested to see what the performance numbers would look like for all three. I agree with you though, locking into Amazon is kind of a drawback :-(
- sbtourist 14y ago
  I disagree here. First, in order to achieve strong consistency in a distributed system with replication enabled, you will always have to write to multiple replicas: provided the number of replicas is the same, whether you're using a consistent-hashing based replication (Cassandra) or a distributed file system (HBase) it really doesn't matter much, as I/O latencies (both network and disk related) will always play a large role here. That said, you can have different performances if you: 1) Change the replication factor: this is something you can do regardless of the actual system you're using, so it doesn't count. 2) Change the number of "acknowledged" writes/reads as a fraction of the replication factor: strongly consistent systems will always provide at least as much acknowledges as Cassandra's QUORUM, so no big win here. 3) Change the point where you acknowledge writes, either memory, disk or physical sync: this is about durability, and if you use different settings for different systems (i.e. Mongo VS Cassandra) it is apples and oranges. 4) Change your physical storage structure (provided same level of durability as explained above): different databases use different structures, Cassandra uses Memtables+SSTables, which I don't expect to be slower than HBase HDFS-based ones. So in the end, Cassandra is at least as fast as other distributed databases, even when working in strongly consistent fashion.