4 ms·
While Cassandra has some nice characteristics, there are a few things I've run into along the way. Don't expect to run a 3-node Cassandra cluster and get much
by mnutt 14y ago
While Cassandra has some nice characteristics, there are a few things I've run into along the way.
Don't expect to run a 3-node Cassandra cluster and get much out of it in terms of availability, in the way you might run a master/slave failover setup. It's somewhat obvious, but your Cassandra deployment can't just start with a couple of nodes and scale up as you run into bottlenecks. The number of nodes needed starts to add up quickly with a replication factor of > 1 and quorum reads. And while you might say "I'm ok with eventual consistency, let's just read from a single node," if you're not reading from multiple nodes, the data may never become consistent, from what I can tell.
And counters should be marked with a big warning "not for production use". Their performance isn't great, and it nosedives as the dataset grows. (each counter update involves a read + a write) Having a node reboot can sometimes cause counters to double. They seem like basically an afterthought.
- jbellis 14y agoPost author here. Your first paragraph is, bluntly, incorrect. Cassandra guarantees that data will always become consistent. This is automatic [1] for normal operation, including in the face of temporary failures. Permanent failures require running a "repair" process to rebuild the failed machine from other replicas [2]. I think you've also misunderstood how quorum works; it is a quorum of the replica count, which tends to stay constant over cluster lifetime, not machine count. You are right that the current counters are an afterthought. I linked in my concluding paragraph, where I talk about improvements for Cassandra, "A new design for distributed counters." [3] [1] http://www.datastax.com/dev/blog/modern-hinted-handoff http://www.datastax.com/dev/blog/modern-hinted-handoff [2] http://www.datastax.com/docs/1.2/operations/node_repair http://www.datastax.com/docs/1.2/operations/node_repair [3] https://issues.apache.org/jira/browse/CASSANDRA-4775 https://issues.apache.org/jira/browse/CASSANDRA-4775
- jmix 14y agoFalse -- if there are nodes being added or deleted from the system, Cassandra provides no guarantee of consistency. Two nodes might disagree on quorum membership and thus quorum accesses may fail to overlap, leading to inconsistency. The consistency claims are overblown.
- tylerhobbs 14y agoCassandra uses a technique known as consistent hashing to allow each node to independently determine what nodes are replicas for a given row. The process really just involves hashing the row key and then comparing the result to the token of each node in the cluster. If the hash falls in between a node's token and the token of the previous node in the ring, then that node is a replica for that row. There's not really any way that nodes can disagree on this. Given that nodes cannot disagree on the set of replicas for a row, quorums must (by definition, and the pigeon hole principal) overlap by at least one replica.
- jmix 14y agoImagine two nodes, both initiating updates to the same row. One has observed a number of node additions and deletions. The other has not. They will perform their updates on disjoint sets. > There's not really any way that nodes can disagree on this. Atomic, dynamic updates to a ring are a difficult thing to do. If I'm missing some additional mechanism that ensures agreement on ring membership, please explain.
- parasubvert 14y agoPlease enlighten us with a cluster database that enables guaranteed consistency with dynamic node membership. They all have quirks handling membership (unless you're looking at a shared disk setup).
- cmccabe 14y agoHBase.
- rescrv 14y agoCheckout HyperDex. We just released a new version, and it is indeed consistent as nodes join and fail.
- linuxhansl 14y agoIt's all about the tradeoffs you are taking. Cassandra favors availability over consistency. For many workloads that is desired for others it is not. HBase for example is always consistent, but if you lose a node unexpectedly some data is not available for some time.
- deleted 14y ago[deleted]
- taligent 14y agoWhat do you mean by "from what I can tell" ? Have you actually experienced this issue or just making it up ?
- mnutt 14y agoIt is what I have experienced. "from what I can tell" is allowing that perhaps I have missed something in the configuration that solves this.
- jmix 14y agoI have looked through the code to conclude that Cassandra's consistency claims are overblown.
- parasubvert 14y agoI've looked at your user profile to suggest you're a troll account. Please elaborate on the observations you have that led to this conclusion.
- deleted 14y ago[deleted]
- jmix 14y agoLet's not revert to ad hominems so quickly. If you looked through my submissions, you'll see that I explained the problem elsewhere. Here it is again: Imagine two nodes, both initiating updates to the same row. One has observed a number of node additions and deletions. The other has not. They will perform their updates on disjoint sets. Additions to the ring are not atomic, therefore quorums aren't.