4 ms·

You don't, if you're smart. http://chip.typepad.com/weblog/2011/08/why-cassandra-is-unfit-for-production.html http://chip.typepad.com/weblog/2011/08/why-cassan

by grout 15y ago

You don't, if you're smart. http://chip.typepad.com/weblog/2011/08/why-cassandra-is-unfit-for-production.html http://chip.typepad.com/weblog/2011/08/why-cassandra-is-unfi...

jbellis 15y ago
That boils down to Chip's opinion that master/slave is the One True Way to do replication, despite its many demonstrated disadvantages. I see your Chip Salzenberg and raise you a Werner Vogels: http://www.allthingsdistributed.com/2008/12/eventually_consistent.html http://www.allthingsdistributed.com/2008/12/eventually_consi...
- grout 15y ago
  sigh And here your PR guy wanted to fly me to New York to talk to you. Obviously the effort would have been wasted. Replication events are too important to be designed to fall on the floor. I want at any given time to know approximately how far behind replication is; I want primary write events to block until replication is at least durably scheduled; and if I pause or slow operation and wait for replication to catch up, I want to know that it _is_ caught up without holes. Cassandra fails utterly to meet these minimal requirements. Master/slave queue is not the only way to meet these needs, but unless a replacement can fulfill the requirements, I can't responsibly switch.
  - jbellis 15y ago
    When I'm learning about a new architecture, I like to take the position of, "let's assume the authors aren't idiots. If they're not, why would they have designed things this way?" With that in mind, let me pose this question to you. Is using a Dynamo architecture (http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html http://www.allthingsdistributed.com/2007/10/amazons_dynamo.h...) for S3 "irresponsible" of Amazon? I submit that by this point, Amazon (among others) has convincingly demonstrated that this approach can indeed achieve a high degree of reliability. If you agree, then I suggest that you read through the Dynamo and Eventually Consistent papers again with the "let's assume these people aren't idiots" approach, and see if you can spot what this architecture offers to achieve a similar goal to your "wait for replication to catch up" design.
    - grout 15y ago
      You're not idiots, and neither are Amazon. But using a Dynamo style design safely requires overprovisioning and performance loss. W=1 speed is your bait; reality is the switch.
  - nicktelford 15y ago
    Your primary argument seems to be that in the event a "replication event" fails, the data is lost forever - this is simply not true. I the event a replica is unavailable for a write, the co-ordinator stores that write itself and delivers it to the replica once it becomes available again (Hinted Handoff, see the oft linked paper on eventual consistency). This makes read-repair far less of an issue, usually only as a mechanism to ensure consistency of requests that occur in the window between the node becoming available and the hint being delivered. It's called eventual consistency for a reason. Writes don't just go missing. If you're uncomfortable with the "eventual" aspect of the replication model then you're better off with a database that sacrifices availability for improved consistency guarantees.
    - grout 15y ago
      Node failure is not the failure case in question. As long as the node is up, hinted handoff can't play any part.
rbranson 15y ago
That's a 24-hour-news-network-pundit grade straw man he's created there. The replication strategy used by Cassandra is sourced from the Amazon Dynamo design (shared with Riak and Voldemort) and is intentional. It's not a bug; it's the way it's supposed to operate. There are many large scale distributed systems built on this replication strategy that have excellent durability track records -- Amazon S3 is one of them. The back pressure argument is laughable. In Cassandra, a write consistency of greater than ONE will ensure the data is replicated to more than one node before acknowledging to the client.
- grout 15y ago
  You use argument from authority ("It's based on Dynamo, which is made by Amazon, so you know it's good!") while accusing me of a strawman for reporting my own personal experiences. Good thing there is no god, or you should have to watch out for lightning.
  - rbranson 15y ago
    Actually, this is empirical evidence of the validity of this approach gained through observation of successful, real-world implementations. I'm not saying it's good purely because it's from Amazon or because any one person said it was good, but that there are many successful large-scale systems built using these principles that do not suffer the durability issues from your hypothesis. While not proof in the scientific sense, this type of evidence is used as a basis for scientific theories that explain much of what we know about the world. However, I still look forward to your detailed, rigorous rebuttal of my arguments in the previous comment.

by grout 15y ago

You don't, if you're smart. http://chip.typepad.com/weblog/2011/08/why-cassandra-is-unfit-for-production.html http://chip.typepad.com/weblog/2011/08/why-cassandra-is-unfi...

jbellis 15y ago

That boils down to Chip's opinion that master/slave is the One True Way to do replication, despite its many demonstrated disadvantages. I see your Chip Salzenberg and raise you a Werner Vogels: http://www.allthingsdistributed.com/2008/12/eventually_consistent.html http://www.allthingsdistributed.com/2008/12/eventually_consi...

grout 15y ago
sigh And here your PR guy wanted to fly me to New York to talk to you. Obviously the effort would have been wasted. Replication events are too important to be designed to fall on the floor. I want at any given time to know approximately how far behind replication is; I want primary write events to block until replication is at least durably scheduled; and if I pause or slow operation and wait for replication to catch up, I want to know that it _is_ caught up without holes. Cassandra fails utterly to meet these minimal requirements. Master/slave queue is not the only way to meet these needs, but unless a replacement can fulfill the requirements, I can't responsibly switch.
- jbellis 15y ago
  When I'm learning about a new architecture, I like to take the position of, "let's assume the authors aren't idiots. If they're not, why would they have designed things this way?" With that in mind, let me pose this question to you. Is using a Dynamo architecture (http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html http://www.allthingsdistributed.com/2007/10/amazons_dynamo.h...) for S3 "irresponsible" of Amazon? I submit that by this point, Amazon (among others) has convincingly demonstrated that this approach can indeed achieve a high degree of reliability. If you agree, then I suggest that you read through the Dynamo and Eventually Consistent papers again with the "let's assume these people aren't idiots" approach, and see if you can spot what this architecture offers to achieve a similar goal to your "wait for replication to catch up" design.
  - grout 15y ago
    You're not idiots, and neither are Amazon. But using a Dynamo style design safely requires overprovisioning and performance loss. W=1 speed is your bait; reality is the switch.
- nicktelford 15y ago
  Your primary argument seems to be that in the event a "replication event" fails, the data is lost forever - this is simply not true. I the event a replica is unavailable for a write, the co-ordinator stores that write itself and delivers it to the replica once it becomes available again (Hinted Handoff, see the oft linked paper on eventual consistency). This makes read-repair far less of an issue, usually only as a mechanism to ensure consistency of requests that occur in the window between the node becoming available and the hint being delivered. It's called eventual consistency for a reason. Writes don't just go missing. If you're uncomfortable with the "eventual" aspect of the replication model then you're better off with a database that sacrifices availability for improved consistency guarantees.
  - grout 15y ago
    Node failure is not the failure case in question. As long as the node is up, hinted handoff can't play any part.

rbranson 15y ago

That's a 24-hour-news-network-pundit grade straw man he's created there. The replication strategy used by Cassandra is sourced from the Amazon Dynamo design (shared with Riak and Voldemort) and is intentional. It's not a bug; it's the way it's supposed to operate. There are many large scale distributed systems built on this replication strategy that have excellent durability track records -- Amazon S3 is one of them. The back pressure argument is laughable. In Cassandra, a write consistency of greater than ONE will ensure the data is replicated to more than one node before acknowledging to the client.

grout 15y ago
You use argument from authority ("It's based on Dynamo, which is made by Amazon, so you know it's good!") while accusing me of a strawman for reporting my own personal experiences. Good thing there is no god, or you should have to watch out for lightning.
- rbranson 15y ago
  Actually, this is empirical evidence of the validity of this approach gained through observation of successful, real-world implementations. I'm not saying it's good purely because it's from Amazon or because any one person said it was good, but that there are many successful large-scale systems built using these principles that do not suffer the durability issues from your hypothesis. While not proof in the scientific sense, this type of evidence is used as a basis for scientific theories that explain much of what we know about the world. However, I still look forward to your detailed, rigorous rebuttal of my arguments in the previous comment.