6 ms·

Dynamo Systems Work Too Hard

78 points by jamesmiller5 13y ago

jeremiahjordan 13y ago
even if switch failures are rarer, couch at W=1 will silently drop data for network partition, dynamo at W=2 won't, how is the comparison at the end valid?
- damienkatz 13y ago
  "...if the client wanted true multi-node durability, then the write wouldn't have succeeded (the client would timeout waiting for replicas(s) to receive the update) and the client wouldn't unknowingly lose data."
  - jeremiahjordan 13y ago
    But that isn't W=1
    - damienkatz 13y ago
      Ok, you're right. I just changed it to W=2, and all the numbers are the same.
crb 13y ago
When your units of networking concern are "availability zones" (i.e. data centers) rather than just switches, wouldn't network failures now be more common than server failures?
- bscanlan 13y ago
  Agreed - the assumption is that the MTBF of a single network device is a relevant statistic is undoubtedly incorrect in anywhere but a hobbyist's network.
jbellis 13y ago
This misses the point. There are two main reasons why, when I was researching scalable databases, I primarily gravitated towards Dynamo-style replication (Cassandra, Voldemort, and at the time, Dynomite): - There is no such thing as failover. Dynamo replication takes node failure in stride. This is what you want for a robust system where "Network Partitions are Rare, Server Failures are Not." Not only does it prevent temporary unavailability during the failover, it rules out an entire class of difficult, edge-case bugs. (Which every master-election-and-failover system out there has been plagued with.) - It generalizes to multiple datacenters as easily as to multiple machines, allowing local latencies for reads AND writes, in contrast to master-based systems where you always have to hit the master (possibly cross-DC) for at least writes. (Couchbase is unusual in that it apparently forces read-from-master as well.) Cassandra has pushed this the farthest, allowing you to choose synchronous replication to local replicas and asynchronous to remote ones, for instance: http://www.datastax.com/docs/1.2/dml/data_consistency http://www.datastax.com/docs/1.2/dml/data_consistency /Cassandra project chair
- damienkatz 13y ago
  I'm not denying that Dynamo features more consistent availability, but it does so at either cost of temporal consistency or a much larger amount of resources. That's the point of the article, the tradeoffs are expensive and you can achieve effectively the same performance and availability with fewer resources. Dynamo may rule out a certain class of bugs, but that doesn't mean other systems must also have those bugs. And Couchbase also has multi-data center master/master capabilities, with any topology: chain, ring, hub and spoke or any combination therein.
  - threeseed 13y ago
    Can you possibly be a bit more specific when talking about 'resources' and 'much larger'. Makes it hard to evaluate if it is even an issue or not. Also can you elaborate more on the multi-DC part of CouchBase. Documentation seems to imply only supports two-DC replication and I find the "currently supports continuous replication of data" line to be odd. What are the future plans for CouchBase in this area ?
  - 13y ago
deleted 13y ago
[deleted]
pbailis 13y ago
There's at least one good reason for Dynamo's write-to-all and read-from-all mechanism: latency. What you've called 'W=2' in Couchbase is "write to master and at least one slave." Dynamo-style 'W=2' means "write to any two replicas." This can decrease tail latencies since you don't have to wait for the master--any two will do; similarly for 'R=2'. Indeed, Dynamo 'W=2, R=2' will incur more read load than master-based reads (at least double, but not necessarily triple, in your figures). So I think it's more accurately a trade-off between latency and server load. There can be big benefits to this redundant work. For example: http://www.bailis.org/blog/doing-redundant-work-to-speed-up-distributed-queries/ http://www.bailis.org/blog/doing-redundant-work-to-speed-up-... But don't take it from me: http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scale/fulltext http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scal... Anyway, I'm pretty sure CASSANDRA-4705 (https://issues.apache.org/jira/browse/CASSANDRA-4705 https://issues.apache.org/jira/browse/CASSANDRA-4705), which allows for Dean-style redundant requests, both decreases the read load (at least from the factor of N in your post) and should still reduce tail latency without compromising on semantics. I don't have skin in this game, but I'm pretty sure that the Dynamo engineers had a good idea of what they were doing. (That said, the regular [non-linearizable] semantics for R+W>N are sort of annoying compared to a master-slave system, but can be fixed with write-backs.)
- damienkatz 13y ago
  Good point. But "writes" are very fast, in our tests write latency is less than half read latency, so we can easily do master to slave replication within the SLA. But you point is correct, a Dynamo system is faster to achieve the same replication factor.
  - jbellis 13y ago
    I think it's worth elaborating that the primary advantage to the dynamo model is not in the best- or average- case, but when everything does not go as planned -- when the master gets behind, when ec2 network latency spikes, etc. Then "any two replicas" instead of "master plus one more" is much more robust.
  - banachtarski 13y ago
leef 13y ago
Despite the name you can't actually assume DynamoDB is based on the Dynamo paper architecture.
- mydynamo 13y ago
  Yeah, for all you know it might just be MySQL back there or something.
- bscanlan 13y ago
  "The original Dynamo design was based on a core set of strong distributed systems principles resulting in an ultra-scalable and highly reliable database system. Amazon DynamoDB, which is a new service, continues to build on these principles, and also builds on our years of experience with running non-relational databases and cloud services, such as Amazon SimpleDB and Amazon S3, at scale." http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....
gukjoon 13y ago
Great article, Damien. This idea that network partitions are exceedingly rare was the reason why ElasticSearch goes CA vs. the AP many other NoSQL datastores choose. http://elasticsearch-users.115913.n3.nabble.com/CAP-theorem-tp891925p894234.html http://elasticsearch-users.115913.n3.nabble.com/CAP-theorem-... Not only are network partitions rare, the most disastrous case where the cluster splits in half is even rarer. Usually, you have a small part of the cluster partition away. I hope people don't take this as a Dynamo vs. Couch discussion, because the relative importance of partition tolerance is a topic that spans all datastores that give up on ACID.
- pkolaczk 13y ago
  Network partitions are extremeley rare only for small clusters. For very large clusters or multi-datacenter clusters there is much more hardware than a single switch between servers. Then, the likelihood that something cuts off the whole room full of servers from the rest of your cluster is not something safe to neglect.
gigq 13y ago
This also exactly describes how HBase works. I've always preferred HBase to Cassandra for this exact reason. You put far less read load on your servers and you don't have to worry about most of the things on http://wiki.apache.org/cassandra/Operations http://wiki.apache.org/cassandra/Operations. Another benefit that is not mentioned is that with a master based system you can easily move who is responsible for the data if a server starts to hotspot. In Cassandra you have to use random key distribution because if you have a server hotspot then the only solution is to split the token ring which is an intensive operation that is hard to do while the server is under heavy load.
- mrphoebs 13y ago
  You could always just do a read=any on a cassandra cluster.
  - gigq 13y ago
    There are a few problems with read=any (or really read=one as any is only supported by writes). One issue is it will still send the request out to all the servers in the quorum so even if you don't have to wait on a quorum you still put the read load on them. The second and more important issue is that if you want consistent data the only way to use read=one is if you use write=all which means you would have no resilience to a server outage. So in a normal cassandra cluster where you have write=quorum a read=one can give you back no data. In my experience this happened frequently enough with long GC pauses on one of the nodes that it wasn't useful to use in production.
- linuxhansl 13y ago
  Yep. I hear the "master design is bad" argument all the time. From many angles it is a bad design, but from other's it is not. First off, it is simpler and easier grok and check for bugs. Debugging a running system is easier. It is also easier to implement different distribution strategies and failure/placementgroups, because that algorithms is centralized. If things go wrong it is easier to track where your data is. No gossip rings to converge. And it is easier to grow the cluster. /HBase committer here.
pkolaczk 13y ago
Why compare MTBF of a single network switch to MTBF of a node? Why not compare MTBF of a single network switch to MTBF of a single CPU or a motherboard? Unless you're talking about hobby-size network, there is usually much more between the nodes than a single network switch.

78 points by jamesmiller5 13y ago

even if switch failures are rarer, couch at W=1 will silently drop data for network partition, dynamo at W=2 won't, how is the comparison at the end valid?

damienkatz 13y ago
"...if the client wanted true multi-node durability, then the write wouldn't have succeeded (the client would timeout waiting for replicas(s) to receive the update) and the client wouldn't unknowingly lose data."
- jeremiahjordan 13y ago
  But that isn't W=1
  - damienkatz 13y ago
    Ok, you're right. I just changed it to W=2, and all the numbers are the same.

crb 13y ago

When your units of networking concern are "availability zones" (i.e. data centers) rather than just switches, wouldn't network failures now be more common than server failures?

bscanlan 13y ago
Agreed - the assumption is that the MTBF of a single network device is a relevant statistic is undoubtedly incorrect in anywhere but a hobbyist's network.

jbellis 13y ago

This misses the point. There are two main reasons why, when I was researching scalable databases, I primarily gravitated towards Dynamo-style replication (Cassandra, Voldemort, and at the time, Dynomite): - There is no such thing as failover. Dynamo replication takes node failure in stride. This is what you want for a robust system where "Network Partitions are Rare, Server Failures are Not." Not only does it prevent temporary unavailability during the failover, it rules out an entire class of difficult, edge-case bugs. (Which every master-election-and-failover system out there has been plagued with.) - It generalizes to multiple datacenters as easily as to multiple machines, allowing local latencies for reads AND writes, in contrast to master-based systems where you always have to hit the master (possibly cross-DC) for at least writes. (Couchbase is unusual in that it apparently forces read-from-master as well.) Cassandra has pushed this the farthest, allowing you to choose synchronous replication to local replicas and asynchronous to remote ones, for instance: http://www.datastax.com/docs/1.2/dml/data_consistency http://www.datastax.com/docs/1.2/dml/data_consistency /Cassandra project chair

damienkatz 13y ago
I'm not denying that Dynamo features more consistent availability, but it does so at either cost of temporal consistency or a much larger amount of resources. That's the point of the article, the tradeoffs are expensive and you can achieve effectively the same performance and availability with fewer resources. Dynamo may rule out a certain class of bugs, but that doesn't mean other systems must also have those bugs. And Couchbase also has multi-data center master/master capabilities, with any topology: chain, ring, hub and spoke or any combination therein.
- threeseed 13y ago
  Can you possibly be a bit more specific when talking about 'resources' and 'much larger'. Makes it hard to evaluate if it is even an issue or not. Also can you elaborate more on the multi-DC part of CouchBase. Documentation seems to imply only supports two-DC replication and I find the "currently supports continuous replication of data" line to be odd. What are the future plans for CouchBase in this area ?
- 13y ago

deleted 13y ago

[deleted]

pbailis 13y ago

There's at least one good reason for Dynamo's write-to-all and read-from-all mechanism: latency. What you've called 'W=2' in Couchbase is "write to master and at least one slave." Dynamo-style 'W=2' means "write to any two replicas." This can decrease tail latencies since you don't have to wait for the master--any two will do; similarly for 'R=2'. Indeed, Dynamo 'W=2, R=2' will incur more read load than master-based reads (at least double, but not necessarily triple, in your figures). So I think it's more accurately a trade-off between latency and server load. There can be big benefits to this redundant work. For example: http://www.bailis.org/blog/doing-redundant-work-to-speed-up-distributed-queries/ http://www.bailis.org/blog/doing-redundant-work-to-speed-up-... But don't take it from me: http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scale/fulltext http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scal... Anyway, I'm pretty sure CASSANDRA-4705 (https://issues.apache.org/jira/browse/CASSANDRA-4705 https://issues.apache.org/jira/browse/CASSANDRA-4705), which allows for Dean-style redundant requests, both decreases the read load (at least from the factor of N in your post) and should still reduce tail latency without compromising on semantics. I don't have skin in this game, but I'm pretty sure that the Dynamo engineers had a good idea of what they were doing. (That said, the regular [non-linearizable] semantics for R+W>N are sort of annoying compared to a master-slave system, but can be fixed with write-backs.)

damienkatz 13y ago
Good point. But "writes" are very fast, in our tests write latency is less than half read latency, so we can easily do master to slave replication within the SLA. But you point is correct, a Dynamo system is faster to achieve the same replication factor.
- jbellis 13y ago
  I think it's worth elaborating that the primary advantage to the dynamo model is not in the best- or average- case, but when everything does not go as planned -- when the master gets behind, when ec2 network latency spikes, etc. Then "any two replicas" instead of "master plus one more" is much more robust.
- banachtarski 13y ago

leef 13y ago

Despite the name you can't actually assume DynamoDB is based on the Dynamo paper architecture.

mydynamo 13y ago
Yeah, for all you know it might just be MySQL back there or something.
bscanlan 13y ago
"The original Dynamo design was based on a core set of strong distributed systems principles resulting in an ultra-scalable and highly reliable database system. Amazon DynamoDB, which is a new service, continues to build on these principles, and also builds on our years of experience with running non-relational databases and cloud services, such as Amazon SimpleDB and Amazon S3, at scale." http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....

gukjoon 13y ago

Great article, Damien. This idea that network partitions are exceedingly rare was the reason why ElasticSearch goes CA vs. the AP many other NoSQL datastores choose. http://elasticsearch-users.115913.n3.nabble.com/CAP-theorem-tp891925p894234.html http://elasticsearch-users.115913.n3.nabble.com/CAP-theorem-... Not only are network partitions rare, the most disastrous case where the cluster splits in half is even rarer. Usually, you have a small part of the cluster partition away. I hope people don't take this as a Dynamo vs. Couch discussion, because the relative importance of partition tolerance is a topic that spans all datastores that give up on ACID.

pkolaczk 13y ago
Network partitions are extremeley rare only for small clusters. For very large clusters or multi-datacenter clusters there is much more hardware than a single switch between servers. Then, the likelihood that something cuts off the whole room full of servers from the rest of your cluster is not something safe to neglect.

gigq 13y ago

This also exactly describes how HBase works. I've always preferred HBase to Cassandra for this exact reason. You put far less read load on your servers and you don't have to worry about most of the things on http://wiki.apache.org/cassandra/Operations http://wiki.apache.org/cassandra/Operations. Another benefit that is not mentioned is that with a master based system you can easily move who is responsible for the data if a server starts to hotspot. In Cassandra you have to use random key distribution because if you have a server hotspot then the only solution is to split the token ring which is an intensive operation that is hard to do while the server is under heavy load.

mrphoebs 13y ago
You could always just do a read=any on a cassandra cluster.
- gigq 13y ago
  There are a few problems with read=any (or really read=one as any is only supported by writes). One issue is it will still send the request out to all the servers in the quorum so even if you don't have to wait on a quorum you still put the read load on them. The second and more important issue is that if you want consistent data the only way to use read=one is if you use write=all which means you would have no resilience to a server outage. So in a normal cassandra cluster where you have write=quorum a read=one can give you back no data. In my experience this happened frequently enough with long GC pauses on one of the nodes that it wasn't useful to use in production.
linuxhansl 13y ago
Yep. I hear the "master design is bad" argument all the time. From many angles it is a bad design, but from other's it is not. First off, it is simpler and easier grok and check for bugs. Debugging a running system is easier. It is also easier to implement different distribution strategies and failure/placementgroups, because that algorithms is centralized. If things go wrong it is easier to track where your data is. No gossip rings to converge. And it is easier to grow the cluster. /HBase committer here.

pkolaczk 13y ago

Why compare MTBF of a single network switch to MTBF of a node? Why not compare MTBF of a single network switch to MTBF of a single CPU or a motherboard? Unless you're talking about hobby-size network, there is usually much more between the nodes than a single network switch.