5 ms·

Stonebraker: Clarifications on the CAP Theorem and Data-Related Errors

25 points by ora600 16y ago

moonpolysoft 16y ago
Wherein Stonebraker misunderstands distributed systems engineering completely.
- ora600 16y ago
  I understood what he said as: Solving the problems of distributed systems is incredibly hard and not necessary if all you need is scalability and high availability. Better think of your problems in terms of engineering trade-offs and not distributed theory and understand what you are giving up and what this gains you. When you decide to give up consistency, your application can no longer assume consistency ever. Giving up availability in case of network partition means few extra minutes of downtime a year. I don't think he completely misunderstood distributed systems - I think he decided to completely side-step the entire field.
  - jhugg 16y ago
    Nice summary.
  - moonpolysoft 16y ago
    > Solving the problems of distributed systems is incredibly hard and not necessary if all you need is scalability and high availability. Once one scales beyond a single node the system becomes distributed. Then by definition one must deal with distributed systems problems in order to achieve scale beyond the capabilities of a single node. > When you decide to give up consistency, your application can no longer assume consistency ever. Giving up availability in case of network partition means few extra minutes of downtime a year. Depends on the application. Giving up availability might mean cascading failures throughout your entire application. For instance if the datastore is unavailable for writes then any kind of queueing systems built around the DB (a common design pattern) run the risk of overflow during the downtime. And I would make the argument that once an application scales beyond a single datacenter it cannot help but give up strict consistency under error conditions. > I don't think he completely misunderstood distributed systems - I think he decided to completely side-step the entire field. If he didn't misunderstand them then he is purposefully ignoring the hard problems. Which is worse?
    - ora600 16y ago
      > Once one scales beyond a single node the system becomes distributed. Then by definition one must deal with distributed systems problems in order to achieve scale beyond the capabilities of a single node I meant that one doesn't need to solve the general problem of distributed systems. Sharding is a common way to scale avoiding most of the problems generally associated with distributed systems. Scaling within LAN is easier than across data centers. You can assume no malicious traffic between your servers and suddenly solving the byzantine generals problem is far easier. Purposefully ignoring really hard problems can be a very good engineering practice.
      - ieure 16y ago
        Cool story, bro.
      - benblack 16y ago
        I purposefully ignored the tornado and so it did not hit my datacenter, tear off a section of the roof, kill all power sources, and drench my servers. Hard problems: solved.
        sophacles 16y ago
        Thanks! My app was in that datacenter too. I mean, it had replicated mongodb instances, and well balanced app servers, and nodes going away have no discernable affect on users. Turns out tho, that with all that distributed engineering, I didn't find out that the hosting company doesn't put your nodes in various data-centers. That tornado would have taken down my service during peak hours. I know you will try to write that off as a "you get what you diserve" but I challenge you to go ask people if their apps would survive a tornado to the data-center. Many of them will say "sure its in the cloud!" Then drop the killer question on them... "How many different data centers are your nodes running on right now". Most will say "i dont know". Some will say "My host has many data centers" (note this doesn't answer the question). A few will actually have done the footwork. Also, the scenario you describe is as easily mitigated with hot failovers and offsite backups. This probably qualifies as distributed engineering, but only is only the same as the above discussions in the most pedantic senses.
dasht 16y ago
A quick summary follows the quick editorial and after that a quick new thought: Editorial: Stonebraker is, imo, and as usual, Right Again. His biggest problem is that he's boring that way. He doesn't open his mouth in contexts like this but to be Right. Summary: People say "No SQL is right because of the CAP theorem." The CAP "theorem" says of DBs that: Consistency, high Availability, or Partition-tolerance .... pick any two. Quite true! So one of the pro no-SQL arguments is that high availability and partition tolerance are often the priorities ... so toss out consistency! SQL assumes consistency. Thus you need No SQL. Stonebraker correctly points out that, hey, you know what? Partitions are pretty rare and tossing out consistency really didn't increase your accessibility average by much .... so you tossed out consistency for no reason whatsoever. If you think the Cap "theorem" justifies NoSQL... you're just wrong. Stonebraker's rant is nearly boring because it makes such an obvious point. New Thought: I don't think NoSQL is popular because of the CAP theorem. I think it is popular because it is easier to get started with (even if that means using it poorly) than SQL. SQL is a little hard to learn. It's a little bit awkward to use in some "scripting" language or other HLL language. NoSQL may be bad engineering in many of its uses ... but its easier. A lot easier. And, people aren't much asking about engineering quality until sites start failing often. Which a heck of a lot of them do but by then the NoSQL architects have collected their money and are out of town or else are still around but able to point fingers of blame away from the abandonment of ACID. An ACID DB that gave a more simple-minded logical model than SQL ... including, sure, relaxing ACID constraints where that was really desirable ... could go a long way fixing the confusion around NoSQL. p.s.: given a typical distributed NoSQL DB, one thing you could do is regard that as the "physical model", implement proper transactions, and build a library that gave you ACID properties. Build up a high level way of using that library so that you have a logical model of the data that is independent of exactly how it is laid out in the underlying thing.... and you've got a Codd-style DB. Great thing to do. SQL 2.0
- jhugg 16y ago
  Not all of the distributed NoSQL systems give up consistency. Notably, H-Base and, in certain scenarios, MongoDB, both offer consistent atomic reads and writes of one thing, be it a row, supercolumn, document or whatever. You make a good point that NoSQL is about so much more than the CAP theorem. That doesn't mean there aren't a ton of people (some very smart) out there citing the CAP theorem as proof that you have to give up consistency to be "web-scale".
jnewland 16y ago
tl;dr version: http://files.jnewland.com/stonebraker-20101021-200946.jpg http://files.jnewland.com/stonebraker-20101021-200946.jpg
- ieure 16y ago
  Also, http://bit.ly/aD0TiH http://bit.ly/aD0TiH
jchrisa 16y ago
This is the comment I left on the post (still moderating): There is an extreme case of partition tolerance that must be considered: disconnected operation. For users at the edge of the network, latency can be the biggest performance killer. If it takes 1 second or more for each user action to be reflected in application state due to round trip time (mobile web) those seconds add up and users can be frustrated. However, if you move the database and web application to the mobile device itself, users no longer see network latency as part of the user experience critical path. Latency has been proven to be correlated directly to revenue, because users engage much more readily with snappy interfaces. Once data is being operated on by the user on the local device, the key becomes synchronization. Asynchronous multi-master replication demands a different approach to consistency, than the traditional model which assumes the database is being run by a central service. The MVCC document model is designed for synchronization. It's a different set of contraints than the relational model, but since it's such a highly constrained problem space it also admits of general solutions and protocols. It's my belief that the MVCC document model is closer to the 80% solution for a large class of applications. Storing strongly typed and normalized representations of data is an artifact of our historically constrained computing resources, so it will always be a good way to optimize certain problems. But for many human-scale data needs, schemaless documents are a very good fit. They optimize for the user, not the computer.
- ora600 16y ago
  MVCC you mean multi-version concurrency? As in writers-don't-block-readers? Because most relational databases have that.
  - jchrisa 16y ago
    the idea that conflicts are detected at write time, so that applications will have conflict resolution capabilities. eg if you get the Etag wrong CouchDB rejects the save. (edited to add) the difference is that CouchDB makes the MVCC semantics visible to the client.
logicalstack 16y ago
He seems to assume a lot in this post, for instance his 200 vs 4 node comparison assumes that you have 200 nodes because the poor performance of your DBMS requires that many nodes. If that's the case, great, use voltdb. If not, it's perfectly reasonable to think that 200 nodes would have more network partitions than 4 nodes which is why one would pick an AP system in the first place.

25 points by ora600 16y ago

moonpolysoft 16y ago

Wherein Stonebraker misunderstands distributed systems engineering completely.

ora600 16y ago
I understood what he said as: Solving the problems of distributed systems is incredibly hard and not necessary if all you need is scalability and high availability. Better think of your problems in terms of engineering trade-offs and not distributed theory and understand what you are giving up and what this gains you. When you decide to give up consistency, your application can no longer assume consistency ever. Giving up availability in case of network partition means few extra minutes of downtime a year. I don't think he completely misunderstood distributed systems - I think he decided to completely side-step the entire field.
- jhugg 16y ago
  Nice summary.
- moonpolysoft 16y ago
  > Solving the problems of distributed systems is incredibly hard and not necessary if all you need is scalability and high availability. Once one scales beyond a single node the system becomes distributed. Then by definition one must deal with distributed systems problems in order to achieve scale beyond the capabilities of a single node. > When you decide to give up consistency, your application can no longer assume consistency ever. Giving up availability in case of network partition means few extra minutes of downtime a year. Depends on the application. Giving up availability might mean cascading failures throughout your entire application. For instance if the datastore is unavailable for writes then any kind of queueing systems built around the DB (a common design pattern) run the risk of overflow during the downtime. And I would make the argument that once an application scales beyond a single datacenter it cannot help but give up strict consistency under error conditions. > I don't think he completely misunderstood distributed systems - I think he decided to completely side-step the entire field. If he didn't misunderstand them then he is purposefully ignoring the hard problems. Which is worse?
  - ora600 16y ago
    > Once one scales beyond a single node the system becomes distributed. Then by definition one must deal with distributed systems problems in order to achieve scale beyond the capabilities of a single node I meant that one doesn't need to solve the general problem of distributed systems. Sharding is a common way to scale avoiding most of the problems generally associated with distributed systems. Scaling within LAN is easier than across data centers. You can assume no malicious traffic between your servers and suddenly solving the byzantine generals problem is far easier. Purposefully ignoring really hard problems can be a very good engineering practice.
    - ieure 16y ago
      Cool story, bro.
    - benblack 16y ago
      I purposefully ignored the tornado and so it did not hit my datacenter, tear off a section of the roof, kill all power sources, and drench my servers. Hard problems: solved.
      - sophacles 16y ago
        Thanks! My app was in that datacenter too. I mean, it had replicated mongodb instances, and well balanced app servers, and nodes going away have no discernable affect on users. Turns out tho, that with all that distributed engineering, I didn't find out that the hosting company doesn't put your nodes in various data-centers. That tornado would have taken down my service during peak hours. I know you will try to write that off as a "you get what you diserve" but I challenge you to go ask people if their apps would survive a tornado to the data-center. Many of them will say "sure its in the cloud!" Then drop the killer question on them... "How many different data centers are your nodes running on right now". Most will say "i dont know". Some will say "My host has many data centers" (note this doesn't answer the question). A few will actually have done the footwork. Also, the scenario you describe is as easily mitigated with hot failovers and offsite backups. This probably qualifies as distributed engineering, but only is only the same as the above discussions in the most pedantic senses.

dasht 16y ago

A quick summary follows the quick editorial and after that a quick new thought: Editorial: Stonebraker is, imo, and as usual, Right Again. His biggest problem is that he's boring that way. He doesn't open his mouth in contexts like this but to be Right. Summary: People say "No SQL is right because of the CAP theorem." The CAP "theorem" says of DBs that: Consistency, high Availability, or Partition-tolerance .... pick any two. Quite true! So one of the pro no-SQL arguments is that high availability and partition tolerance are often the priorities ... so toss out consistency! SQL assumes consistency. Thus you need No SQL. Stonebraker correctly points out that, hey, you know what? Partitions are pretty rare and tossing out consistency really didn't increase your accessibility average by much .... so you tossed out consistency for no reason whatsoever. If you think the Cap "theorem" justifies NoSQL... you're just wrong. Stonebraker's rant is nearly boring because it makes such an obvious point. New Thought: I don't think NoSQL is popular because of the CAP theorem. I think it is popular because it is easier to get started with (even if that means using it poorly) than SQL. SQL is a little hard to learn. It's a little bit awkward to use in some "scripting" language or other HLL language. NoSQL may be bad engineering in many of its uses ... but its easier. A lot easier. And, people aren't much asking about engineering quality until sites start failing often. Which a heck of a lot of them do but by then the NoSQL architects have collected their money and are out of town or else are still around but able to point fingers of blame away from the abandonment of ACID. An ACID DB that gave a more simple-minded logical model than SQL ... including, sure, relaxing ACID constraints where that was really desirable ... could go a long way fixing the confusion around NoSQL. p.s.: given a typical distributed NoSQL DB, one thing you could do is regard that as the "physical model", implement proper transactions, and build a library that gave you ACID properties. Build up a high level way of using that library so that you have a logical model of the data that is independent of exactly how it is laid out in the underlying thing.... and you've got a Codd-style DB. Great thing to do. SQL 2.0

jhugg 16y ago
Not all of the distributed NoSQL systems give up consistency. Notably, H-Base and, in certain scenarios, MongoDB, both offer consistent atomic reads and writes of one thing, be it a row, supercolumn, document or whatever. You make a good point that NoSQL is about so much more than the CAP theorem. That doesn't mean there aren't a ton of people (some very smart) out there citing the CAP theorem as proof that you have to give up consistency to be "web-scale".

jnewland 16y ago

tl;dr version: http://files.jnewland.com/stonebraker-20101021-200946.jpg http://files.jnewland.com/stonebraker-20101021-200946.jpg

ieure 16y ago
Also, http://bit.ly/aD0TiH http://bit.ly/aD0TiH

jchrisa 16y ago

This is the comment I left on the post (still moderating): There is an extreme case of partition tolerance that must be considered: disconnected operation. For users at the edge of the network, latency can be the biggest performance killer. If it takes 1 second or more for each user action to be reflected in application state due to round trip time (mobile web) those seconds add up and users can be frustrated. However, if you move the database and web application to the mobile device itself, users no longer see network latency as part of the user experience critical path. Latency has been proven to be correlated directly to revenue, because users engage much more readily with snappy interfaces. Once data is being operated on by the user on the local device, the key becomes synchronization. Asynchronous multi-master replication demands a different approach to consistency, than the traditional model which assumes the database is being run by a central service. The MVCC document model is designed for synchronization. It's a different set of contraints than the relational model, but since it's such a highly constrained problem space it also admits of general solutions and protocols. It's my belief that the MVCC document model is closer to the 80% solution for a large class of applications. Storing strongly typed and normalized representations of data is an artifact of our historically constrained computing resources, so it will always be a good way to optimize certain problems. But for many human-scale data needs, schemaless documents are a very good fit. They optimize for the user, not the computer.

ora600 16y ago
MVCC you mean multi-version concurrency? As in writers-don't-block-readers? Because most relational databases have that.
- jchrisa 16y ago
  the idea that conflicts are detected at write time, so that applications will have conflict resolution capabilities. eg if you get the Etag wrong CouchDB rejects the save. (edited to add) the difference is that CouchDB makes the MVCC semantics visible to the client.

logicalstack 16y ago

He seems to assume a lot in this post, for instance his 200 vs 4 node comparison assumes that you have 200 nodes because the poor performance of your DBMS requires that many nodes. If that's the case, great, use voltdb. If not, it's perfectly reasonable to think that 200 nodes would have more network partitions than 4 nodes which is why one would pick an AP system in the first place.