4 ms·

Sooo, being able to distribute data globally is good for performance? Who knew? The thing about distributed systems, including distributed databases is that the

by astine 9y ago

Sooo, being able to distribute data globally is good for performance? Who knew? The thing about distributed systems, including distributed databases is that they need to navigate around the the CAP theorem(Consistency, Availability, Partition Tolerance, pick two, essentially,) and every solution is ultimately a trade off. This article would be a lot more interesting if it showed how CockroachDB made a better trade off than the other solutions listed.

AYBABTME 9y ago
The article they link at the end explains this: https://www.cockroachlabs.com/docs/stable/transactions.html https://www.cockroachlabs.com/docs/stable/transactions.html
- astine 9y ago
  Thanks, I got the answer to questions I was asking from CDB FAQ, actually. I just wish this article in particular was less fluffy.
marknadal 9y ago
Also a distributed database engineer, and while I disagree with CockroachDB's CAP Theorem tradeoff decisions, they are definitely well reasoned and principled. RethinkDB was Master-Slave (strongly consistent) with an amazing developer community and actually survived Aphyr's tests better than most systems. CockroachDB is also in the Master-Slave (strongly consistent) camp, but more enterprise focused, and therefore will probably not fail, however their Aphyr report worked... but was unfortunately very slow. But hey, being strongly consistent is hard and correctness is a necessary tradeoff from performance. Other systems, like Cassandra, us (https://github.com/amark/gun https://github.com/amark/gun), Couch/Pouch, etc. are all in the Master-Master camp, and thus AP not CP. Our argument is that while P holds, realtime sync with Strong Eventual Consistency is good enough yet has all the performance benefits, for everything except for banking. Sure, use Rethink/Cockroach for banking (heck, better yet, Postgres!), but for fun-and-games you can do banking on top of AP systems if you use a CRDT or blockchain (although that kills performance, CRDTs don't) on top. So yeah, I agree with you about CAP Theorem and stuff, disagree with Cockroach's particular choice - but they do have some pretty great detailed explainers/documentation on their view, and therefore should be treated seriously and not written off.
- elvinyung 9y ago
  Master-master doesn't automatically imply AP. e.g. EPaxos [1] is basically a master-master CP system. [1] https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf
- manigandham 9y ago
  Banking is eventually consistent and the most incorrectly used example when it comes to any database scenarios. CRDB is also master-master, all the nodes are the same and can serve reads and writes. That has no bearing on AP or CP.
Chickenosaurus 9y ago
I think the PACELC theorem [1] should be preferred to the CAP theorem as it is more precise. It states in case of a partition (P), there is a trade-off between availability (A) and consistency (C). Else (E), there is a trade-off between latency (L) and consistency (C). [1] https://en.m.wikipedia.org/wiki/PACELC_theorem https://en.m.wikipedia.org/wiki/PACELC_theorem
- tormeh 9y ago
  Ah, much better, I have to say. Much more informative as well. The CAP theorem is a bit ungainly since the only way to have a CA database is to only have a single node.
  - ithkuil 9y ago
    > the only way to have a CA database is to only have a single node. I know that this is meant to mean: in the real world you cannot just write off partition resilience and still call your system highly available, since partitions will happen sooner or later and when they do your CA system won't be available. But in the other hand, having a system that is always available _except_ during a network partition is a useful thing: you can design a network where partitions happen much rarely that the rate at which individual machines die. I.e. in practice a single node, while if you nitpick is the only true CA, will available for less time in average than a multi node CA system which if you nitpick is not CA (provided that the underlying network is partition resilient; it's not a boolean, it's a probability) (See Google spanner)
    - zjaffee 9y ago
      It's my understanding that Google Spanner is leveraging hardware (direct lines between machines), so the degree at which there is a network partition is different than compared to other systems.
      - Chickenosaurus 9y ago
        According to a paper by Eric Brewer, the author of the original CAP theorem, Google Spanner technically is not a CA system [1]. It is advertised as such because partitons are supposed to be exceedingly rare as the infrastructure is outstanding and a lot of operating experience is present [1]. I believe at the end of the day the CAP theorem is too fuzzy for such discussions. [1] https://research.google.com/pubs/pub45855.html https://research.google.com/pubs/pub45855.html
philip1209 9y ago
When I think of distributed databases, I think of Bitcoin - and that's not fast :-)
skybrian 9y ago
From a bit of reading, it looks like they do it with optimistic concurrency, probably at the expense of write throughput. When there is contention (writing the same keys) there may be a lot of retries. It seems like a good solution for data that doesn't change too quickly?

by astine 9y ago

AYBABTME 9y ago

The article they link at the end explains this: https://www.cockroachlabs.com/docs/stable/transactions.html https://www.cockroachlabs.com/docs/stable/transactions.html

astine 9y ago
Thanks, I got the answer to questions I was asking from CDB FAQ, actually. I just wish this article in particular was less fluffy.

marknadal 9y ago

Also a distributed database engineer, and while I disagree with CockroachDB's CAP Theorem tradeoff decisions, they are definitely well reasoned and principled. RethinkDB was Master-Slave (strongly consistent) with an amazing developer community and actually survived Aphyr's tests better than most systems. CockroachDB is also in the Master-Slave (strongly consistent) camp, but more enterprise focused, and therefore will probably not fail, however their Aphyr report worked... but was unfortunately very slow. But hey, being strongly consistent is hard and correctness is a necessary tradeoff from performance. Other systems, like Cassandra, us (https://github.com/amark/gun https://github.com/amark/gun), Couch/Pouch, etc. are all in the Master-Master camp, and thus AP not CP. Our argument is that while P holds, realtime sync with Strong Eventual Consistency is good enough yet has all the performance benefits, for everything except for banking. Sure, use Rethink/Cockroach for banking (heck, better yet, Postgres!), but for fun-and-games you can do banking on top of AP systems if you use a CRDT or blockchain (although that kills performance, CRDTs don't) on top. So yeah, I agree with you about CAP Theorem and stuff, disagree with Cockroach's particular choice - but they do have some pretty great detailed explainers/documentation on their view, and therefore should be treated seriously and not written off.

elvinyung 9y ago
Master-master doesn't automatically imply AP. e.g. EPaxos [1] is basically a master-master CP system. [1] https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf
manigandham 9y ago
Banking is eventually consistent and the most incorrectly used example when it comes to any database scenarios. CRDB is also master-master, all the nodes are the same and can serve reads and writes. That has no bearing on AP or CP.

Chickenosaurus 9y ago

I think the PACELC theorem [1] should be preferred to the CAP theorem as it is more precise. It states in case of a partition (P), there is a trade-off between availability (A) and consistency (C). Else (E), there is a trade-off between latency (L) and consistency (C). [1] https://en.m.wikipedia.org/wiki/PACELC_theorem https://en.m.wikipedia.org/wiki/PACELC_theorem

tormeh 9y ago
Ah, much better, I have to say. Much more informative as well. The CAP theorem is a bit ungainly since the only way to have a CA database is to only have a single node.
- ithkuil 9y ago
  > the only way to have a CA database is to only have a single node. I know that this is meant to mean: in the real world you cannot just write off partition resilience and still call your system highly available, since partitions will happen sooner or later and when they do your CA system won't be available. But in the other hand, having a system that is always available _except_ during a network partition is a useful thing: you can design a network where partitions happen much rarely that the rate at which individual machines die. I.e. in practice a single node, while if you nitpick is the only true CA, will available for less time in average than a multi node CA system which if you nitpick is not CA (provided that the underlying network is partition resilient; it's not a boolean, it's a probability) (See Google spanner)
  - zjaffee 9y ago
    It's my understanding that Google Spanner is leveraging hardware (direct lines between machines), so the degree at which there is a network partition is different than compared to other systems.
    - Chickenosaurus 9y ago
      According to a paper by Eric Brewer, the author of the original CAP theorem, Google Spanner technically is not a CA system [1]. It is advertised as such because partitons are supposed to be exceedingly rare as the infrastructure is outstanding and a lot of operating experience is present [1]. I believe at the end of the day the CAP theorem is too fuzzy for such discussions. [1] https://research.google.com/pubs/pub45855.html https://research.google.com/pubs/pub45855.html

philip1209 9y ago

When I think of distributed databases, I think of Bitcoin - and that's not fast :-)

skybrian 9y ago

From a bit of reading, it looks like they do it with optimistic concurrency, probably at the expense of write throughput. When there is contention (writing the same keys) there may be a lot of retries. It seems like a good solution for data that doesn't change too quickly?