5 ms·

Adding Context to CockroachDB’s Article “Your Database Should Work Like a CDN”

84 points by dantiberian 9y ago

lykr0n 9y ago
My problem with CockroachDB comes down to two major issues: 1. I need a read slave. With MariaDB Galera, I can slave a database of a local master, but Cockroach AFAIK can only do master master. Adding another node to a cluster that will just be read from seems like a lot of unneeded overhead, and can cause massive issues if there are sync issues. 2. WAN sucks. Where I work, I've seen mysql slaves get hours out of sync due to packet loss. With a centralized system, I don't have to worry about reconciling data. If every site can accept data, then I have to assume that every site will remain in sync. If Site 1 gets out of sync and/or stops transmitting data, another site could make further changes to the record in question causing issues. Some places this is perfectly fine, but dealing with transactions of counting clicks, I do not want to be dropping data or running into conflicts. > Sharded Relational databases come in many shapes and suffer from as many different types of ailments when deployed across regions: some sacrifice replication and availability for consistency, some do the opposite. I think this is a great point. You have to know what your database is doing, and how it does that. It seems to becoming more common for people to develop on something without know how it works; just to get bitten down the road. Any problem is going to get exasperated when you introduce it to the wonderful world of WAN. I'm not ragging on the product; it's a great good enough product for most use cases. I just more wanted to share my thoughts. If you can tolerate partial consistency, go for it. It a great run and done product. > Only companies with very sophisticated operations teams can seriously set an SLA for five nines. It’s doable, but comes with a heavy lift if your service is doing anything non-trivial. It’s certainly not the default position among the industry as far as I can tell. Five 9s is do-able for a small group of you work from the ground up (infrastructure to application). Dual site Postgres Citus with a slave coming off the Citus cluster, sitting in-front of a pgpool-II load balancer. Then replicate a slave out to each site you are running in the world. If you are running an API out there, use rabbitmq or kafka to relay transactions back to one of the sites or accept the latency and run raw commands over the wire. Five 9s when you are trying to drop in a solution is extremely hard, so I give a major kudos of CockroachDB for making that reachable with a drop in solution.
- uluyol 9y ago
  I'm just trying to understand this better. > 1. I need a read slave. With MariaDB Galera, I can slave a database of a local master, but Cockroach AFAIK can only do master master. Adding another node to a cluster that will just be read from seems like a lot of unneeded overhead, and can cause massive issues if there are sync issues. I don't know whether CockroachDB can do this or not, but there's no reason raft followers can't serve stale reads. Given that, I don't understand this concern. > 2. WAN sucks. Where I work, I've seen mysql slaves get hours out of sync due to packet loss. With a centralized system, I don't have to worry about reconciling data. If every site can accept data, then I have to assume that every site will remain in sync. If Site 1 gets out of sync and/or stops transmitting data, another site could make further changes to the record in question causing issues. Some places this is perfectly fine, but dealing with transactions of counting clicks, I do not want to be dropping data or running into conflicts. The whole point of CockroachDB is that you never see inconsistencies (at least not without opting in to it). But if you are unable to talk to any of the other replicas then you would have to be dropping data. You don't really have a choice here, if you have a network partition, pick unavailability or consistency. This isn't something Galera can fix either.
  - lykr0n 9y ago
    1. In my research, raft followers would still be thinking for themselves. Each node could potentially become master, which would be fine in most cases, but I just want a dum stupid slave database to pull a read only copy. My concern is latency and load. I want to be able to predict hotspots. I don't want api-node-3 to become the leader for that site, I want db-1 and db-2 to be where everything happens.
    - jasonwatkinspdx 9y ago
      > In my research, raft followers would still be thinking for themselves. There is a lot more going on than vanilla raft. Cockroach's range lease feature (which is NOT the same as the raft leader election) automatically moves the lease to the node that minimizes latency. You can control this behavior via meta data and matching rules. I'd suggest skimming: https://www.cockroachlabs.com/docs/stable/demo-follow-the-workload.html https://www.cockroachlabs.com/docs/stable/demo-follow-the-wo... Again, you're making a lot of assumptions that seem rooted in "cockroach == a raft cluster". There's a lot more going on than that.
luxpsycho 9y ago
> on it’s merits.
jacksmith21006 9y ago
Here is a great paper on Spanner which was the inspiration for CockroachDB https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46103.pdf https://static.googleusercontent.com/media/research.google.c...

84 points by dantiberian 9y ago

My problem with CockroachDB comes down to two major issues: 1. I need a read slave. With MariaDB Galera, I can slave a database of a local master, but Cockroach AFAIK can only do master master. Adding another node to a cluster that will just be read from seems like a lot of unneeded overhead, and can cause massive issues if there are sync issues. 2. WAN sucks. Where I work, I've seen mysql slaves get hours out of sync due to packet loss. With a centralized system, I don't have to worry about reconciling data. If every site can accept data, then I have to assume that every site will remain in sync. If Site 1 gets out of sync and/or stops transmitting data, another site could make further changes to the record in question causing issues. Some places this is perfectly fine, but dealing with transactions of counting clicks, I do not want to be dropping data or running into conflicts. > Sharded Relational databases come in many shapes and suffer from as many different types of ailments when deployed across regions: some sacrifice replication and availability for consistency, some do the opposite. I think this is a great point. You have to know what your database is doing, and how it does that. It seems to becoming more common for people to develop on something without know how it works; just to get bitten down the road. Any problem is going to get exasperated when you introduce it to the wonderful world of WAN. I'm not ragging on the product; it's a great good enough product for most use cases. I just more wanted to share my thoughts. If you can tolerate partial consistency, go for it. It a great run and done product. > Only companies with very sophisticated operations teams can seriously set an SLA for five nines. It’s doable, but comes with a heavy lift if your service is doing anything non-trivial. It’s certainly not the default position among the industry as far as I can tell. Five 9s is do-able for a small group of you work from the ground up (infrastructure to application). Dual site Postgres Citus with a slave coming off the Citus cluster, sitting in-front of a pgpool-II load balancer. Then replicate a slave out to each site you are running in the world. If you are running an API out there, use rabbitmq or kafka to relay transactions back to one of the sites or accept the latency and run raw commands over the wire. Five 9s when you are trying to drop in a solution is extremely hard, so I give a major kudos of CockroachDB for making that reachable with a drop in solution.

uluyol 9y ago
I'm just trying to understand this better. > 1. I need a read slave. With MariaDB Galera, I can slave a database of a local master, but Cockroach AFAIK can only do master master. Adding another node to a cluster that will just be read from seems like a lot of unneeded overhead, and can cause massive issues if there are sync issues. I don't know whether CockroachDB can do this or not, but there's no reason raft followers can't serve stale reads. Given that, I don't understand this concern. > 2. WAN sucks. Where I work, I've seen mysql slaves get hours out of sync due to packet loss. With a centralized system, I don't have to worry about reconciling data. If every site can accept data, then I have to assume that every site will remain in sync. If Site 1 gets out of sync and/or stops transmitting data, another site could make further changes to the record in question causing issues. Some places this is perfectly fine, but dealing with transactions of counting clicks, I do not want to be dropping data or running into conflicts. The whole point of CockroachDB is that you never see inconsistencies (at least not without opting in to it). But if you are unable to talk to any of the other replicas then you would have to be dropping data. You don't really have a choice here, if you have a network partition, pick unavailability or consistency. This isn't something Galera can fix either.
- lykr0n 9y ago
  1. In my research, raft followers would still be thinking for themselves. Each node could potentially become master, which would be fine in most cases, but I just want a dum stupid slave database to pull a read only copy. My concern is latency and load. I want to be able to predict hotspots. I don't want api-node-3 to become the leader for that site, I want db-1 and db-2 to be where everything happens.
  - jasonwatkinspdx 9y ago
    > In my research, raft followers would still be thinking for themselves. There is a lot more going on than vanilla raft. Cockroach's range lease feature (which is NOT the same as the raft leader election) automatically moves the lease to the node that minimizes latency. You can control this behavior via meta data and matching rules. I'd suggest skimming: https://www.cockroachlabs.com/docs/stable/demo-follow-the-workload.html https://www.cockroachlabs.com/docs/stable/demo-follow-the-wo... Again, you're making a lot of assumptions that seem rooted in "cockroach == a raft cluster". There's a lot more going on than that.

luxpsycho 9y ago

> on it’s merits.

jacksmith21006 9y ago

Here is a great paper on Spanner which was the inspiration for CockroachDB https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46103.pdf https://static.googleusercontent.com/media/research.google.c...