3 ms·

So you’re setting up a multi-region RDS. If region A goes down, do you continue to accept writes to region B? A bank: No! If region A goes down, do not process

So you’re setting up a multi-region RDS. If region A goes down, do you continue to accept writes to region B? A bank: No! If region A goes down, do not process updates in B until A is back up! We’d rather be down than wrong! A web forum: Yes! We can reconcile later when A comes back up. Until then keep serving traffic! CAP theorem doesn’t let you treat the cloud as a magic infinite availability box. You still have to design your system to pick the appropriate behavior when something breaks. No one without deep insight into your business needs can decide for you, either. You’re on the hook for choosing.

candiddevmike 2y ago
How do you setup a multi-region RDS with multi-region write?
- kstrauser 2y ago
  I think that's a thing? It's been a while since I've looked. Even if not, multi-AZ replication with failover is a standard setup and it has the same issues. Suppose you have some frontend servers and an RDS primary instance in us-west-2a and other frontend servers and an RDS replica in us-west-2b. The link between AZs goes down. For simplicity, pretend a failover doesn't happen. (It wouldn't matter if it did. All that would change here is the names of the AZs.) Do you accept writes in us-west-2a knowing the ones in 2b can't see them? Do you serve reads in 2b knowing they might be showing old information? Or do you shut down 2b altogether and limp along at half capacity in only 2a? What if the problem is that 2a becomes inaccessible to the Internet so that you can no longer use a load balancer to route requests to it? What if the writes in 2a hadn't fully replicated to 2b before the partition? You can probably answer those for any given business scenario, but the point is that you have to decide them and you can't outsource it to RDS. Some use cases prioritize availability above all else, using eventual consistency to work out the details once connectivity's restored. Others demand consistency above all else, and would rather be down than risk giving out wrong answers. No cloud host can decide what's write for you. The CAP theorem is extremely freaking relevant to anyone using more than one AZ or one region. (I know you weren't claiming otherwise, just taking a chance to say why cross-AZ still has the same issues as cross-region, as I have heard well meaning people say they were somehow different. AWS does a great job of keeping things running well to the point that it's news when they don't. Things still happen though. Same for Azure, GCP, and any other cloud offering. However flawless their design and execution, if a volcano erupts in the wrong place, there's gonna be a partition.)
phh 2y ago
Yes. I see CAP theorem as a (perfectly right) answer to managers demanding literally impossible requirements
slaymaker1907 2y ago
Actually, banks use eventually consistent systems in many cases. They'd rather let a few customers overdraw a bit than stop a customer from being able to buy something on their card when they have the funds for it.
- kstrauser 2y ago
  In many cases. There are few absolutes at the edges. They have to be decided on a case-by-case basis. Also, failure to decide is deciding.
- wongarsu 2y ago
  Good point. But I'd argue that's an artifact of history. Traditional banks started out when eventual consistency was the only choice. First with humans manually reconciling paper ledgers based on messages from couriers and telegraphs, later with the introduction of computers in batch processes that ran over night. The ability to do this live is new to them, and thus their processes are designed to not require it. Financial institutions set up in this century (like paypal or fintecs) usually do want consistency at all cost. Your risk exposure is a lot lower if you know people don't spend or withdraw more than they have.
  - darby_nine 2y ago
    > Good point. But I'd argue that's an artifact of history. I think there's also less liability for the banks if they accept and build around the risk of both eventual consistency and fraud. Even now wiring is necessary to pull off a digital bank theft, and even then you need to be extremely fast to get away with the money, and extremely fast to not go to prison, and liquidate it extremely quickly (likely into bitcoin). Even small delays in the transactions would make this basically impossible with modern banking, let alone a full clearing house day.
  - dalyons 2y ago
    fintechs have some inhouse tech but typically involve all kinds of third party systems. Even if you are interacting with these third parties via apis(instead of csv or whatever), you will always need an eventually consistent recon mechanism to make sure both sides agree on what was done. It doesnt have to take days though i agree. The only way around this would be to replace stateful apis with some kind of function that takes a reference to a verifiable ledger, that can mutate that and anyone can inspect themselves to see if the operation has been completed. Oh wait :)
cryptonector 2y ago
You are mischaracterizing what TFA says. It says that partitions either keep a client from reaching any servers (in which case we don't care) or they keep some servers from talking to others (or some clients from talking to all servers), and that in the latter case whenever you can have a quorum (really, a majority of the servers) reachable then you can continue. You've completely elided the quorum thing to make it sound like TFA is nuts.
- kstrauser 2y ago
  Quorum doesn't let you ignore partitions. It can't. You still have to decide what happens to the nodes on the wrong side of the partition, or how to handle multiple partitions (you have 5 nodes. You have 2 netsplits. Now you have clusters of 2 nodes, 2 nodes, and 1 node. Quorum say what?). I don't think TFA is nuts. I do think the premise that engineers using cloud can ignore CAP theorem is wrong, though. It's a decision you must consider when you're designing a system. As long as everything is running well, it doesn't matter. You've got to make sure that the behavior when something breaks is what you want it to be. And you can't outsource that decision. It can't be worked around.
  - cryptonector 2y ago
    > You still have to decide what happens to the nodes on the wrong side of the partition You don't because their clients won't also be isolated from the other servers. That's TFA's point, that in a cloud environment network partitions that affect clients generally deny them access to all servers (something you can't do anything about), and network partitions between servers only isolate some servers and none of the clients.
    - kstrauser 2y ago
      That’s fine if you value availability over consistency. Which is fine if that’s the appropriate choice for your use case! Even then you have to decide whether to accept updates in that situation, know that they may never be reconciled if the partition is permanent.

candiddevmike 2y ago

How do you setup a multi-region RDS with multi-region write?

kstrauser 2y ago
I think that's a thing? It's been a while since I've looked. Even if not, multi-AZ replication with failover is a standard setup and it has the same issues. Suppose you have some frontend servers and an RDS primary instance in us-west-2a and other frontend servers and an RDS replica in us-west-2b. The link between AZs goes down. For simplicity, pretend a failover doesn't happen. (It wouldn't matter if it did. All that would change here is the names of the AZs.) Do you accept writes in us-west-2a knowing the ones in 2b can't see them? Do you serve reads in 2b knowing they might be showing old information? Or do you shut down 2b altogether and limp along at half capacity in only 2a? What if the problem is that 2a becomes inaccessible to the Internet so that you can no longer use a load balancer to route requests to it? What if the writes in 2a hadn't fully replicated to 2b before the partition? You can probably answer those for any given business scenario, but the point is that you have to decide them and you can't outsource it to RDS. Some use cases prioritize availability above all else, using eventual consistency to work out the details once connectivity's restored. Others demand consistency above all else, and would rather be down than risk giving out wrong answers. No cloud host can decide what's write for you. The CAP theorem is extremely freaking relevant to anyone using more than one AZ or one region. (I know you weren't claiming otherwise, just taking a chance to say why cross-AZ still has the same issues as cross-region, as I have heard well meaning people say they were somehow different. AWS does a great job of keeping things running well to the point that it's news when they don't. Things still happen though. Same for Azure, GCP, and any other cloud offering. However flawless their design and execution, if a volcano erupts in the wrong place, there's gonna be a partition.)

phh 2y ago

Yes. I see CAP theorem as a (perfectly right) answer to managers demanding literally impossible requirements

slaymaker1907 2y ago

Actually, banks use eventually consistent systems in many cases. They'd rather let a few customers overdraw a bit than stop a customer from being able to buy something on their card when they have the funds for it.

kstrauser 2y ago
In many cases. There are few absolutes at the edges. They have to be decided on a case-by-case basis. Also, failure to decide is deciding.
wongarsu 2y ago
Good point. But I'd argue that's an artifact of history. Traditional banks started out when eventual consistency was the only choice. First with humans manually reconciling paper ledgers based on messages from couriers and telegraphs, later with the introduction of computers in batch processes that ran over night. The ability to do this live is new to them, and thus their processes are designed to not require it. Financial institutions set up in this century (like paypal or fintecs) usually do want consistency at all cost. Your risk exposure is a lot lower if you know people don't spend or withdraw more than they have.
- darby_nine 2y ago
  > Good point. But I'd argue that's an artifact of history. I think there's also less liability for the banks if they accept and build around the risk of both eventual consistency and fraud. Even now wiring is necessary to pull off a digital bank theft, and even then you need to be extremely fast to get away with the money, and extremely fast to not go to prison, and liquidate it extremely quickly (likely into bitcoin). Even small delays in the transactions would make this basically impossible with modern banking, let alone a full clearing house day.
- dalyons 2y ago
  fintechs have some inhouse tech but typically involve all kinds of third party systems. Even if you are interacting with these third parties via apis(instead of csv or whatever), you will always need an eventually consistent recon mechanism to make sure both sides agree on what was done. It doesnt have to take days though i agree. The only way around this would be to replace stateful apis with some kind of function that takes a reference to a verifiable ledger, that can mutate that and anyone can inspect themselves to see if the operation has been completed. Oh wait :)

cryptonector 2y ago

You are mischaracterizing what TFA says. It says that partitions either keep a client from reaching any servers (in which case we don't care) or they keep some servers from talking to others (or some clients from talking to all servers), and that in the latter case whenever you can have a quorum (really, a majority of the servers) reachable then you can continue. You've completely elided the quorum thing to make it sound like TFA is nuts.

kstrauser 2y ago
Quorum doesn't let you ignore partitions. It can't. You still have to decide what happens to the nodes on the wrong side of the partition, or how to handle multiple partitions (you have 5 nodes. You have 2 netsplits. Now you have clusters of 2 nodes, 2 nodes, and 1 node. Quorum say what?). I don't think TFA is nuts. I do think the premise that engineers using cloud can ignore CAP theorem is wrong, though. It's a decision you must consider when you're designing a system. As long as everything is running well, it doesn't matter. You've got to make sure that the behavior when something breaks is what you want it to be. And you can't outsource that decision. It can't be worked around.
- cryptonector 2y ago
  > You still have to decide what happens to the nodes on the wrong side of the partition You don't because their clients won't also be isolated from the other servers. That's TFA's point, that in a cloud environment network partitions that affect clients generally deny them access to all servers (something you can't do anything about), and network partitions between servers only isolate some servers and none of the clients.
  - kstrauser 2y ago
    That’s fine if you value availability over consistency. Which is fine if that’s the appropriate choice for your use case! Even then you have to decide whether to accept updates in that situation, know that they may never be reconciled if the partition is permanent.