4 ms·

I believe you're right. They go into their design in their MySQL HA post: https://githubengineering.com/mysql-high-availability-at-github/#semi-synchronous-rep

by zawerf 8y ago

I believe you're right. They go into their design in their MySQL HA post: https://githubengineering.com/mysql-high-availability-at-github/#semi-synchronous-replication https://githubengineering.com/mysql-high-availability-at-git... > In MySQL’s semi-synchronous replication a master does not acknowledge a transaction commit until the change is known to have shipped to one or more replicas. It provides a way to achieve lossless failovers: any change applied on the master is either applied or waiting to be applied on one of the replicas. They only require one other replica (as opposed to a quorum) to be reachable from the master for the master to continue acknowledging writes. If a new master has been elected on the other side of the partition, both will continue acknowledging writes. They noted this as a limitation that they were working on (but unfortunately a bit too late in hindsight): > Notably, on a data center isolation scenario, and assuming a master is in the isolated DC, apps in that DC are still able to write to the master. This may result in state inconsistency once network is brought back up. We are working to mitigate this split-brain by implementing a reliable STONITH from within the very isolated DC. As before, some time will pass before bringing down the master, and there could be a short period of split-brain. The operational cost of avoiding split-brains altogether is very high.

donavanm 8y ago
I only skimmed that post, and havent dealt with mysql in years, but .... that bit has another wiggle with writes committed to one of many replicas. Im presuming thats any one of many. Which means I can now have writes committed not just in two places (terrible), but to N number of places (zomg). Recovering that (linearly) would tend towards impossible without something like a vector clock. But if you had a reliable clock of changes you probably wouldnt be in that mess at all.
- teraflop 8y ago
  You don't need vector clocks for master/slave replication, even if there's 1 master and N slaves. For consistency, all of the slaves should be applying the stream of updates to their local replicas in the same order that they happened on the master. That means each slave has seen a prefix of the master's data; its state can be uniquely represented by a position in the replication log. So you can easily determine which replica is most up-to-date by taking the one with the largest position. (If you're doing synchronous replication correctly, at least one replica is guaranteed to be completely up-to-date, even after an unexpected failure.) Vector clocks only come into play when you are simultaneously accepting writes on multiple masters, and you need to keep track of which updates have already been applied from each source.
  - donavanm 8y ago
    > If you're doing synchronous replication correctly, at least one replica is guaranteed to be completely up-to-date, even after an unexpected failure. Exactly? Maybe? Im reading a lot in to little information here. It wasnt clear to me if "one or more" was the master + a single specific replica, or something closer to master + 1/N. And then there's the automated optimistic leader election. Combine those with (multiple) partitions and you could have a very bad day. Rereading their RCA now I'm guessing the above isnt the case. "Restore the entire datacenter" reads more like they preserved the ~40 seconds of east coast partition writes, restored everything in east to a known checkpoint, and resumed replication off the more advanced masters in the west coast. If it's "just" two masters per cluster, both of which are available and internally consistent, you "just" have to reconcile those two write streams.
sealjam 8y ago
I'm by no means an expert on any of this stuff, but: > They only require one other replica (as opposed to a quorum) to be reachable from the master for the master to continue acknowledging writes (from your comment) > Orchestrator considers a number of variables during this process and is built on top of Raft for consensus... (from the article) Doesn't quite make sense to me. Doesn't raft require the master to wait for a quorum before committing writes? I understood it as a pretty important aspect. > A candidate must contact a majority of the cluster in order to be elected, which means that every committed entry must be present in at least one of those servers (from the raft paper [1]) I understood that the bold is only true if commits are acknowledged by the quorum Edit: I'm not implying your post is incorrect, just trying to understand how the two fit together [1]: https://raft.github.io/raft.pdf https://raft.github.io/raft.pdf
- sethammons 8y ago
  This was my first thought, you beat me to it. It seems like they contradict each other. Is it Raft with quorum or is it a single replica node that is caught up with master? You can't have both (unless you only have three nodes).
  - zawerf 8y ago
    > unless you only have three nodes Interestingly enough this seems to be how managed sql solutions do it. You just have one primary and one failover replica in another zone. So a quorum write is just a synchronous write to both (2 out of 2). You don't have the split-brain problem because the original primary can't make progress if it can't contact the replica. https://cloud.google.com/sql/docs/mysql/high-availability https://cloud.google.com/sql/docs/mysql/high-availability https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Conce... Github's problem is that they were trying to be too smart and allow any of their replicas to be candidate masters without increasing their quorum size. In theory this is higher availability than the managed sql solutions (they can be available even if their entire coast gets nuked) but they do it at the cost of consistency in the more common failure scenarios.
- 8y ago
tirumaraiselvan 8y ago
Github chose availability over consistency. Bad choice?
- pacuna 8y ago
  I would say yes. The service was available but it was even more chaotic because of all the inconsistencies.

by zawerf 8y ago

donavanm 8y ago

I only skimmed that post, and havent dealt with mysql in years, but .... that bit has another wiggle with writes committed to one of many replicas. Im presuming thats any one of many. Which means I can now have writes committed not just in two places (terrible), but to N number of places (zomg). Recovering that (linearly) would tend towards impossible without something like a vector clock. But if you had a reliable clock of changes you probably wouldnt be in that mess at all.

teraflop 8y ago
You don't need vector clocks for master/slave replication, even if there's 1 master and N slaves. For consistency, all of the slaves should be applying the stream of updates to their local replicas in the same order that they happened on the master. That means each slave has seen a prefix of the master's data; its state can be uniquely represented by a position in the replication log. So you can easily determine which replica is most up-to-date by taking the one with the largest position. (If you're doing synchronous replication correctly, at least one replica is guaranteed to be completely up-to-date, even after an unexpected failure.) Vector clocks only come into play when you are simultaneously accepting writes on multiple masters, and you need to keep track of which updates have already been applied from each source.
- donavanm 8y ago
  > If you're doing synchronous replication correctly, at least one replica is guaranteed to be completely up-to-date, even after an unexpected failure. Exactly? Maybe? Im reading a lot in to little information here. It wasnt clear to me if "one or more" was the master + a single specific replica, or something closer to master + 1/N. And then there's the automated optimistic leader election. Combine those with (multiple) partitions and you could have a very bad day. Rereading their RCA now I'm guessing the above isnt the case. "Restore the entire datacenter" reads more like they preserved the ~40 seconds of east coast partition writes, restored everything in east to a known checkpoint, and resumed replication off the more advanced masters in the west coast. If it's "just" two masters per cluster, both of which are available and internally consistent, you "just" have to reconcile those two write streams.

sealjam 8y ago

I'm by no means an expert on any of this stuff, but: > They only require one other replica (as opposed to a quorum) to be reachable from the master for the master to continue acknowledging writes (from your comment) > Orchestrator considers a number of variables during this process and is built on top of Raft for consensus... (from the article) Doesn't quite make sense to me. Doesn't raft require the master to wait for a quorum before committing writes? I understood it as a pretty important aspect. > A candidate must contact a majority of the cluster in order to be elected, which means that every committed entry must be present in at least one of those servers (from the raft paper [1]) I understood that the bold is only true if commits are acknowledged by the quorum Edit: I'm not implying your post is incorrect, just trying to understand how the two fit together [1]: https://raft.github.io/raft.pdf https://raft.github.io/raft.pdf

sethammons 8y ago
This was my first thought, you beat me to it. It seems like they contradict each other. Is it Raft with quorum or is it a single replica node that is caught up with master? You can't have both (unless you only have three nodes).
- zawerf 8y ago
  > unless you only have three nodes Interestingly enough this seems to be how managed sql solutions do it. You just have one primary and one failover replica in another zone. So a quorum write is just a synchronous write to both (2 out of 2). You don't have the split-brain problem because the original primary can't make progress if it can't contact the replica. https://cloud.google.com/sql/docs/mysql/high-availability https://cloud.google.com/sql/docs/mysql/high-availability https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Conce... Github's problem is that they were trying to be too smart and allow any of their replicas to be candidate masters without increasing their quorum size. In theory this is higher availability than the managed sql solutions (they can be available even if their entire coast gets nuked) but they do it at the cost of consistency in the more common failure scenarios.
8y ago

tirumaraiselvan 8y ago

Github chose availability over consistency. Bad choice?

pacuna 8y ago
I would say yes. The service was available but it was even more chaotic because of all the inconsistencies.