7 ms·

The timeline of events was interesting (and much appreciated), but the root cause analysis doesn't really go much deeper than "we had a brief network partition,

The timeline of events was interesting (and much appreciated), but the root cause analysis doesn't really go much deeper than "we had a brief network partition, and our systems weren't designed to cope with it", which still leaves a whole lot of question marks. Of course, without detailed knowledge of how GitHub's internals work, all we can do is speculate. But just based on what was explained in this blog post, it sounds like they're replicating database updates asynchronously, without waiting for the updates to be acknowledged by slaves before the master allows them to commit. Which means the data on slaves is always slightly out-of-date, and becomes more out-of-date when the slaves are partitioned from the master. Which means that promoting a slave to master will by definition lose some committed writes. If "guarding the confidentiality and integrity of user data is GitHub’s highest priority", then why would they build and deploy an automated failover system whose purpose is to preserve availability at the cost of consistency? And why were they apparently caught off-guard when it operated as designed? (Reading point 1 under "technical initiatives", it seems that they consider intra-DC failover to be "safe", and cross-DC failover to be "unsafe". But the exact same failure mode is present in both cases; the only difference is the length of the time during which in-flight writes can be lost.)

zawerf 8y ago
I believe you're right. They go into their design in their MySQL HA post: https://githubengineering.com/mysql-high-availability-at-github/#semi-synchronous-replication https://githubengineering.com/mysql-high-availability-at-git... > In MySQL’s semi-synchronous replication a master does not acknowledge a transaction commit until the change is known to have shipped to one or more replicas. It provides a way to achieve lossless failovers: any change applied on the master is either applied or waiting to be applied on one of the replicas. They only require one other replica (as opposed to a quorum) to be reachable from the master for the master to continue acknowledging writes. If a new master has been elected on the other side of the partition, both will continue acknowledging writes. They noted this as a limitation that they were working on (but unfortunately a bit too late in hindsight): > Notably, on a data center isolation scenario, and assuming a master is in the isolated DC, apps in that DC are still able to write to the master. This may result in state inconsistency once network is brought back up. We are working to mitigate this split-brain by implementing a reliable STONITH from within the very isolated DC. As before, some time will pass before bringing down the master, and there could be a short period of split-brain. The operational cost of avoiding split-brains altogether is very high.
- donavanm 8y ago
  I only skimmed that post, and havent dealt with mysql in years, but .... that bit has another wiggle with writes committed to one of many replicas. Im presuming thats any one of many. Which means I can now have writes committed not just in two places (terrible), but to N number of places (zomg). Recovering that (linearly) would tend towards impossible without something like a vector clock. But if you had a reliable clock of changes you probably wouldnt be in that mess at all.
  - teraflop 8y ago
    You don't need vector clocks for master/slave replication, even if there's 1 master and N slaves. For consistency, all of the slaves should be applying the stream of updates to their local replicas in the same order that they happened on the master. That means each slave has seen a prefix of the master's data; its state can be uniquely represented by a position in the replication log. So you can easily determine which replica is most up-to-date by taking the one with the largest position. (If you're doing synchronous replication correctly, at least one replica is guaranteed to be completely up-to-date, even after an unexpected failure.) Vector clocks only come into play when you are simultaneously accepting writes on multiple masters, and you need to keep track of which updates have already been applied from each source.
    - donavanm 8y ago
      > If you're doing synchronous replication correctly, at least one replica is guaranteed to be completely up-to-date, even after an unexpected failure. Exactly? Maybe? Im reading a lot in to little information here. It wasnt clear to me if "one or more" was the master + a single specific replica, or something closer to master + 1/N. And then there's the automated optimistic leader election. Combine those with (multiple) partitions and you could have a very bad day. Rereading their RCA now I'm guessing the above isnt the case. "Restore the entire datacenter" reads more like they preserved the ~40 seconds of east coast partition writes, restored everything in east to a known checkpoint, and resumed replication off the more advanced masters in the west coast. If it's "just" two masters per cluster, both of which are available and internally consistent, you "just" have to reconcile those two write streams.
bleair 8y ago
It sounds as if a switch was replaced so network connectivity was lost for 40 seconds (a partition if you will). The network recovered but during the 40 seconds a lot of "things" had to queued and processed and things cascaded out of control. Whenever people talk about CAP they seem to spend time thinking about behavior when there is a partition, and that's neat an all, but the real challenging parts come when the partition is healed, and people don't seem to spend enough time thinking about that mode of execution.
- BillinghamJ 8y ago
  Or when there isn’t a clear/complete partition and the network is kinda working but not consistently.
tbrock 8y ago
> it sounds like they're replicating database updates asynchronously, without waiting for the updates to be acknowledged by slaves before the master allows them to commit. Which means the data on slaves is always slightly out-of-date, and becomes more out-of-date when the slaves are partitioned from the master. Which means that promoting a slave to master will by definition lose some committed writes. Unless you temporarily pause writes to the primary before promting a slave.
- teraflop 8y ago
  Sure, that's fine if you're talking about planned downtime. You still lose data if there's an unexpected failure, because by the time you realize the primary is gone, it's too late; the writes have already happened and been acknowledged.
laquisha 8y ago
OMG you cannot call them master and slave anymore!
nstart 8y ago
> But just based on what was explained in this blog post, it sounds like they're replicating database updates asynchronously, without waiting for the updates to be acknowledged by slaves before the master allows them to commit. Which means the data on slaves is always slightly out-of-date... Anecdotally I think I see this regularly. Part of the build pipeline I work on requires recursively downloading a particular folder for a given repo. From time to time, I'll have the latest git ref give me data from the previous git ref. Everywhere else on GitHub the information is up-to-date. But not the API endpoint to download files. When this happens it takes roughly 5-7 minutes for the correct information to show up. After reading this and all the other resources people have posted I'm definitely going to see if I can reliably reproduce the error.
- richardwhiuk 8y ago
  Why would you use an API endpoint to download files in a build pipeline? Surely you should just use a clone of the repository? I could believe that an endpoint to download files looks at a cache, and if you are just using 'master' or 'latest' as opposed to a commit, I could believe that would be out of date.
  - nstart 8y ago
    Bit late, but replying to this because my comment does have missing context. Most of the repos we have are large enough that downloading three individual files is faster than cloning the whole repo. Even with a shallow clone with a depth of 1. I've tried the combinations and even tried downloading a zip file of the repo. Still faster to download the three files. And I haven't even added parallel downloads because it's fine as is. But given the issues, why continue to use the API? Because those files are related to the build pipeline logic and they change maybe once in 3-4 months. I can live with an error once in 3-4 months. Recently I was working on changing some of the logic of the pipeline and that's when I ran into it enough to say that there was something odd from Github's end.
Rapzid 8y ago
Typical solution is to mix synchronous block level replication intra-DC with asychronous replication inter-DC. I believe RDS utilizes DRBD for this intra-region. This may be why failover is "safe" in certain circumstances.
ransom1538 8y ago
"Which means that promoting a slave to master will by definition lose some committed writes." Sure. But that is where the application code comes in. You usually have at least two db connections one to the slave connection and one to master. If you are checking if an account exists [important] you can ask the master since it will be in an up to date state. If you want to build someone's feed on github, then doing all your sql though the slave is fine. They also probably have layers of atomic cache on top which handles 99% of the requests [memcache,redis,whatever]. EG. Does the account exist? Check the cache, check master.
- specialist 8y ago
  Restating what I think you're suggesting: Use master for transactional read/writes. Use replicas (slaves) for readonly. More or less?
  - sethammons 8y ago
    I'm sure that is what they do. But application code does not save you from this scenario. If your master changes, the application does not know that the new master was a node that did not have full replication.

zawerf 8y ago

I believe you're right. They go into their design in their MySQL HA post: https://githubengineering.com/mysql-high-availability-at-github/#semi-synchronous-replication https://githubengineering.com/mysql-high-availability-at-git... > In MySQL’s semi-synchronous replication a master does not acknowledge a transaction commit until the change is known to have shipped to one or more replicas. It provides a way to achieve lossless failovers: any change applied on the master is either applied or waiting to be applied on one of the replicas. They only require one other replica (as opposed to a quorum) to be reachable from the master for the master to continue acknowledging writes. If a new master has been elected on the other side of the partition, both will continue acknowledging writes. They noted this as a limitation that they were working on (but unfortunately a bit too late in hindsight): > Notably, on a data center isolation scenario, and assuming a master is in the isolated DC, apps in that DC are still able to write to the master. This may result in state inconsistency once network is brought back up. We are working to mitigate this split-brain by implementing a reliable STONITH from within the very isolated DC. As before, some time will pass before bringing down the master, and there could be a short period of split-brain. The operational cost of avoiding split-brains altogether is very high.

donavanm 8y ago
I only skimmed that post, and havent dealt with mysql in years, but .... that bit has another wiggle with writes committed to one of many replicas. Im presuming thats any one of many. Which means I can now have writes committed not just in two places (terrible), but to N number of places (zomg). Recovering that (linearly) would tend towards impossible without something like a vector clock. But if you had a reliable clock of changes you probably wouldnt be in that mess at all.
- teraflop 8y ago
  You don't need vector clocks for master/slave replication, even if there's 1 master and N slaves. For consistency, all of the slaves should be applying the stream of updates to their local replicas in the same order that they happened on the master. That means each slave has seen a prefix of the master's data; its state can be uniquely represented by a position in the replication log. So you can easily determine which replica is most up-to-date by taking the one with the largest position. (If you're doing synchronous replication correctly, at least one replica is guaranteed to be completely up-to-date, even after an unexpected failure.) Vector clocks only come into play when you are simultaneously accepting writes on multiple masters, and you need to keep track of which updates have already been applied from each source.
  - donavanm 8y ago
    > If you're doing synchronous replication correctly, at least one replica is guaranteed to be completely up-to-date, even after an unexpected failure. Exactly? Maybe? Im reading a lot in to little information here. It wasnt clear to me if "one or more" was the master + a single specific replica, or something closer to master + 1/N. And then there's the automated optimistic leader election. Combine those with (multiple) partitions and you could have a very bad day. Rereading their RCA now I'm guessing the above isnt the case. "Restore the entire datacenter" reads more like they preserved the ~40 seconds of east coast partition writes, restored everything in east to a known checkpoint, and resumed replication off the more advanced masters in the west coast. If it's "just" two masters per cluster, both of which are available and internally consistent, you "just" have to reconcile those two write streams.

bleair 8y ago

It sounds as if a switch was replaced so network connectivity was lost for 40 seconds (a partition if you will). The network recovered but during the 40 seconds a lot of "things" had to queued and processed and things cascaded out of control. Whenever people talk about CAP they seem to spend time thinking about behavior when there is a partition, and that's neat an all, but the real challenging parts come when the partition is healed, and people don't seem to spend enough time thinking about that mode of execution.

BillinghamJ 8y ago
Or when there isn’t a clear/complete partition and the network is kinda working but not consistently.

tbrock 8y ago

> it sounds like they're replicating database updates asynchronously, without waiting for the updates to be acknowledged by slaves before the master allows them to commit. Which means the data on slaves is always slightly out-of-date, and becomes more out-of-date when the slaves are partitioned from the master. Which means that promoting a slave to master will by definition lose some committed writes. Unless you temporarily pause writes to the primary before promting a slave.

teraflop 8y ago
Sure, that's fine if you're talking about planned downtime. You still lose data if there's an unexpected failure, because by the time you realize the primary is gone, it's too late; the writes have already happened and been acknowledged.

laquisha 8y ago

OMG you cannot call them master and slave anymore!

nstart 8y ago

> But just based on what was explained in this blog post, it sounds like they're replicating database updates asynchronously, without waiting for the updates to be acknowledged by slaves before the master allows them to commit. Which means the data on slaves is always slightly out-of-date... Anecdotally I think I see this regularly. Part of the build pipeline I work on requires recursively downloading a particular folder for a given repo. From time to time, I'll have the latest git ref give me data from the previous git ref. Everywhere else on GitHub the information is up-to-date. But not the API endpoint to download files. When this happens it takes roughly 5-7 minutes for the correct information to show up. After reading this and all the other resources people have posted I'm definitely going to see if I can reliably reproduce the error.

richardwhiuk 8y ago
Why would you use an API endpoint to download files in a build pipeline? Surely you should just use a clone of the repository? I could believe that an endpoint to download files looks at a cache, and if you are just using 'master' or 'latest' as opposed to a commit, I could believe that would be out of date.
- nstart 8y ago
  Bit late, but replying to this because my comment does have missing context. Most of the repos we have are large enough that downloading three individual files is faster than cloning the whole repo. Even with a shallow clone with a depth of 1. I've tried the combinations and even tried downloading a zip file of the repo. Still faster to download the three files. And I haven't even added parallel downloads because it's fine as is. But given the issues, why continue to use the API? Because those files are related to the build pipeline logic and they change maybe once in 3-4 months. I can live with an error once in 3-4 months. Recently I was working on changing some of the logic of the pipeline and that's when I ran into it enough to say that there was something odd from Github's end.

Rapzid 8y ago

Typical solution is to mix synchronous block level replication intra-DC with asychronous replication inter-DC. I believe RDS utilizes DRBD for this intra-region. This may be why failover is "safe" in certain circumstances.

ransom1538 8y ago

"Which means that promoting a slave to master will by definition lose some committed writes." Sure. But that is where the application code comes in. You usually have at least two db connections one to the slave connection and one to master. If you are checking if an account exists [important] you can ask the master since it will be in an up to date state. If you want to build someone's feed on github, then doing all your sql though the slave is fine. They also probably have layers of atomic cache on top which handles 99% of the requests [memcache,redis,whatever]. EG. Does the account exist? Check the cache, check master.

specialist 8y ago
Restating what I think you're suggesting: Use master for transactional read/writes. Use replicas (slaves) for readonly. More or less?
- sethammons 8y ago
  I'm sure that is what they do. But application code does not save you from this scenario. If your master changes, the application does not know that the new master was a node that did not have full replication.