3 ms·

This incident report tells very little. I hope they release what actually happened and how it affected their services. And how they are going to avoid it in the

This incident report tells very little. I hope they release what actually happened and how it affected their services. And how they are going to avoid it in the future. Almost any issue can be publicly described as: stuff broke because of network.

deleted 8y ago
[deleted]
WorldMaker 8y ago
"Partition failure" is a relatively specific term of art that implies it was a partial database failure, so "stuff broke because we lost part of a database". It may not be in GitHub's best interest to describe much more than that because which database, which tables, and how they were partitioned can be secret performance sauce. Postgres partitioning: https://www.postgresql.org/docs/9.1/static/ddl-partitioning.html https://www.postgresql.org/docs/9.1/static/ddl-partitioning.... Redis partitioning: https://redis.io/topics/partitioning https://redis.io/topics/partitioning
- zzzcpan 8y ago
  "Network partition" is unrelated to partial database failure or database partitioning. It means that database servers got disconnected from each other. Normally this shouldn't be a problem, but as far as I know they don't use proper distributed algorithms and so it's possible that disconnected servers each became masters and were serving requests independently, which is why they speak about inconsistencies. I believe this problem is commonly known as split brain [1]. [1] https://en.wikipedia.org/wiki/Split-brain_(computing) https://en.wikipedia.org/wiki/Split-brain_(computing)
  - WorldMaker 8y ago
    It's not "unrelated", it's an intertwined phenomenon, especially in practical consequences. In Redis, as one primary example I already mentioned, all database partitioning is network partitioning (and vice versa). The logical object is still a "database" even if the physical object suffering problems is the "network". For similar reasons that you suppose they "don't use proper distributed algorithms" (which seems an overly harsh way to put it), I simply presumed the logical entity in question is still a "database", even in the case of a network partition problem. I don't think it was either Postgres or Redis in this case, but they are simple enough examples to illustrate the overall problem. Either way, my case still seems to stand that we aren't likely to get more information about the specifics of the partition failure because likely both the logical (which "database") and physical layer specifics (which datacenters/"network") are things that are internal to GitHub that they may not be able to publicly describe much more than what they have.
    - LaserToy 8y ago
      It doesn’t really matter what they used, I just pointed out that their incident report is BS. I can blame virtually any production problem on network, but the ugly truth is - it is not networks fault in most of cases, but fault of engineers who developed the system with unreasonable assumptions.

deleted 8y ago

[deleted]

WorldMaker 8y ago

"Partition failure" is a relatively specific term of art that implies it was a partial database failure, so "stuff broke because we lost part of a database". It may not be in GitHub's best interest to describe much more than that because which database, which tables, and how they were partitioned can be secret performance sauce. Postgres partitioning: https://www.postgresql.org/docs/9.1/static/ddl-partitioning.html https://www.postgresql.org/docs/9.1/static/ddl-partitioning.... Redis partitioning: https://redis.io/topics/partitioning https://redis.io/topics/partitioning

zzzcpan 8y ago
"Network partition" is unrelated to partial database failure or database partitioning. It means that database servers got disconnected from each other. Normally this shouldn't be a problem, but as far as I know they don't use proper distributed algorithms and so it's possible that disconnected servers each became masters and were serving requests independently, which is why they speak about inconsistencies. I believe this problem is commonly known as split brain [1]. [1] https://en.wikipedia.org/wiki/Split-brain_(computing) https://en.wikipedia.org/wiki/Split-brain_(computing)
- WorldMaker 8y ago
  It's not "unrelated", it's an intertwined phenomenon, especially in practical consequences. In Redis, as one primary example I already mentioned, all database partitioning is network partitioning (and vice versa). The logical object is still a "database" even if the physical object suffering problems is the "network". For similar reasons that you suppose they "don't use proper distributed algorithms" (which seems an overly harsh way to put it), I simply presumed the logical entity in question is still a "database", even in the case of a network partition problem. I don't think it was either Postgres or Redis in this case, but they are simple enough examples to illustrate the overall problem. Either way, my case still seems to stand that we aren't likely to get more information about the specifics of the partition failure because likely both the logical (which "database") and physical layer specifics (which datacenters/"network") are things that are internal to GitHub that they may not be able to publicly describe much more than what they have.
  - LaserToy 8y ago
    It doesn’t really matter what they used, I just pointed out that their incident report is BS. I can blame virtually any production problem on network, but the ugly truth is - it is not networks fault in most of cases, but fault of engineers who developed the system with unreasonable assumptions.