10 ms·
GitHub: October 21 Incident Report
- dvfjsdhgfv 8y agoIt would be interested to know what was the cause of the network partition failure.
- jbb67 8y agoUgh I detest the term " abundance of caution " as used in that message. Weasel words designed to stop you thinking about the problem as who wouldn't want them to be overly cautious
- deathhand 8y agoIt's not that bad. X definitely modifies Y but it may also modify Z. You are informing the audience of this fact and it provides them with 'warm fuzzies'.
- k_ 8y agoAlso discussed there: https://news.ycombinator.com/item?id=18271180 https://news.ycombinator.com/item?id=18271180
- vqng 8y ago> During this time, information displayed on GitHub.com is likely to appear out of date; if the new data is not presented, users will typically retry which may result in duplicated new content.
- rashthedude 8y agoHappened to me already. Created a new branch but it kept me informing me the non-existence, retried a few times and shortly after was created with a bunch of duplicates.
- geerlingguy 8y agoSame here, but last night I was trying to post a comment to another repo, refreshed and tried again like 5 times before realizing there was an outage and it wasn't on my end. This morning that issue has 5 dupe comments (so I look like an idiot), and deleting them does nothing; they just reappear when I refresh the issue.
- nik736 8y agoI don't understand why code hosting platforms like GitHub, GitLab or BitBucket have so many issues regularly. Is there anything special about it?
- skocznymroczny 8y agoLots of users. Also, a lot of package managers rely on hosting platforms like GitHub to host their packages, so if Github breaks, a lot of CI processes around the world break.
- TeMPOraL 8y agoWhich is kind of ridiculous. If your CI breaks because GitHub is down, it means it's not caching dependencies locally, but keeps re-downloading them every time it runs (e.g. every commit), generating tons of waste and unnecessary load on the hosting service. Or, to put it bluntly, if your CI works like this, it's contributing to climate change.
- realreality 8y agoThat’s an excellent point. How can you tell if the CI system uses caching, other than waiting for a github outage to notice something broke?
- goykasi 8y agoI think you are wrong. Our CI infra caches all dependencies, but it depends on github for new internal code pushes (kind of the point). If github is not sending events, CI doesnt kick. Youre ignoring half of the problem. If you dont receive events from github because they are down, your CI doesnt work either -- dependency caching doesnt matter at that point.
- JAdamMoore 8y agoIt won't keep me logged in long enough to make any changes or do anything. I'd rather not have access than have time wasting, useless access.
- josteink 8y ago> Further, this incident only impacted website metadata stored in our MySQL databases, such as issues and pull requests. Basically the only things I was going to do on Github today :)
- richardwhiuk 8y agoI hope they are going to provide a better RCA than that!
- rplnt 8y agoSure, but I don't think they have the time since the outage is still ongoing.
- shashwat986 8y agoYeah, we're still facing issues with, erm, Github issues. Also, while they haven't updated this blog post for a while, their status page has been very up-to-date and informative: https://status.github.com/messages https://status.github.com/messages
- AdamJacobMuller 8y ago> very up-to-date and informative Is that satire? It said 2-hour ETA 5 hours ago and the last update was over two hours ago.
- lol768 8y agoI see an update 7 minutes ago. >12:56 British Summer Time >The majority of restore processes have completed. We anticipate all data stores will be fully consistent within the next hour.
- LeoNatan25 8y agoEvery hour they promise something will be done until the next hour. I haven’t been able to work all day so far.
- Piskvorrr 8y agoConsider this a lesson on serverlessness. (We have been similarly afflicted, but their git backend seems to be up; and even further, we have rediscovered what we stopped paying attention to: that with Git, a centralized repo is just a convenience, not a requirement.)
- LeoNatan25 8y agoYes, I agree, but here it’s not up to me to choose the infrastructure. I wonder what the total cost of this ordeal must be. Surely in the tens of millions.
- 8y ago
- diegoperini 8y agoAs long as the same problem doesn't occur again, I have no hard feelings against them.
- edejong 8y agoLets do a quick back of the envelope calculation: Github reports 28,337,706 users by 2018-06-05 [1]. Lets assume 50% of these are active. Lets also assume that, due to the unavailability of GH, around 2 usable hours per developer are lost. Another assumption is that each developer contributes around 50 US$ per hour. This means, this outage has cost us users: (28337706 * .5 * 2 * 50) = 1.351 billion US$. Perhaps not use MySQL for such critical systems? [1] https://github.com/search?q=type:user&type=Users https://github.com/search?q=type:user&type=Users
- cddotdotslash 8y agoThat's a vast exaggeration. You're assuming every single one of GitHub's active users: 1) was active during the exact incident time and 2) is a business user earning US wages.
- edejong 8y agoHow would you do the calculation of the costs of this outage? Don't forget that the 50 US$ is lost added value, not wages.
- tjoff 8y agoAverage 50 percent active (as in daily usage)? 2 hours lost? I'd estimate a couple of order of magnitudes less...
- edejong 8y agoI don't think 2 hours is an exaggeration for an active user. The 50% might be too high, I agree.
- tjoff 8y ago2 hours lost? Maybe 2 hours affected, but then just don't push/fetch during that time. Very few would have to stop and wait. Average lost I'd guess to 10 minutes (most of it being "huh, wonder what's up with github")
- mncharity 8y agoAs I rearrange today's todo list, I recall wishing that I'd used the Microsoft purchase event to encourage folks to increase their familiarity with gitlab. So I now note: https://gitlab.com/explore/projects https://gitlab.com/explore/projects is a live feed of project activity, suitable for code surfing. It also sorts by stars and trending.
- foxhop 8y agoDon't try creating a new repo, it will create part of the meta data but not allow you to see or use the repo and the repo name gets taken.
- Symbiote 8y agoI created a new repository at about 9 UTC, and it started to work some of the time at 12 UTC — I've pushed the code from my computer, and pulled it from elsewhere. However, it's still intermittently failing as of 14 UTC, so I haven't managed a Maven release build yet.
- hoppelhase 8y agoI created a new repository 30 minutes ago. As I initialized it with a readme and an ignore file, I had to clone it. Cloning only succeeded by doing `watch git clone` and waiting a few minutes. But it worked.
- _verandaguy 8y agoI saw some comments on reddit which highlighted a pretty serious problem - many orgs rely on github as a fully integrated CD platform, with everything from code hosting, to running CI hooks, to pushing to staging or prod. It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over, but which has regularly been targeted in attacks by nation-state-level actors because of their role as a code hosting platform. EDIT: GH hasn't been targeted regularly, but it has been so historically, so this is a plausible thing which might happen again.
- Klathmon 8y agoIt's a tradeoff, just like anything else. We use Github heavily at my work, and at past jobs as well. But at the same time it's not the ONLY way we can work. If github has an outage, our CD will shut down, but we can still run all tests locally, and we can still push to the server directly, and we can still push code between ourselves manually and review it. Sure, it's a hiccup in our day when it goes down, but it's not like the entire company grinds to a halt. And the alternative of maintaining servers and systems to replicate all of that would take significantly more time, potentially cost more, and is probably more likely to go down.
- colonelpopcorn 8y agoOn premises git hosting with Gittea or GitLab with mirrors to GitHub seems like a smart idea going forward.
- elliotlarson 8y agoBut then you have to host it and maintain it. It's a slippery slope. How many 3rd party services do you in-house with hosted OS software. Pretty soon you're spending a huge chunk of your time doing ops work. And, where do you host it? On AWS, which can also go down, or on hardware hosted at your office. With on premise hosting, now you're in the hardware game too.
- smilesmile2018 8y agoGitHub team seems to be VERY unprofessional. 15 hour outage means ~99.82% availability which is extremely bad. 9 hours ago they also told that they would fix the problem within 2 hours... still not fixed!!!
- fishnchips 8y agoIn order to objectively assess their level of professionalism, you'd need to know what happened, and what's going on in there right now. Think about it this way - can someone break things where you work right now to cause an outage of this magnitude? For all places that I've worked the answer would be "yes, of course".
- platinium 8y agoIt doesn't matter, really. Its a black box from a business perspective. Some users have lost faith, and some people will migrate to other solutions. Regardless of how fair or unfair the incident was, it is a fact that it was poor up-time especially for such an important cloud provider for code.
- fishnchips 8y agoMaking decisions based on a single event is risky. One could even argue that an event of this magnitude is likely to cause significant improvements in reliability. And you just paid the price for that as a user, so unless they keep failing repeatedly, you may be better off sticking with them.
- zzzcpan 8y ago> One could even argue that an event of this magnitude is likely to cause significant improvements in reliability. Doesn't happen in practice and usually the whole thing is just blamed on "process" with subsequent "process changes".
- deleted 8y ago[deleted]
- platinium 8y agoOn the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo's: https://stackoverflow.com/questions/11690709/can-a-project-have-multiple-origins https://stackoverflow.com/questions/11690709/can-a-project-h... Quoted: "Try adding a remote called "github" instead: $ git remote add github https://github.com/Company_Name/repository_name.git # push master to github $ git push github master # Push my-branch to github and set it to track github/my-branch $ git push -u github my-branch # Make some existing branch track github instead of origin $ git branch --set-upstream other-branch github/other-branch" Actually, I don't know why I pay for Github private repo's.. I might as well set-up two origins, one at Gitlab and one at Bitbucket, for all my privates. Then keep Github as a public front-facing portal.
- robinhood 8y agoEvery company at some point has some kind of incidents. It just happens. GitHub is most of the time rock solid and doesn’t deserve to be judged based on one major incident like this. On the contrary they need our support. Bitbucket and GitLab both have had problems of the same magnitude.
- platinium 8y agoBut a multiple origin solution seems the most sensical. We have failover for everything infra and services.. it seems we now need failover for cloud-based code. It just seems logical, especially if all 3 of the big cloud providers have big incidents.
- shaklee3 8y agoI think it's more about people not looking for alternatives while things are going smooth. Now they will, even though the grass my not be greener.
- simlevesque 8y agoWhy does GitHub need my support ?
- 8y ago
- LaserToy 8y agoThis incident report tells very little. I hope they release what actually happened and how it affected their services. And how they are going to avoid it in the future. Almost any issue can be publicly described as: stuff broke because of network.
- deleted 8y ago[deleted]
- WorldMaker 8y ago"Partition failure" is a relatively specific term of art that implies it was a partial database failure, so "stuff broke because we lost part of a database". It may not be in GitHub's best interest to describe much more than that because which database, which tables, and how they were partitioned can be secret performance sauce. Postgres partitioning: https://www.postgresql.org/docs/9.1/static/ddl-partitioning.html https://www.postgresql.org/docs/9.1/static/ddl-partitioning.... Redis partitioning: https://redis.io/topics/partitioning https://redis.io/topics/partitioning
- zzzcpan 8y ago"Network partition" is unrelated to partial database failure or database partitioning. It means that database servers got disconnected from each other. Normally this shouldn't be a problem, but as far as I know they don't use proper distributed algorithms and so it's possible that disconnected servers each became masters and were serving requests independently, which is why they speak about inconsistencies. I believe this problem is commonly known as split brain [1]. [1] https://en.wikipedia.org/wiki/Split-brain_(computing) https://en.wikipedia.org/wiki/Split-brain_(computing)
- WorldMaker 8y agoIt's not "unrelated", it's an intertwined phenomenon, especially in practical consequences. In Redis, as one primary example I already mentioned, all database partitioning is network partitioning (and vice versa). The logical object is still a "database" even if the physical object suffering problems is the "network". For similar reasons that you suppose they "don't use proper distributed algorithms" (which seems an overly harsh way to put it), I simply presumed the logical entity in question is still a "database", even in the case of a network partition problem. I don't think it was either Postgres or Redis in this case, but they are simple enough examples to illustrate the overall problem. Either way, my case still seems to stand that we aren't likely to get more information about the specifics of the partition failure because likely both the logical (which "database") and physical layer specifics (which datacenters/"network") are things that are internal to GitHub that they may not be able to publicly describe much more than what they have.
- Ajedi32 8y agoWow, is this still going on? I noticed this yesterday and figured it would be fixed in a few hours, but [GitHub's status page][1] is still showing red. Is this the longest outage GitHub has had? [1]: https://status.github.com/messages https://status.github.com/messages
- sneak 8y agoWhy bother telling people five incorrect estimates for service recovery? https://twitter.com/sneakdotberlin/status/1054345379716558849 https://twitter.com/sneakdotberlin/status/105434537971655884... After one or two wrong guesses it just shows that you’re making it up.
- sneak 8y agoGitHub also silently published a private repo of mine this week. I checked audit logs for both the owner user and the org and it didn’t show a permissions change anywhere. Netlify has caused me to stop using GitHub Pages and between the clownshoes outage reports and the security issue I am now a GitLab user. This is GitHub’s jump the shark episode. :(
- andyana 8y agoGitlab has had its own issues.
- mr_toad 8y agoSomething that struck me yesterday as this started, is that Github isn’t really just a dvcs hosting solution, Github is a social network. It’s easy to change the remote origin of a git repository. It’s not hard to migrate a project to Gitlab etc, and duplicate all the technical features of Github. What’s really hard is replacing the social graph. If you’re a large project with a lot of contributors, onboarding everyone is not going to be easy. About as easy as convincing all your friends to stop using Facebook.
- wangyjx 8y agoThis was a head-on blow for MS?