24 ms·
Status.github.com: “We're failing over a data storage system”
Updates to gist are getting lost. It accepts them as normal but the next page load for the gist is the previous version. Twitter accounts describe the same sort of problem for git repos.
status.github.com reports "We're failing over a data storage system in order to restore access to GitHub.com."
- 0freeman00 8y agoI think they started running over Microsoft infrastructure. ;)
- reindeerer 8y agoNice, they already migrated to MS SQL i see.
- mohammedbin 8y agoSnark, except snark only comes from people with little knowledge, as in this case you seem unaware that they are a completely independent entity as of today.
- benatkin 8y agohttp://howfuckedismydatabase.com/mssql/ http://howfuckedismydatabase.com/mssql/
- dfcowell 8y agoPR comments are also failing with HTTP 405 error code. Great way to start the week.
- keyle 8y agoWell for the poor sods fixing it in the US, it's still Sunday evening...
- geerlingguy 8y agoIt's when I typically get an hour or two to crank out some open source PRs and issue queue cleanup. Sadly, the outage means I don't get that time to devote this week :(
- hayd 8y agoSurely they have SRE teams around the world?
- karaokeyoga 8y agoIs it just me or is that status message wonky? Failing over something to restore access?
- karaokeyoga 8y agofailing -> migrating
- pinneycolton 8y agoWell, they've got the "failing" part right. It's the "over" that's taking a while ;)
- rphillips 8y agoSounds like an engineer knee deep in diagnosing the issue.
- diegoperini 8y agoThat person needs our support more than anyone else right now.
- radicality 8y ago"Failing over" in this context most likely means either migrating to some replica (say doing a dead master mysql promotion) or spinning up some backup storage system to serve master reads/writes.
- cesarb 8y agoIn this context, "failing over" means switching the active system to a replica. Which means their primary data storage system had some issue, and they were switching to a secondary data storage system, which hopefully contains the same data (replicated in real time, or nearly real time). Clearly that switch didn't quite work as expected, otherwise it would have taken just a couple of minutes before everything went back to normal...
- samontar 8y ago
- 6t6t6t6 8y agoNext: "We are restoring tape backups from some of our storage systems"
- 6t6t6t6 8y ago"We are continuing to repair a data storage system for GitHub.com. You may see inconsistent results during this process."
- newnewpdro 8y agoEnjoy hosting your source code with Microsoft.
- hartator 8y agoI can access GitHub.com here (Austin,TX), but it's taking forever.
- igni 8y agoBefore all the trolling about Microsoft starts up, does anyone have current information on what these systems are? In the enterprise space, a 'data storage system' could be an Array or a SAN or a lightpath etc, with usually quite long failover times. For an org like GitHub I'd think more like an object store (an Array-of-Hosts, if you will) or whatever storage mechanism holds their database files. Do they self host this sort of thing or is it an AWS/GCE/Azure service? FWIW, all git commands are working fine for me (create a branch, push, colleagues can fetch my branch), but the UI doesn't show my branch & nor can I review/comment on PRs.
- reindeerer 8y agoI think it's obvious that it's their SQL storage that holds the website up that is pretty much in read-only mode, and has been for a couple hours now, not the git repos themselves.
- geerlingguy 8y agoYeah, been trying to post comments and new issues and keep getting "405 Not Allowed" responses.
- sciurus 8y agoFor storing things other than git repos, GitHub is heavily invested in MySQL. AFAIK all of GitHub is hosted on their own hardware. https://githubengineering.com/mysql-high-availability-at-github/ https://githubengineering.com/mysql-high-availability-at-git...
- rurban 8y agoThis blog post describes exactly the scenario we were experiencing here. A master (single writer) failure, with missing fail over. You can only guess what went wrong with this plan. Looks good on paper, but some unexpected network or HW or routing problem could have caused the problem to identify the single writer.
- friedman23 8y ago
- beamso 8y agoSeeing a lot of issues with Pull Requests.
- RandomGuyDTB 8y agoEarlier GitHub pages was "down for maintenance". Probably not related but might be of note. (source: my website is hosted on github pages)
- gavreh 8y agoMy issue comments are not being saved, and if I do a tag push I don't see that reflected on the "releases/tags" area of GitHub.
- diegoperini 8y agoI just lost my session.
- keyle 8y agoI don't know if they keep changing the text updates with a slightly different version to - prove it's a human that typed it - there is code the prevents repeating twice the same message either way it's entertaining... But it's Monday morning in Australia and we need to release! (yep we do this via pr/tagging etc.)
- Fuzzwah 8y agoIt is to ensure that the updates get mirrored onto twitter, where exact duplicates can't be posted.
- hiccuphippo 8y agoIsn't it enough to delete the older tweet? Or maybe just add a timestamp to the message. Off-Topic: If it's not possible to write a twit with the exact text from a deleted twit, then a way to prove someone wrote a twit and then deleted it would be to have them try to write it again.
- askmike 8y agoDeleting tweets is a terrible workaround. Once github start tweeting and people start linking to those tweets they can't go ahead and delete them 50 minutes later..
- owl57 8y agoRe: off-topic: You have to do it fast. You certainly can tweet a lot of identical tweets if you aren't in a hurry: there is a "Rock in the forest"[1] that tweets "Nothing happened today" (in Russian) every day for several years, and most of the time the text is exactly the same. Interestingly enough, the time of tweet varies widely from day to day, so it looks like a human or a program specifically made to imitate one. [1] https://twitter.com/kamen_v_lesu https://twitter.com/kamen_v_lesu
- mehrdadn 8y agoThey can't alternate?
- Memosyne 8y agoAnyone know if this could be related to the Youtube outage? It's been a while since I've seen these big websites go down.
- bognition 8y agoIn what way? Youtube runs on internal google hardware, GitHub runs on its own hardware too. It could be a networking issue, but you'd expect more sites to be impacted. If it were a software issue, you'd expect a big player like google to be aggressively patching and talking about a bad release. More likely is its just a coincidence.
- danielhlockard 8y agoI'm not sure how you could postulate that they're related
- ObsoleteNerd 8y agoTargeted attacks on specific aspects/assets of a major websites infrastructure? Eg target a specific service that both sites have in common in their back end? Not even sure if that's feasible but it's an idea.
- Memosyne 8y agoI figured it was just a coincidence, but the HN network is so well informed I thought I'd just ask the question on the off-chance it wasn't. I personally don't find a relation likely.
- ObsoleteNerd 8y agoOh I find the relation very unlikely. I was just trying to think up a potential connection for fun/curiosity, as the coincidence is pretty crazy that two huge sites known for very few/little outages had major outages in the same week.
- jpatokal 8y ago
- geggam 8y agoInteresting how everyone uses a tool designed to eliminate SPOF in a way it has a SPOF.
- geerlingguy 8y agoGitHub does not equal git; the reason I’m paused is because I use Github’s issue queues to organize my OSS work. Much easier than self hosting an issue repository/bug tracker. I am still able to do all my work, run new containers, etc., but GitHub is more tied into business processes than actual code (which is what causes the pain during these outages).
- dfcowell 8y agoThis is certainly making me rethink the workflows we are using in my team, particularly those around PRs and code review.
- ____Sash---701_ 8y agoWhat!?
- platinium 8y agoThis has been super frustrating, as people have deadlines and are working to finish projects before Monday morning. What are the (good) alternatives to Github? Gitlab supposedly is Google-backed, so I don't want to have my private code there. Is Bitbucket the only one left? I don't mind paying monthly, which I already do for GitHub.
- Tecuane 8y ago> Gitlab supposedly is Google-backed, so I don't want to have my private code there. This strikes me as odd. May I ask why usage of GCP is a deal-breaker for you? While I can understand not wanting to use Google products directly as a consumer, I believe it would be - for lack of a better term - platform suicide for Google to intercept and perform its usual analytical shenanigans on the data content of transmissions to/from their platform. Either way, Phacility's Phabricator[1] is $20/user/mo. 1. https://www.phacility.com/pricing/ https://www.phacility.com/pricing/
- Jedi72 8y agoNobody trusts Google for any reason any more as they have proven unworthy of our trust.
- hactually 8y agoThat's a silly statement. By saying 'nobody' - a single point of data invalidates your assertion. I use gmail, I'm quite happy to trust that contract.
- jammygit 8y agoYou're saying that you trust that contract today, or are you saying that you have always trusted that contract? Its only recently that gmail's contract involved keeping out of your data. I think they also only say they abstain from using your data for targeted advertising, not that they don't use it for other purposes. I haven't read the terms in quite a while though and I could be mistaken. Great products though. I really do wish I could pay for them in exchange for a real, trustworthy, comprehensive privacy promise.
- ComputerGuru 8y agoI can't add comments to pull requests("you can't do that right now") and any commits pushed to branches are not updating updating the visible status in the web interface, no are newly created branches showing up. However, if you navigate to a new commit directly with its SHA (so you can share it with someone if you really want), it'll show up (so they're just not being indexed). EDIT: Obligatory "that's what happens when the whole world relies on a centralized git repo" and a reference to gitea, which has a very slick github-esque UI and is incredibly easy and light to deploy/run (on an existing server, your own PC, a raspberry pi, a docker vm, or whatever): https://gitea.io/en-us/ https://gitea.io/en-us/
- rodorgas 8y agoI’m receiving emails (a lot of duplicates) from comments in PR, but they won’t show up in the browser. I guess people are trying to submit multiple times, the email is sent but comment isn’t posted.
- majidazimi 8y agoWelcome to eventual consistency.
- hnarn 8y agoMañana Consistency(tm)
- samuel1604 8y agoyes, but does it scale tho ? is it DR ? I mean sure fine it's easy and pretty but there is some operation challenge to figure out,
- perlgeek 8y agoIf you host just your own Open Source projects, does it need to scale?
- theSage 8y agoI'm really looking forward to the post mortem that comes out of this (if it does). I always learn a lot from reading those.
- reindeerer 8y agoA post-mortem from Microsoft ? nfw
- JoyrexJ9 8y agoGitHub are not part of Microsoft yet, the deal has only just been approved. They are still totally separate, and will be for some considerable time. Also Microsoft post plenty of postmortems, like this detailed one from the VSTS outage in Sept. https://blogs.msdn.microsoft.com/vsoservice/?p=17485 https://blogs.msdn.microsoft.com/vsoservice/?p=17485
- deleted 8y ago[deleted]
- olingern 8y agoLocal to Tokyo, Github has been down for most of the day. I never realized how much of my day centers around it: pr review, creating / commenting on issues, etc. Looking forward to the write-up. I'm also curious as to how this significant outage lines up with their SLA for enterprise users.
- guywhocodes 8y agoWhat's strange to me is how many hours we've been given the same message. Today is an important day for my organization and not getting any information that hints how long this will last is a huge problem for our planning.
- steventhedev 8y agoAre you a paying customer of GitHub Enterprise? If not, then you're getting your money's worth. Snark aside, this is a great time to reassess your deployment strategies and look into things like local apt and pypi proxies. I'm confident you can find similar projects that will transparently cache your dependencies.
- guywhocodes 8y agoI don't know how much my org is paying but it's no small amount, Enterprise no. Are we getting what we pay for? I don't think so, I think effective SLA of 99% isn't good enough for any SasS. But of course you understand that moving away from github is no small decision. Absolutely is this a perfect time to assess deployment strategies and challenge all the advice of how big a company has to be before it's worth to do X.
- steventhedev 8y agoIt sounds like you're using GH for ticketing also, which means that your R&D org's productivity is tightly coupled with an external companies uptime. Self-hosting Gitlab is easy, as is running Jira, redmine, and a dozen other tools. If you can't do your job without a tool, then you need to have a plan B on hand for when that tool fails. Both in the micro sense of the tools you use to code (editor, browser, laptop, mouse, coffee mug, etc) and in the macro sense of tools your organization uses (ticketing systems, chat, bathrooms). Show some initiative, figure out some mirrors for your dependencies, and try standing up a local caching proxy for your team. It'll probably take a lot less time than you think.
- samontar 8y agoIf you’re paying for GHE you don’t have any trouble. That’s delivered as an appliance you host.
- BadassFractal 8y agoWonder if GitHub is "too useful to fail" at this point. As in, most people and companies won't switch git repo providers unless GitHub is down for many days at a time during the work week.
- avip 8y agoTransition to BitBucket or gilab is one click away. Companies surely will move if incentives are there.
- mellisdesigns 8y agoWhat particular services are down? I noticed I can hit the UI.
- avip 8y agoFor me: can't fork, can't login (tested incognito), can't clone new repos.
- julienfr112 8y agoIs that a job of the Microsoft special project team ?
- CodeM0nkey 8y agoGitHub has not yet been acquired by Microsoft, they're completely separate.
- arthurcolle 8y agoI wonder if this will vaporize the deal.
- manigandham 8y agoThat's not how these deals work. A little infrastructure downtime is not going to stop a 7.5B platform acquisition.
- gyvastis 8y agoGreat takeover, Microsoft.
- jypepin 8y agoIt's been over 6hrs without update, except the hourly message which states the same. I'm really looking forward this post mortem, but having been in the situation where I've had to deal with large scale outages like this one, I guarantee some engineers are having a bad time right now, and I feel for them. Github engineers, if you are reading me (probably not), KEEP IT UP, it happens to the best of us! <3
- shry4ns 8y agoI don't know if someone's mentioned it yet but TravisCI is also not running on the commits that do show up on git, in Houston
- stephenr 8y agoObligatory: to all the people running around like de-headed chickens: this is what happens when you focus on a service not on the dvcs. Your code and your wiki (if you use it) are already in git repos. If the industry moved forward on issues in dvcs repos, you have another central point of failure removed. You also have less hassle to work across dvcs hosts, which is exactly why GitHub would never embrace this.
- nodesocket 8y agoI created a gist four hours ago, which as of now is not returning and showing a 404. Hopefully the gist did not get lost.
- josteink 8y agoSeems like they're on their way back up: > We are currently in the later stages of a restore operation, with the aim of serving fully consistent data within the next 2 hours."
- _shadi 8y agoI always thought it is silly that we pay for both private repos on github and host our own enterprise github on premise, turns out whoever have set this up knew what they were doing.
- calmconviction 8y agoThey probably moved their storage backends to Windows 10
- r_singh 8y agoJust a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab and github will cost the same in the long run. Time costs money and self hosted has a lot of downsides as well. However, gitlab being self hosted (and not costing anything in terms of separate invoice) makes us feel like we're more in control and are somehow saving some money (as we add more developers as well). This comment here was just sparked by a sigh of relief because this week is an important release for us and had we been on github this monday wouldn't have started too well. Overall, all the comments listing the trade-offs are true. Time costs $$$ and a paid hosted solution is worth the minimal cost, time and expertise required.
- woolvalley 8y ago+ cost of additional labor
- ToFab123 8y agoPlus cost of nobody can work because their single server crashed
- Grue3 8y agoGit is a distributed system. People can work if they want to, but it's certainly a convenient excuse to not be working.
- deadbunny 8y agoSure, but the tooling around git isn't. Hard to do things if you can't make PRs, which trigger tests, which trigger builds, which trigger test deployments, which trigger tests, etc...
- 8y ago
- reidrac 8y agoSomeone posted a comment on an open issue, I got the mail, but there's nothing on the web interface. What I don't understand is why they don't set it all to "read only" until the problem is sorted. Looks like any update goes to /dev/null, just let your users know! (unless they plan to replay those updates, somehow)
- deleted 8y ago[deleted]
- askmike 8y agoI don't think it's easy to predict what is going to happen to all updates now. They run a big distributed infrastructure. I doubt they can even predict which updates will be commited and which ones will not. If replaying is even on the table (which sounds very dangerous to me), that requires a huge coordination and I am pretty sure they are not able to tell right now if that is going to work once the actual issue is fixed. Better to stay quiet while fixing the issue and only say things that you actually know 100% to be correct.
- k_ 8y agoAn update has been posted there: https://blog.github.com/2018-10-21-october21-incident-report/ https://blog.github.com/2018-10-21-october21-incident-report... > At 10:52 pm Sunday UTC, multiple services on GitHub.com were affected by a network partition and subsequent database failure resulting in inconsistent information being presented on our website. Out of an abundance of caution we have taken steps to ensure the integrity of your data, including pausing webhook events and other internal processing systems. > We are aware of how important our services are to your development workflows and are actively working to establish an estimated timeframe for full recovery. We will share this information with you as soon as it is available. During this time, information displayed on GitHub.com is likely to appear out of date; however no data was lost. Once service is fully restored, everything should appear as expected. Further, this incident only impacted website metadata stored in our MySQL databases, such as issues and pull requests. Git repository data remains unaffected and has been available throughout the incident. > We will continue to provide updates and an estimated time to resolution via our status page.
- k_ 8y ago1.5 hour later (a couple minutes ago), despite an estimated time of less than an hour: > We continue to monitor restores which are taking longer than anticipated. We estimate they will be caught up in an hour and a half.
- woogle 8y agoThey've just posted an "Incident Report": https://blog.github.com/2018-10-21-october21-incident-report/ https://blog.github.com/2018-10-21-october21-incident-report... > Multiple services on GitHub.com were affected by a network partition and subsequent database failure resulting in inconsistent information being presented on our website. Out of an abundance of caution we have taken steps to ensure the integrity of your data, including pausing webhook events and other internal processing systems. [...] > Information displayed on GitHub.com is likely to appear out of date; however no data was lost. Once service is fully restored, everything should appear as expected. Further, this incident only impacted website metadata stored in our MySQL databases, such as issues and pull requests. Git repository data remains unaffected and has been available throughout the incident.
- tomas789 8y agoMy Jenkins is creating CI Builds like crazy. If your CI is on the pay-as-you-go service, make sure you are not burning money (or credits).
- dustinmoris 8y agoGitHub must be migrating to Azure...
- guardian5x 8y agoActually, they just announced to stay with AWS, as long as things run well. Maybe now they have a reason to.
- benmmurphy 8y agoMy guess at what happened: They had a split brain when multiple masters were running. Then they were not able to choose a master to keep because the data in both masters was 'corrupted' so they are now restoring from a backup. So how do they get data corruption from multiple masters running: 1) Performing reads from slaves during an update operation. If you perform a read from a slave then you might get data from the other master. If you update data on one master based on data from another master then you get data corruption. Probably they don't do this much because if you had any slave lag then you would notice this problem during normal operation. However, they might do it for checking permissions. You can imagine because this data almost never changes it would never show up as a problem normally. 2) Data stored outside of the database. This would be the repositories themselves and cookie-session storage. Imagine repos have an incrementing id. Then if you have two masters a repo gets created with the same id on both masters. This is very bad because now two people can see each others data. You have the same problem with cookies. Imagine if you have user id as an incrementing id. The two masters create a user with the same id and an encrypted cookie or another storage system (redis) stores the user id. Now depending on which master you get routed to you appear as a different user. Weirdly enough this would always be a problem based on how their failover system works. The only safe way i know how to turn a master-slave system into a safe HA system is how joyent does it (https://github.com/joyent/manatee/blob/master/docs/user-guide.md https://github.com/joyent/manatee/blob/master/docs/user-guid... : you basically need a vote at commit time. if you have this then only 1 master can commit). However, I'm guessing most of the time they only have two masters running for < 1 minute but probably this time they had two masters running for a long time. EDIT: oh. they said it affected issues & pull requests [they have different dbs for different stuff] so repo and authentication stuff wouldn't apply. oh, they said no data was lost as well so presumably that would exclude a split brain running for a long period of time.
- sudhirj 8y agoSplit brain can run for a while if the all identifiers are UUIDs and tables are used as append only. Restoration is complex, though.
- benmmurphy 8y ago
- biddlesby 8y agoThe status updates are bearing a remarkable resemblence to a Windows loading bar. • We aim to serve fully consistent data within the next 2 hours. • 45 minutes later. On track for serving fully consistent data within the next 1.5 hours • 45 minutes later. On track to serve consistent data within the hour. • One and a half hours later. We estimate they will be caught up in an hour and a half.
- fredley 8y agoThey got to the end of the restore, realised there was a problem, and had to start over.
- tjoff 8y agoI appreciate that they try. As a user I would never rely on those estimates, but it at least gives you a sense of what's going on and what they are doing. I take that over the more common silence or "we are working on it" any day.
- Ma8ee 8y agoExactly. We all know not to trust that those estimates are exact, but they are much better than nothing. When they say nothing we don't know if it's half an hour, half a day or half a week.
- radiospiel 8y agoWell, so they say it is within the hour, but we still don't know if it's half an hour, half a day or half a week.
- OnlyRepliesToBS 8y agoPing? Pong! It's still useless to us.
- sajithdilshan 8y agoI think it is obvious what happened. They must have moved their backend to Windows 10.
- sajithdilshan 8y agoIt's a joke people. Why downvote?
- deadbunny 8y agoBecause it's a boring, obvious joke which wasn't funny even in the 90s?
- megakid 8y agoI'm trying to add a new user to my organisation (Monday morning, new joiners...!) and despite getting confirmations I purchased another seat, I cannot invite them. Or when I do, those seats are gone, guess it's down to which data store I'm hitting on each request.
- mattio 8y agoThey started to replay all webhook events yesterday evening, triggering all kinds of events in our systems, as deployments and the like :-(