18 ms·
GitHub was down
- pcr910303 7y agoHN already has a 'Github downtime' post as soon as I find Github weird and check HN if it's only me. How is everybody so fast? :-)
- chimprich 7y agoIf Github is down, thousands of programmers suddenly have nothing better to do.
- djsumdog 7y agoThe tragedy of the cloud.
- jbverschoor 7y agogit is distributed...
- kempbellt 7y agoYeah, but people are lazy and GitHub has a user-friendly UI with big green "Merge" buttons.
- mirekrusin 7y agoAnd ci integration; which runs automated tests; which must pass to release package/docker image/whatever; which is required for deploying new version for your system etc. The whole thing is as distributed as business of a guy selling hot dogs on the street.
- 0xffff2 7y agogit is distributed. The GitHub issue and PR tracker isn't.
- ithkuil 7y agoI rolled my eyes twice, then merged a few PRs manually and moved on with my day. (i.e. the git server itself and all the API required to interact with the CI automation _appears_ to work just fine)
- umanwizard 7y agoThe actual server was broken for me: $ git push Enumerating objects: 26, done. Counting objects: 100% (26/26), done. Delta compression using up to 8 threads Compressing objects: 100% (15/15), done. Writing objects: 100% (15/15), 1.49 KiB | 1.49 MiB/s, done. Total 15 (delta 12), reused 0 (delta 0) remote: Resolving deltas: 100% (12/12), completed with 10 local objects. remote: Internal Server Error To git+ssh://github.com/<redacted>/<redacted> ! [remote failure] wip -> wip (remote failed to report status) error: failed to push some refs to 'git+ssh://git@github.com/<redacted>/<redacted>'
- tasogare 7y agoConversely, if Hacker News was down GitHub would suddenly see a spike in traffic :)
- davidajackson 7y agoFortunately the first I saw of the one a few days ago started right around lunchtime.
- arparthasarathi 7y agoHN is the next place after Github that a programmer frequently visits ;)
- jansan 7y agoIf Github is down, the software developing world comes to a standstill. So people either get a coffee, take a shit or go to YCNews.
- kevinmannix 7y agoThis seems to be the third or so day in the past week I've had issues with GitHub around this time in the morning. They've typically been really good. I'm a bit surprised there hasn't been more talk about it on HN.
- giancarlostoro 7y agoThey seem to be doing heavy work on it. Now on Mobile you can't see repos in "Desktop Mode" which is unfortunate. I have to tell my browser to pretend to be in desktop mode. Plus the regex post from the other day seems to imply they are working on new things when somebody from GH replied in said thread. I don't mind improvements, but don't break production guys...
- izolate 7y agoWow. The apparent level of conceit with which you ended your message is astounding. Do you think they intend to break production? You don’t mind? There are real people behind this who are hard at work delivering us a great product, we should cut them some slack. I apologize if you meant it differently but it came off a little off-putting.
- askl56 7y agoIntentions are meaningless. If you're providing a service, (especially charging money for said service), you can't break it because "you're hard at work".
- cocire 7y ago> Intentions are meaningless Are you a robot? Have some humanity...
- desiderantes 7y agoGitHub is not a human being, it's a company.
- krallja 7y agoFourth time this month.
- charrondev 7y agoIt’s quite unfortunate. I’m hoping the post some kind of RCA/analysis. I know I would if we had a month with only 2 9s of uptime.
- nalllar 7y agothey're doing the usual corporate status page uptime lying: https://i.imgur.com/vy7onDT.png https://i.imgur.com/vy7onDT.png https://www.githubstatus.com/uptime https://www.githubstatus.com/uptime
- SuperSandro2000 7y agoI hope they catch up with this. If you guarantee SLA than you need to be honest with it or guarantee less.
- nalllar 7y agolooking again, the reporting under 'API Requests' downtime seems accurate but none of the earlier outages report downtime under 'Git Operations', which were also broken. didn't notice there was a selector for which type of downtime on that page that didn't default to all.
- rohansingh 7y agoI don't get it. Their own incidents tab on the same site shows four incidents this month: https://www.githubstatus.com/history https://www.githubstatus.com/history I'm not sure how that results in 99.98% uptime on the other tab.
- nalllar 7y agoturns out it's bad/misleading UI, there's a dropdown for which type of downtime which defaults to 'GIT Operations'
- savrajsingh 7y agoNo matter how many talented engineers you have on staff, your entire service can still go down. Let's pause and reflect on that. ;)
- pdelgallego 7y ago"You can't legislate against failure, but you can focus on fast detection and response" -- Chris Pinkham
- PenguinCoder 7y ago"Prevention is ideal, but detection is a must"
- iso947 7y agoIt’s amazing how this is accepted in the software world. Move fast and break things, such a different philosophy to other areas.
- dasil003 7y agoIt’s evolutionary pressure; software is malleable and potential functionality is limitless. Software companies that didn’t ascribe to this philosophy were repeatedly killed by ones that did until it became the status quo.
- dspillett 7y agoI think the quote from Pinkham is more about dealing with genuinely unavoidable failures, not those due to moving without appropriate due diligence.
- mattigames 7y agoI mean, that's rather disputable: The Apollo 1 exploded, medical mistakes have a toll of 250.000 deaths per year in the US alone; among many other serious mistakes on vast different areas, I think unreliability is unfortunately a constant on the human race.
- 7y ago
- Jaygles 7y agoHave they written any postmortems regarding their last couple of degradations? I tried searching their blog but the only ones that popped up were over a year old.
- SuperSandro2000 7y agoI would also be really interested in them but didn't find anything yet. Maybe their new notification system has something to do with it?
- bflesch 7y agoAre they migrating to azure?
- anticensor 7y agoPreparing the infrastructure for Windows open-source event?
- donkeydoug 7y agoI had the same guess... migrating from aws to azure & hitting some bumps. Have to assume they won't be very forthcoming about it if that is the reason.
- masklinn 7y ago> migrating from aws to azure & hitting some bumps Could be the IO? I remember colleagues working on getting stuff running on azure and they experienced horrible IO latency, as well as very low throughput for lots of small IO (aka unix-style software). Was a few years back so it might have improved since, but if those things are still non-optimal and github is built with a unix-style vision of tons of small IO access… This specific outage taught me that github apparently stores git repos on-disk which I was not expecting though (because the API access complained it could not delete repos until they'd fully backed to disk or something).
- akx 7y agoI don't think they mainly run on AWS, but on "bare metal": https://github.com/holman/ama/issues/553 https://github.com/holman/ama/issues/553
- speculatrix 7y agoI guess they switched github to run on Windows Server, and they forgot to turn off scheduled updates and automatic reboots!
- pcr910303 7y agoLooks like they are fine now. Yay!
- zymhan 7y agoStatuspage still isn't green https://www.githubstatus.com/ https://www.githubstatus.com/
- ivanfon 7y agoI'm still unable to leave any code reviews, and occasionally I'm getting error pages.
- glenneroo 7y ago> We continue to investigate the issues with GitHub services and will shift to a slower update cadence to provide more meaningful updates going forward. Posted 18 minutes ago. Feb 27, 2020 - 16:12 UTC
- aalleavitch 7y agoCentralization is great, isn't it?
- dana321 7y agoI think in a few years time we will look back and think "Waa, that was so archaic the way we used to do things!"
- eeZah7Ux 7y agoIt would be nice if someone wrote a decentralized VCS...
- mirekrusin 7y agoWith issues built in like fossil, oh wait...
- mirekrusin 7y agoGets philosophical as well - if you use laptop as a lamp in your tent - is it good or bad?
- anonsivalley652 7y agoWhen Microsoft buys companies, they tend to progressively decay as the original architects leave, the morale of remaining employees grinds down from the stress and they bring in cheaper contractors to duct tape the bits together and plug the holes in levee with their fingers. I've BTDTBTTS. cough LinkExchange, WebTV, Hotmail, Skype, Softricity, Nokia, LinkedIn, Danger/Sidekick cough GH maybe next. ¯\_(ツ)_/¯
- erikbye 7y agoIME GitHub has not had increased downtime after Microsoft's acquisition.
- anonsivalley652 7y agoWell, some other people in the comments disagree. And it hasn't happened yet, but it's the way they don't manage / integrate acquisitions very well unless they're wowie complementary products like Visio. Danger dropped off a cliff and Softricity was absolutely amazing but shelved, so friends of mine basically repeated the theme for VMware View and were acquihired by VMware. Time will tell where GH goes.
- bbrree66 7y agoThere are so many logical flaws here. People's comments are meaningless, you can look at historical GitHub up-time and see that it hasn't changed meaningfully. "And it hasn't happened yet" Ah yes, now you have to backtrack from: it happened! to... no wait I promise it will happen! Based on... what? The fact that some acquisitions don't go well? This is all pure speculation with no substantiation. I recommend learning about confirmation bias.
- endorphone 7y agoTo be fair, they didn't say that was the cause of the current outage. They made a general observation that Microsoft-acquired companies degrade, which seems like a fairly reasonable observation: The goal of being the best Git service/repo falls by the wayside as other corporate goals push in.
- arnonejoe 7y agohttps://downdetector.com/status/github/ https://downdetector.com/status/github/
- tomphoolery 7y agoI've been noticing the issue they're describing for a few days now, with errors in GitHub Actions requiring rebuilds and webhooks not seeming to fire which caused Jira to go out of sync.
- mirekrusin 7y agoMssql only once guaranteed delivery in action?
- waynenilsen 7y agoDon't forget to check your SLAs Enterprise = 99.95% (quarterly) https://help.github.com/en/github/site-policy/github-enterprise-cloud-addendum#enterprise-cloud-uptime-sla https://help.github.com/en/github/site-policy/github-enterpr... They're having a bad February but January was good. We will see what March has in store
- cle 7y ago> How do we calculate Uptime? > Our Uptime calculation is based on the percentage of successful requests we serve through our web, API, and Git client interfaces. Just curious, how do they measure this? What is the actual calculation?
- dorfsmay 7y agoSame here. They obviously don't have beacons on the client side, I wonder if it's based on statistics, at this time of the day on a Tuesday we should be getting x requests but are getting only x/n.
- Benjammer 7y agoWhat do you mean? All modern enterprise analytics/monitoring solutions are going to be able to give you some kind of top-level "request success rate" metric. I assume they mostly just lean into whatever monitoring tooling they have set up. What kind of "calculation" are you imagining here? Like a very specific SRE formula for availability windows or something?
- ArchReaper 7y ago>We continue to investigate the issues with GitHub services and will shift to a slower update cadence to provide more meaningful updates going forward. Translation: our shitty software update practices are now affecting Github, not just Windows! If anyone from Microsoft is reading this, why is your company so incompetent at software updates in the past few years?
- scumbert 7y agoIs it not ostensibly the same company, just with a new owner...?
- barbecue_sauce 7y agoInitially, but that is subject to change or more likely evolve over time.
- applecrazy 7y agoGithub is part of MS, but as I understand it, is run separately from the rest of Microsoft.
- herbderb 7y agoI think that's talking about status update cadence, not software updates.
- salmon30salmon 7y agoYou honestly believe that a company the size of GitHub has had their software update practices appreciably change since the acquisition? Relax, GitHub had update issues before and they will have them again.
- MrMorden 7y agoAll of this has happened before and all of this will happen again. So say we all.
- uniformlyrandom 7y ago
- thereyougo 7y agoI appreciate the way they keep everyone informed in the downtime. It tells a lot about the company
- zeisss 7y agoThat explains why my test suite suddenly takes >1h and after I canceled it, the status was green O.o
- vivan 7y agoWhat do you guys recommend as a good way to continue work undisrupted when GitHub goes down? A second remote mirror?
- cuspycode 7y agoUse self-hosted repos like gitlab or fossil, and then mirror the public parts to github.
- roblabla 7y agoA second mirror doesn't really help - when github goes down, the code should still be available locally on your computer. The things that become truly available when github dies are all the non-git features: issues, PRs, etc... There are several ways to work around this, but none are really satisfying.
- mirekrusin 7y agoGit or fossil mesh.
- natfriedman 7y agoSincere apologies to all GitHub users for the downtime this morning, and the brief outages last week as well. We take reliability very seriously, and will publish a full RCA in the near future.
- latchkey 7y agoThanks Nat! Keep up the good work and thanks for contributing here. Github is still my favorite. =)
- MuffinFlavored 7y agoWhat did you guys deploy/what scale tipping point did you guys hit that caused the past 3 days of problems? At my job, if something we go wrong... management just tells us roll it back. That always fixes the problem, right? :P
- uniformlyrandom 7y agoYep, it is Dunning-Kruger effect. If rolling back works for a simple system, why wouldn't it work for a complex one? Because one cannot step into the same river twice. Heraclitus would probably make a good engineering manager.
- tcbasche 7y agoThis is 100% not the Dunning-Kruger effect. How on Earth would it be? edit - from wikipedia: In the field of psychology, the Dunning–Kruger effect is a cognitive bias in which people assess their cognitive ability as greater than it is.
- behindsight 7y agoMaybe not directly but it could still apply. I interpreted GP as saying the aforementioned management has just enough cursory knowledge to want to apply the same hammer that worked on a simple system to that on a complex system, but not enough knowledge to realize the unknown-unknowns that they aren't even aware of.
- dvdhnt 7y agoWell, given the timing of GitHub reliability issues over the last few days, I think we can all agree it has everything to do with dates and timezones? /s Appreciate the work.
- lcfcjs2 7y agoMicrosoft takes over and fucks it up. Let me guess, its hosted on Azure.