11 ms·
Has GitHub been down more since its acquisition by Microsoft?
- lwheelock 6y agoIrony aside, there is some degree of credibility lost when offering a critical opinion on availability and your cloud hosted site throws a 502
- mikece 6y agoAny chance that prior to acquisition there was more focus on stability —- because it was assumed there would be a buyer —- than upgrading to the latest version of Rails and adding features?
- battery423 6y agoIt could also mean that they are more active and creating more chaos due to more changes. shameless and useless advertising.
- blueflow 6y agoPersonally, I'd prefer some availability over sparkly features.
- echelon 6y agoNot counting the redesign, there haven't been "sparkly" features. Sparkly features are the kinds of things the old management used to let engineers have free reign to implement. Github pre-Microsoft was rudderless. They were more prone to implementing silly 3d model diff tools and things instead of supporting enterprise features or building powerful CI/CD tools and automation. New Github is on the right track.
- blueflow 6y ago- Personal Status - Round Avatars (all catgirl ears are now cut off) - Collapsing of some similar messages on the Dashboard into one (i missed some things because of this) - Not just one, but several redesigns (im old) - The whole "marketplace" functionality (imitation of an app store) - The "explore" and "trending" functionality (i see this like a Facebook feed) Not sure about others, but i used GitHub as a git host and issue/MR tracker. All the other stuff is just distraction in my eyes.
- echelon 6y ago> (a bunch of comments about the UX redesign) I agree. I'll add one: README files with tables now require horizontal scrolling, and that's utterly disappointing. > The "explore" and "trending" functionality I do work in ML and computer vision, and this is how I discover all of the new models and code people are using. It's awesome. > Not sure about others, but i used GitHub as a git host and issue/MR tracker. All the other stuff is just distraction in my eyes. You're missing out! Deploying code is so easy once you're using CI/CD. Github actions are powerful.
- blueflow 6y agoMany of my projects have a fixed scope and thus are 'done' at some point. If you develop like this, CI/CD is rarely worth the effort. Its better suited for projects that suffer from scope creep or regular upstream breakages.
- robertlagrant 6y agoYeah it's only needed for projects that might change. Anything you can guarantee will never change, is fine.
- ies7 6y agoBut before that ‘done’ point or maybe for a new project, Github Action is a good feature introduced by Microsoft.
- danpalmer 6y agoOn my team our downtime is very closely correlated with how much we're working and changing the product. I'd be surprised if GitHub is much different.
- waheoo 6y agoOr in other words, facebook is more stable on the holidays.
- virtue3 6y agoexactly. Before M$ there wasn't much product movement at Github :/
- danpalmer 6y agoI think they got out of the rut before Microsoft's involvement, but I suspect MS has allowed them to accelerate a lot of it with more investment and a more defined direction. I've heard that for a number of years the Product org at GitHub essentially considered the product "finished", and that it did not need more features. Things like the rise of GitLab and the "Dear GitHub" letter, plus I believe a change of leadership, helped that completely turn around. Obviously that took time to yield benefits, so I think they've been in a much better place for quite a few years now.
- Lammy 6y ago> and a more defined direction Except for Atom :(
- mrmonkeyman 6y agoThing is though, they were right. All this nonsense features are literally messing up (a big part of) their core business: uptime.
- bnegreve 6y agoThat's pretty much their conclusion > [..] GitHub has been down more since the acquisition by Microsoft. But that could be all a part of coordinated effort to be more transparent about their service status, an effort that should be applauded.
- robertlagrant 6y agoThat's not a conclusion, it's a theory.
- jbarberu 6y agoOr hypothesis.
- deleted 6y ago[deleted]
- tnolet 6y agochanges = downtime. Has been since the advent of computing
- h91wka 6y agoNot if you have a competent devops team and a deployment pipeline.
- marcosdumay 6y agoYour devops team assures software quality? But well, no QA effort is complete enough to really assure the quality of a large system. Thus, the GP holds true.
- battery423 6y agoNonethless we are building and operating complex systems. Testing something 100% has diminishing returns. Therefore -> you will never be able to prevent all issues. As you are not aware how many changes are happening, you don't have a value which could indicate the healthiness of it. I personally think that when i'm getting older and better in my job, i still make errors, the amount and severity is going down though.
- marcinzm 6y agoI'm sure they have both but complex shit breaks when you change it often.
- IceWreck 6y agoYeah really. I feel the same thing. Can't comment anything right now and Ive been trying for the past hour. It dies every three weeks.
- Denvercoder9 6y agoIronically this link is down for me.
- colinbartlett 6y agoAuthor here, just put a CDN in front of the blog so maybe that will help.
- shawabawa3 6y agoCDN's not working i'm afraid (SSL error, 403 forbidden if you ignore it)
- robertlagrant 6y agoNot yet :)
- RMPR 6y agoWas about to say the same, but it's working now
- majkinetor 6y agoDefinitelly by my perosnal experience. Part of it might be due to covid, but it started happening before that. On the positive side, github had tone of changes since then, all seem to be good ones, so its understandable that it has more problems now as well.
- rvz 6y agoYes. Last time it was down was just 6 days ago: [0] I think its time to look for alternatives or frankly in the long term, follow what some open-source orgs are doing and self-host instead. [0] https://news.ycombinator.com/item?id=23604944 https://news.ycombinator.com/item?id=23604944
- Fiveplus 6y agoIronically, Firefox tells me to stay away from your website. Which, I will. update: the website is back and well now from my end.
- sbmthakur 6y agoI am also using Firefox. It blocked Google analytics on the site but did not get a warning.
- colinbartlett 6y agoI can't reproduce, would you mind emailing me a screenshot of this? (I'm the author) colin at any of the domains in my bio.
- Fiveplus 6y agoI would have happily complied. On a welcome note, however, the backend changes made after the initial hickups brought the website back on its feet.
- brylie 6y agoThey are porting things over to ASP.NET on Azure.
- cdbattags 6y agoSuch an underrated comment. I applaud.
- quickthrower2 6y agoIf the implication is that asp.net or Azure are unreliable then think again
- quickthrower2 6y agoIf it has, what is p?
- tmpfs 6y agoBoth the link and github comments are down for me too; I am unsure whether it has been down more since the acquisition but this is unacceptable to me. I have been trying to post a comment to an issue i made for the last few hours and every time it says "You can't comment at this time" which I find very misleading. It should say "Sorry, our system has a problem, please try again later". The implication of the current message is that my permissions are wrong or something else has happened. This has bumped self-hosting all my repositories much higher up my list.
- scribu 6y agoIt's easy to see from the uptime history [1] that there have been many more incidents in April-June than in January-April. Don't know if that has anything to do with the Microsoft acquisition, but it is concerning. [1] https://www.githubstatus.com/uptime?page=1 https://www.githubstatus.com/uptime?page=1
- rattray 6y agoWow, that's stark. I count 6 in April-June (2 major outages) and only 1 in January-March (0 major outages. I wonder if COVID has affected this somehow. Anecdotally I've heard of at least one other ~peer company with a large rise in incidents/outages since April. Strange since both companies previously had a strong culture of remote work (maybe 1/4 to 1/3 of eng was remote) going into the pandemic, so I'd be quite surprised if all-WFH contributed somehow...
- rb12345 6y agoIf GitHub's been migrating to Azure infrastructure on the backend, it's possible that everyone else moving to all-WfH helped to cause it. Teams use massively increased during lockdown for obvious reasons and that's definitely Azure-based. If Teams managed to overload Azure capacity, I could see that having knock-on effects on GitHub.
- rattray 6y agoThe other company does not use Azure.
- robertlagrant 6y ago> it's possible that everyone else moving to all-WfH helped to cause it No, MS prioritising Teams land grabbery is what caused it.
- esperent 6y agoThey're still averaging 99.81% up-time over the last three months. This means they have been down for a total of 4 hours out of 90 days. I don't think it's all that concerning.
- vadasambar 6y agoMirror on wayback machine: https://web.archive.org/web/20200629103627/https://nimbleindustries.io/2020/06/04/has-github-been-down-more-since-its-acquisition-by-microsoft/ https://web.archive.org/web/20200629103627/https://nimbleind...
- robertlagrant 6y agoHas nimbleindustries.io been down more since Github's acquisition by Microsoft?
- thescribbblr 6y agoYes
- bryanrasmussen 6y agoI think it's pretty fair to say we've all been pretty down since GitHub's acquisition by Microsoft.
- waheoo 6y agoI'm on sourcehut. Whats a microsoft?
- ToolsDevler 6y agoIt sounds like a kind of toilet paper
- bryanrasmussen 6y agoit must be a very niche market, people who prefer small toilet paper.
- dilandau 6y agoI got a sheet of Microsoft. See you guys at the dead show.
- squarefoot 6y agoA few years from now it will be the brand of the most widely used Linux distribution.
- 6y ago
- m12k 6y agoWhat I don't get is how they can be down for hours on end. If I make a deploy that turns out to be buggy, I'll roll my site back to a stable previous version within minutes. Sure, that's not always possible if I've made incompatible database schema changes, but in my experience those are very, very rare (i.e. I almost always add db columns, and only rarely delete or rename columns - and when I do, I do so after those columns haven't been in use for a while).
- Softcadbury 6y agoI'm sure these down times are more about infrastructure problems than just database updates and I'm also sure that this infrastructure is a little more complexe than your site.
- m12k 6y agoI'm sure their deployment process is way more complex than mine. I'm also sure that unlike me, they have engineers dedicated to managing the complexity. GitHub is a company that has blogged about how easy they have made deployment and how they deploy many times a day. I don't believe there are any compelling reasons why that ease of deployment shouldn't also extend to re-deploying previous versions of their service, so they don't need to leave a bad version up for hours.
- scaryclam 6y agoI think you're missing the point a bit. Not all changes are simple code deploys. They take time to diagnose, mitigate and fix. This is especially true when you have a lot of services that talk to each other, working on infrastructure that can support so many services and users.
- cranekam 6y agoIt’s often not as easy as deploy -> immediate problem -> rollback. Problems can take a while to diagnose, or may cause some kind of poisoning that needs to be fixed (eg rebuild a lost or corrupt cache), or be in some part of the system that nobody knew was related (eg maybe someone deployed code that talks to a hitherto-unqueried accounting system and that worked fine at 4pm on Thursday but come 9am Monday it melts). My point is that in big complex systems sometimes there is not a straight line between cause end effect. Sometimes there’s just effect and you need to work out the cause.
- solidninja 6y agoI really wish there was an option to decouple the core git functionality from the 'fancy' stuff (PRs, actions, issues) etc. When your CI pipeline runs directly from your git repository, not being able to commit/merge means you cannot release to production. For binary artifacts, you can get around dependencies on a single provider by mirroring, but because git is mutable, you can't just mirror the repo and allow changes there, because you need the same permissions, ssh keys etc. for the repositories and because changes will need to be synced back to the source repository. You might as well not use Github in the first place and self-host (with all the problems that entails). As far as I know, none of the major git providers offer this - I've experienced outages with GitLab, Bitbucket and GitHub that all affected the production environment (luckily, it's never been critical so far).
- hashhar 6y agoThat is exactly how GitHub is architectured. There have been a lot of time I've been to able to push/pull but GitHub pages was down or their web backend was down. Bitbucket is the worst in this regard.
- nitinreddy88 6y agoThere's already one news on front page https://news.ycombinator.com/item?id=23675864 https://news.ycombinator.com/item?id=23675864 Can we merge them? Mods
- heinrich5991 6y agoMail hn@ycombinator.com to talk to the mods (see footer).
- st_goliath 6y ago> There's already one news on front page And the comment that is currently at/near the top and mostly discussed happens to link to exactly this site.
- bovermyer 6y agoI'm more inclined to believe this is due to the increased reporting and attention to the status rather than an actual increase in downtime. However, I have no hard data to suggest that, only my experience as someone who's had to manage and maintain reports like this.
- zxcvbn4038 6y agoWhat’s really wonderful is when Github goes down I get a flurry of people broadcasting on Slack channels that they can’t work and all their builds are failing, and I’m wondering What they want me to do about it. Should I get out and push? Call Bill Gates? Tweet Trump? When Github doesn’t work that’s your excuse to go get coffee, grab a pint, browse Hacker News, etc. - people should be happy with GitHub goes down. (Or self-host gitea and work from that if your so determined to keep on)
- maypeacepreva1l 6y agoGithub is evolving pretty rapidly than I thought it would. I like their new changes. Deep underneath it being with Microsoft just reminds me of oracle/google debacle. Definitely discourages big projects to be hosted there.
- DrBazza 6y agotl;dr yes - and an exception to Betteridge's law!!! https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
- w0m 6y agotl;dr maybe - is that an exception?
- samcrawford 6y agoIndependent of whether it's been down more frequently since the acquisition, I find Github's status page reporting to be lacking. It's also quite generous with definitions of outages and downtime. For example, on June 22nd they had an issue whereby half of their nameservers were responding with an empty answer for queries for github.com. A very nice explanation is here: https://news.ycombinator.com/item?id=23605409 https://news.ycombinator.com/item?id=23605409. So for roughly half of users (including myself) this would have manifested itself as a complete outage. It also lasted for a good couple of hours. Yet on their status page it's listed as a 46 minute degradation only. So relying on their status page reporting to draw conclusions about availability (as Statusgator seems to) will mean that an overly optimistic picture of availability is presented.
- oknoorap 6y agomaybe this down caused by cloud migration from AWS to Azure?
- srean 6y agoTaking the numbers at their face value this is a good exercise in some text book statistical test of hypothesis. Incidents: Before 89. After 126. What is the chance of this happening if the 'rate' of occurrence has not changed ? Assuming an unknown but constant Poisson rate, we get the probability of observing what has been observed to be 0.00225. A fortuitous thing about this test is that one does not need to know what that unknown constant rate is.
- visarga 6y agoMaybe Github has grown significantly since then. More complexity, more down time.
- dutchmartin 6y agoWell, Github introduced way more new features since Microsoft took over.
- paulintrognon 6y agoAlso, according to the article itself, "[it] could be all a part of coordinated effort to be more transparent about their service status".
- jcrubino 6y agoMicrosoft has every right to move fast and break things on their watch.
- jtdev 6y agoMaybe Microsoft is less averse to complexity - which would still make uptime degradation land squarely on Microsoft’s shoulders. Microsoft seems to be an organization that actually introduces too much complexity to their products imo.
- solidasparagus 6y ago> Assuming a Poisson rate That feels like a mighty big assumption. Probably big enough that trying to calculate the probability is more misleading than enlightening.
- Yuioup 6y agoI have zero evidence but I would not be surprised if they are busy absorbing Azure Devops into GitHub and that's causing some hiccups.
- alt_f4 6y agoAnecdotally, yes. I almost never saw internal error pages before, now I've been seeing them every week for a few weeks. Some people need to be fired.
- jna_sh 6y agoA number of the staff who left GitHub in protest of the ICE contract were senior SRE. I wonder if the data says anything about the dates of their departures...
- samwhiteUK 6y agoThis is a thinly veiled advert, and tells you literally nothing concrete, apart from Github's status page is updated more. The article even admits this!
- tehjoker 6y agoLooking at the chart, there were more warns but fewer downtimes so I'd say no. There was one giant downtime that happened, but I don't know why it happened so it's hard to say if there was a cause relevant to the acquisition. The other downtimes that happened seemed to be slightly longer in duration than before.
- darkwater 6y agoAnyone from GH willing to share - anonymously maybe - some insights? I really find the lack of post-mortems from GitHub outages a bit... weird given GH audience. I think all we dev could learn from a properly written public GH post-mortem.
- xtracto 6y agoIt´s Microsoft, when has Microsoft shared a downtime postmortem?
- z9e 6y agoHas anyone noticed their search has gotten worse? I sometimes will search for some code that I know exists in a repo, but it gives no results, and when searching a little later it shows them as I expect. We have Github enterprise, and Slack notifications when Github has any issues. Nearly every week there’s a problem, sometimes it’s resolved in a few minutes, other times it goes for an hour or so. I’ve pondered the question if there has been more outages since the MS acquisition and in my experience that’s a hard yes.
- oltdaniel 6y agoPersonally I think it is connected to adding more features to GitHub. Microsoft is rolling out more and more small but also big changes, and as everyone knows, this always can have a few minutes downtime for each update.
- bizzleDawg 6y agoThe charts and diagrams in that post are really nice, how did you make them?
- pbnjay 6y agoHow has traffic/attacks changed since acquisition? Microsoft name surely gives them an even larger target on their back.
- oliwarner 6y agoGitHub is a lot more enticing now it has free private repos and the launch of Actions. I suspect they're a lot busier than they were before Microsoft. More features also naturally means more status tickets.
- deleted 6y ago[deleted]
- samth 6y agoThis analysis has a significant comparability problem because of added features. The core git-repository service is almost never down, but GitHub Actions has had a lot of reliability problems (you can see this clearly on the status page). But GitHub Actions is new, so it skews the recent availability problems up. Unfortunately, as the post notes, the detailed status information is _also_ new, so it's not really possible to analyze the data accurately.
- julius_set 6y agoI mean makes sense. They are probably migrating their infrastructure over to Microsoft’s and such. Disclaimer: I don’t work at MS so no clue, but have been part of acquisitions
- jtolds 6y agoI'm sure MS is super uncomfortable with GitHub's AWS bill. I've just assumed they're fast-tracking a migration to Azure
- dang 6y agoThis was prompted by the latest "Github is down" (now transformed to "Github was down" by the arrow of time): https://news.ycombinator.com/item?id=23675864 https://news.ycombinator.com/item?id=23675864