6 ms·
GitHub was down
- jaekash 6y agoagain ... well done microsoft. well done. I really don't get why people keep using it, gitlab seems objectively better to me.
- ishansharma 6y agoOut of curiosity, what could have Microsoft done to cause this? I'm not aware of the acquisition details. But did they make GitHub switch to Azure or make significant changes to infrastructure? I've seen several people blaming Microsoft for GitHub outages and am trying to understand why this is a common thing.
- Illniyar 6y agoWhen microsoft took over the rate of new features increased dramatically. Adding new features fast often entails tradeoffs in other areas - such as reliability. Which seems to be what happened in github.
- iddqd 6y agoI haven’t used their new features but am heavily affected every time they go down. I wonder if they are making the right trade offs here.
- varbhat 6y agoYes. Github used to be hosted on AWS. Now,they have migrated (at least partially) to Azure.
- goldfishlover 6y agothere isn't an azure outage right now.
- jaekash 6y agoI'm not privy to their management decisions, but I have used windows and azure, and I use office 365 every day, and this seems about right for Microsoft Quality.
- smcleod 6y agoYeah you’re bang on there, office365 has to be easily the most unreliable SaaS product I’ve ever used.
- rhacker 6y agoWell, let's not forget the Gitlab outage where they basically lost a shit-ton of production data. I mean if we're just comparing apples. https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/blog/2017/02/10/postmortem-of-datab...
- rfw300 6y agoWow, what a read. I’m sure GitLab has improved and grown a lot since then, but that’s like a comprehensive list of what not to do from an IT perspective. Not ever checking to see if your backups are working, wiping production DBs without a second thought, this post has it all.
- k_ 6y agoI prefer gitlab too but github is where most projects are, and that plays a overwhelming huge role on deciding where devs (and so other projects too) will be.. :/ It could change, but would require a pretty huge fuck-up or a lot of time and many not-so-smal mistakes from github.
- jaekash 6y ago"small" fuckups like this one have outsized impacts due to the amount of projects that use github. And the amount of people who just refuse to even consider gitlab is staggering.
- tannhaeuser 6y agoHow come we first get the almighty git software for free, then the next thing happening is people flocking to "hosted" services that provide very little on top of git for the vast majority of projects (apart from the social networking effect of course, but that also means your project has tens or hundreds of issues and build status "failing"). Then people bitch about gh on HN in a consumerish "coke-vs-pepsi" manner. Have F/OSS devs no self-respect? If you're into F/OSS, why do you lead your peers into a github silo where clicks to the content you create with lots and lots of effort in your spare time only helps big data, and which is even blocking indie search engines?
- varbhat 6y agoBut, UI of Gitlab is very bloated. I feel that UI of Github is less bloater than Gitlab. Gitea is OK,but there is no one big instance(yes,I know about codeberg) . Sourcehut is made by one-man in his one-man shop,and it lacks many features that one needs.Also,it is Alpha software.
- jaekash 6y ago> But, UI of Gitlab is very bloated. I feel that UI of Github is less bloater than Gitlab. Bloated how? There is a lot of functionality in gitlab, and maybe there is a fair argument that the product itself is bloated, but the UI seems pretty appropriate for the functionality. Maybe I'm just used to it though.
- varbhat 6y agoGitlab is functional and has more features/functionalities than Github. But,UI wise , Github is very much slim ,fast and quick. It will function without js. Gitlab although functional will take very much time to load a page(because ,js is used a lot).Making it PWA would solve the problem but it is still not made PWA.
- tjomk 6y agoProbably both. I remember the very first versions of Gitlab when it was just announced on habr.com, and it used to be a minimalistic (more in terms of UI) version of Github. As the functionality grew, things got a bit out of control. Currently UI looks more like it's been designed by people who write code rather than those who have spent considerable time thinking through user interactions and designing based on that. So it's not bloated in terms of the code and libraries, but rather all of the functionality is just thrown to the user as is without much thought.
- rplnt 6y agoI'm sure they have progressed a lot since, but last time we evaluated a self-hosted git "hub" solution at work, Gitlab couldn't handle our main repository (history starting in the 90s, but probably more imporantly loads of junk and a 10GB size on disk). We went with Stash (now Bitbucket) instead.
- kchoudhu 6y agoSecond time in what, a week? What is going on at Microsoft?
- jaekash 6y agoWhat is going on that people expect something better from Microsoft? Really this is quite on par with the quality they deliver. The only surprising thing is that people are surprised by this.
- nodesocket 6y agoTheir subpar uptime dates way back before Microsoft acquired them. Nothing to do with Microsoft. To be fair, I am sure they have unique scaling and infrastructure problems.
- ip_addr 6y agoI wasn't aware of that reputation, besides Windows ME, that is.
- thih9 6y agoLooks like they need some high level analysis on why these outages happen so often. At this point it feels like it’s no longer a series of accidents and that they should improve something.
- haik90 6y agoafter this downtime today. We're finally start discussing (again) to use our own Gitlab.
- sschueller 6y agoI just setup auto deployment of my Android app to the app store view gitlab-ci, very cool. At this point im not sure if I should spend the time to replicate this in github actions on the copy of the repo.
- diarrhea 6y agoAt work, we have an employer-provided (some groups also have their own server/setup) GitLab instance. I work on there much more than on GitHub, and am really liking it. Everything seems neatly integrated, especially the CI/CD is nice. For all I care, GitLab can remain as it is right now. My primary fear is for it to become too bloated, trying to be all things to all people (project management, CI/CD pipeline, bugtracker, Wiki, Git repository, ...). I would be happy to find out about people's experiences rolling their own GitLab, like what pitfalls to look out for. We are on GitLab CE 13.1.3. The biggest bug plaguing us has been CI/CD failures [0] for no apparent reason; but this is solved by a retry: max: 1 when: runner_system_failure block in the CI YAML, and seems to have been fixed for new releases. [0]: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450 https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450
- aaomidi 6y agoI love when people suggest this and when someone asks okay who do we put in charge of having better availability than github no one has a good answer.
- 6y ago
- rvz 6y ago14 Days ago, they went down [0]. And today it's happening again. Twice in less than a month. Another reminder to self host via solutions like GitLab or Gitea. [1] [0] https://news.ycombinator.com/item?id=23675864 https://news.ycombinator.com/item?id=23675864 [1] https://news.ycombinator.com/item?id=23676072 https://news.ycombinator.com/item?id=23676072
- dtech 6y agoself-hosting requires quite a bit of scale to be more reliable, otherwise you'll most likely still have possibly longer outages, just at different times.
- flukus 6y agoVery often that's ok if you can choose when the outages occur. With most you can because outages tend to be for regular maintenance and upgrades far more often than hardware failures. A small geographically concentrated company only needs 9 fives. Even for bigger companies that need more uptime they can at least give a shout out to everyone that it will be unavailable from X to Y, it doesn't just go down randomly with no notice whenever github decides to push a release.
- gregoriol 6y agoIf you think self-hosting doesn't randomly go down, you have never done self-hosting. In the last few months we had a server that physically crashed and had to be replaced, we had a bad kernel that paniced randomly and a gitlab upgrade that failed in the middle. Is you team equiped and ready to handle these? If not, don't self-host anything critical to your company, or your costs, downtimes and losses will be much much higher.
- dijit 6y agothis always comes up, and it's one of those things that's obviously true on the surface until you understand context. I have a self-hosted gitlab instance and it has much higher uptime than github in the last year. At some point a weekly downtime is going to trump any potential random outages. The major difference might be your ability to affect those outages (positive or negative) because humans are the primary cause of outages, and you can restore service to a gimped passable state if its your own thing.
- Illniyar 6y agoSeems like things started to go down the drain somewhere around February 2020. https://www.githubstatus.com/uptime/kr09ddfgbfsf?page=2 https://www.githubstatus.com/uptime/kr09ddfgbfsf?page=2 Wonder what was the trigger for the reliability hit - actions went GA on nov 2019, so it's something else (or possibly a combination of things)
- tjomk 6y agoMaybe they're migrating their infra to Azure?
- kenhwang 6y agoCOVID related work from home adjustments is my guess.
- kchoudhu 6y agoThat is actually an extremely good guess, and I am ashamed I didn't think of it to begin with.
- hnlmorg 6y agoThat might be the trigger but it wouldn't be the root cause. For example web and backend servers for cloud services shouldn't be affected by whether people are sat in one location or another. However if those systems requires lots of maintenance to keep running and people are less available due to COVID then you'd see a rise in downtime. But that would mean it's not COVID that's the problem, it's the amount of maintenance required and COVID only surfaced that problem.
- kenhwang 6y agoDoesn't have to be on service side either. Could be that user patterns are changing too, testing scenarios that aren't well worn on GitHub.
- deleted 6y ago[deleted]
- 6y ago
- zhdc1 6y agoI received a 500 error when I went to GitHub, and the first thought on my mind was to check Hacker News. I wasn't disappointed. In all fairness, I GitHub has more or less been fairly reliable, minus whatever has been going on over the last week.
- politicdone 6y agoSorry crashed the server. Just pushed a big commit
- tmsh 6y agodeleted
- rimliu 6y agoYou are bad at trolling.
- gitgud 6y agoCan't wait for the article about this outage, what will it be? - Auto-scaling issue - DDOS - DNS error - Datacenter outage Any other possible problems?
- aspectmin 6y agoIs there historical data of Github Uptime/Downtime? CSV format or other? I'd love to do some analysis on how things were pre, vs post the acquisition (and trends in availability)
- dtech 6y agoThey have some self-reported data [1]. You could scrape that and transform it. Eyeball analysis suggests it started in december 2019-february 2020, and rapidly went downhill starting april. [1] https://www.githubstatus.com/uptime?page=2 https://www.githubstatus.com/uptime?page=2
- Symbiote 6y agoThree months as HTML: https://www.githubstatus.com/history https://www.githubstatus.com/history
- Whiteshadow12 6y agohttps://nimbleindustries.io/2020/06/04/has-github-been-down-more-since-its-acquisition-by-microsoft/ https://nimbleindustries.io/2020/06/04/has-github-been-down-...
- svntid 6y agoyet again - I have not experienced anywhere close the amount of outage before Github was swallowed by Micro$oft
- onyb 6y agoThat moment when you submit a very long comment on GitHub, and realise that it is down. :(
- mullikine 6y agoI have noticed a pattern than when I generate markdown from org-mode and have the text 'language' selected for highlighting push, this causes github to hang like crazy. I don't think I'm crazy in thinking it might be me. I push frequently to my blog and am starting to notice a correlation. I export this into the below markdown. #+BEGIN_SRC text -n :f "translate-shell -s fr -t en" :async :results verbatim code I learned some French so that I can talk to you during tennis. I hope I know enough so you will not get bored. #+END_SRC When I get a page build failure it's usually my fault for creating . This is the markdown which was pushed to my blog. The 'Page Build failure' messages take a long time to arrive to my inbox and I can see that the page build is hanging. {{< highlight text "linenos=table, linenostart=1" >}} I learned some French so that I can talk to you during tennis. I hope I know enough so you will not get bored. {{< /highlight >}}
- noble_pleb 6y agoSeems to me like a Deja-Vu after I answered this[1] on quora yesterday. [1] https://www.quora.com/Github-now-allows-unlimited-private-repos-with-unlimited-collaborators-Why-should-one-stay-with-Bitbucket-now/answer/Prahlad-Yeri https://www.quora.com/Github-now-allows-unlimited-private-re...
- amirathi 6y agoPost mortem for February events[1]. It was resource contention on their primary mysql cluster caused by unintended config change. There's a total of 34 incidents in 2020 so far[2]. I wonder if all are DB woes or there are other factors at play (like move to Azure). [1] https://github.blog/2020-03-26-february-service-disruptions-post-incident-analysis/ https://github.blog/2020-03-26-february-service-disruptions-... [2] https://www.githubstatus.com/history https://www.githubstatus.com/history
- mundanevoice 6y agoMove off Rails now GitHub, is clearly not working for you. It took you this far, now go move to something that doesn't sh*ts the bed twice every month.
- noble_pleb 6y agoFlask (python) or even CodeIgniter (php) are my frameworks of choice for this sort of thing. They may be a bit old and organizing a large project could be difficult but nothing can beat them on performance!
- mundanevoice 6y agoIndeed. Anything that has so much magic in it is bound to bite you in the a. There will be a time when there is so much code and magic that no one really understands the system well. Twitter too was on Rails in the initial days which helped it took off, and then realised they need to move to JVM(Scala) to scaled and they have been performing brilliantly since then. Now, that MS has bought them, why don't they use the brilliant tech MS has (.NET, .NET Core) and be more reliable?
- brobinson 6y agoRails is (apparently still) a popular target for haters, but for projects at Github's scale it's rarely a code logic/framework-level blunder that's taking the service down. It's generally a cascade of failures in things like multiple database systems, auto-scaling, dns/caching, etc.
- mundanevoice 6y agoThere are projects bigger than GitHub that do all of these things and still doesn't fail as much as GitHub. Honestly, both of us speculating. You can't say for sure if it's not Rails and I can't say if it's not anything else. > for projects at Github's scale it's rarely a code logic/framework-level blunder that's taking the service down. It's generally a cascade of failures in things like multiple database systems, auto-scaling, dns/caching, etc. This is just guessing. You can't honestly tell what is causing the issue.
- niffydroid 6y agoGithub is probably still more reliable than Bitbucket, that has weekly disruptions (small but you do get a performance impact)
- quyleanh 6y agoI still don't understand people who always mentions to Microsoft's acquisition. Until the official statement, it isn't Microsoft failure. Don't blame them.
- MattGaiser 6y agoIt’s a pattern. GitHub is now down a lot.
- hnarn 6y agoI have to agree that this feels like an abnormal amount of downtime the past few months -- it would be interesting to see some actual data over a longer term.
- dijit 6y agohttps://nimbleindustries.io/2020/06/04/has-github-been-down-more-since-its-acquisition-by-microsoft/ https://nimbleindustries.io/2020/06/04/has-github-been-down-... 97% increase in minutes of downtime since Microsoft. Could be better reporting now though.
- jaekash 6y agoYou know if you use Microsoft products every day, and every day they let you down, and every day you experience the worst most unintuitive design in the world, and every day you have to deal with their reliability issues, and then MS acquires github, and github starts to behave like everything else MS touches ... Clearly something about how MS runs is responsible for their past outcomes, why is it a stretch to assume it is responsible for another similar outcome? It is like saying we don't know the rotation of the earth is why the sun rose this morning because we have not had an official investigation into the matter.
- quyleanh 6y agoIt's just your own feelings. I haven't had any issue with Microsoft products (and yes I use their services every day in my working life). All I can see from you is making a guess with your full of sentiments and negative thoughts. So as I said, don't blame them until official statement.
- ijelliti 6y agonow we are talking about GitHub availability report!! happy Monday
- mindfreeze 6y agoI am thinking when was the last time GitLab went down, I rarely or did not see downtime experience like this bad with Gitlab, I was seeing history, https://status.gitlab.com/pages/history/ https://status.gitlab.com/pages/history/ They do have some latency or slowness issues, but couldn't find like whole system down thing, Like in one of the comments here, reminded me of 2017 incident, https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/blog/2017/02/10/postmortem-of-datab... They should have improved a lot by now, but still I am curious, why such large or frequent downtimes are happening to GitHub. Is it due to making it more open for teams with Private repos, and more perks along with quarantine and WFH things
- letientai299 6y agoThe correct link for history (at of now) should be: https://status.gitlab.com/pages/history/5b36dc6502d06804c08349f7 https://status.gitlab.com/pages/history/5b36dc6502d06804c083... Above link result in err 500.
- humaid 6y agoThat GitLab downtime happened when we had deadlines, luckily git isn't a centralised platform so we merged our changes on a new GitHub repo we created. Also the GitLab sluggishness reminds me of their daemon which kills the server to control memory leaks[1], although this probably isn't the main cause of the platform's slowness. [1]: https://about.gitlab.com/blog/2015/06/05/how-gitlab-uses-unicorn-and-unicorn-worker-killer/ https://about.gitlab.com/blog/2015/06/05/how-gitlab-uses-uni...
- echelon 6y agoWell, there goes my night. I was waiting on a build triggered by Github actions and was wondering what was up. I guess this is my sign to get some sleep. Microsoft needs to slow things down and focus on stability. This really isn't good. I need these weekend and late night hours for my side hustle. I already have enough trouble as is, I don't need an injection of additional difficulty. (That's just my frustration; I can't imagine what y'all are all going through.) They're making some very frustrating choices lately. Their redesign broke READMEs with tables (which now require horizontal scrolling), and they don't seem to care about all the repos they impacted. Pull it together, Microsoft.
- jaekash 6y agoWhy are you not using an alternative?
- echelon 6y ago> Why are you not using an alternative? I don't need two CI/CD pipelines. Nor do I have have time to build something like that. I need to spend my time working on the core product. The risk here is that my deploy SLA is tied to Github. I accepted this risk as the cadence of my deploys tends to be once to twice a day on average and Github is usually available (many nines). I made a choice, and now that choice is biting me. Now I'll evaluate if I should spend the time and effort to migrate off it. If this is the last of it and Microsoft makes a commitment to not break things, I'll likely stay as the effort to move is nonzero. If this begins to happen every month, on the other hand...
- jaekash 6y ago> I don't need two CI/CD pipelines. Maybe my question was unclear, but I was not asking why are no not using two services, I was asking why are you not using an alternative, and I meant instead of GitHub, not in addition to GitHub. > If this is the last of it and Microsoft makes a commitment to not break things, I'll likely stay as the effort to move is nonzero. Commitment without liability is not a commitment. It is just empty words. And I don't see Microsoft making any commitments with liability on their part for a free service.
- NiceWayToDoIT 6y agoOn June 29th they also had outage (lasting 2 hrs), does anyone know what was the cause back then?
- dmpetrov 6y agoWhy this is usually happen on weekends? The only time that I have for coding :)
- anemic 6y agoProtip: Always have an extra customer, like the flowershop downstairs. Let her borrow your wifi in exchange for some office flowers. Now she is technically your customer. When your shit goes down and nothing works you can still write "some of our customers are experiencing issues" in the statuspage as the flowershop still has wifi (hopefully).