5 ms·
GitHub partial outage
- danfritz 1y agoRelated to the recent announcement they are moving to Azure?
- the_af 1y agoWow. It wasn't already running on Azure? What was it (or is it) running on?
- noir_lord 1y agoiirc (it's been a while) they where on rackspace when Microsoft bought them out - there was an article a few months ago saying they where moving to Azure and freezing new features while they do the move[1]. [1] https://thenewstack.io/github-will-prioritize-migrating-to-azure-over-feature-development/ https://thenewstack.io/github-will-prioritize-migrating-to-a... Honestly I don't know half the features they have added because the surface is huge at this point everyone seems to be using a (different) subset of them anyway. So a feature freeze isn't likely to have much impact on me. EDIT: went and checked - https://github.blog/news-insights/github-is-moving-to-rackspace/ https://github.blog/news-insights/github-is-moving-to-racksp... not sure if they moved again before the MS acquisition though.
- nixgeek 1y agoA team of us moved it off Rackspace in 2013, it’s been mostly in a set of GitHub operated colo since then. Used to be there was some workloads on AWS and a bit of DirectConnect. Now it’s some workloads on Azure. To the best of my knowledge there’s been no Rackspace in the picture since about 2013, the details behind that are fuzzy as it’s been 10+ years since I worked on infrastructure at GitHub.
- antn 1y agoyeah, we did not have anything in Rackspace for many years before the Microsoft acquisition. I remember having to migrate some tiny internal things off of Heroku, though!
- le_stoph 1y agoIn the Pragmatic Engineer podcast episode with the former CEO of Github, the latter mentioned that they had their own infra for everything. If I remember correctly, this was due to the fact that Github is quite old and at the time when Github Actions became a thing, cloud providers were not really offering the kind of infra that was necessary to support the feature.
- Kwpolska 1y agoGitHub is old, but GitHub Actions are not. Indeed, GitHub Actions launched two months after the Microsoft acquisition was announced [0], and it is a half-assed clone of Azure Pipelines. [0] https://en.wikipedia.org/wiki/Timeline_of_GitHub https://en.wikipedia.org/wiki/Timeline_of_GitHub
- le_stoph 1y agoI'll be damned, I feel like I've been using GA since forever! You're right though, just re-listened to the segment[0] and the ex-CEO mentions they were initially using AWS, then moved to their own servers because of the limitations of AWS at the time and their particular needs. Github Actions did however always run on Azure! [0] https://www.youtube.com/watch?v=2oq__5tDFZI&t=491s https://www.youtube.com/watch?v=2oq__5tDFZI&t=491s
- saghm 1y agoI can't read the entirety of this article[1] because it's paywalled, but it looks like they ran their own servers: > GitHub is currently hosted on the company’s own hardware, centrally located in Virginia I imagine this predates their acquisition from Microsoft. Honestly, given how often Github seems to be down compared to the level of dependency people have on it, this might be one of the few cases where I might have understood if Microsoft embraced and extended a bit harder. [1]: https://www.theverge.com/tech/796119/microsoft-github-azure-migration-move-notepad https://www.theverge.com/tech/796119/microsoft-github-azure-...
- loloquwowndueo 1y agoWell… https://www.reuters.com/technology/microsoft-azure-down-thousands-users-downdetector-shows-2025-10-29/ https://www.reuters.com/technology/microsoft-azure-down-thou...
- saghm 1y agoFair enough, my Azure experience is minimal enough that maybe I shouldn't make assumptions about whether this would improve things. That being said, I do think there's merit in the idea that if Microsoft is going to be able to solve this problem, they probably should try to solve it just once, and in a general way, rather than just for Github?
- balamatom 1y ago>Microsoft >solve it just once, and in a general way
- drcongo 1y agoOh no. I look forward to watching my browser redirect 40 times on every attempted page load.
- wpm 1y agoAnd the URL will be 400 characters long
- bob1029 1y agohttps://news.ycombinator.com/item?id=45517173 https://news.ycombinator.com/item?id=45517173
- stackskipton 1y agoDoubt it. I'm Ops person on Azure, while they just had terrible outage recently, they tend to be as stable as any other cloud provider and I haven't had many issues with Azure itself compared to whatever slop the devs are chucking into production.
- whoknowsidont 1y ago>they tend to be as stable as any other cloud provider Absolutely not.
- Fokamul 1y agoNot Sharepoint? What a bummer.
- arccy 1y agoYour weekly reminder to take a break
- hoherd 1y agoI still can't pull new branches even though the incident says it's resolved. I don't think my boss would be happy with me taking a break this long... but what else can I do when our business uses GH?
- arccy 1y agoWrite up a document on the business continuity risks of depending on Github.
- theletterf 1y agoI was getting crazy thinking that there was something wrong with my SSH keys all of a sudden. Thanks $DEITY it's just GitHub.
- no_wizard 1y agoSame. I reflex replaced mine thinking it needed to be. Glad its working now though
- rvz 1y agoLooking forward to the postmortem. Are they using AI agents this time to resolve the outage? Probably not. But this time, there is no CEO of GitHub to contact and good luck contacting Satya to solve the outage.
- gunalx 1y agoYep. Was using github for oauth on a petproject of mine. Got the unicorn, and was considering takingthe break, or just etting up something else. Seems to be running again for me now though.
- nkzd 1y agoI thought my SSH keys were revoked, whew.
- coffeebeqn 1y agoJust started to replace mine when I saw someone post a message about GitHub
- JLCarveth 1y agoThis sure does seem to happen a lot
- Scribesley 1y ago[dead]
- fishgoesblub 1y agoMust be a day ending in Y.
- numbsafari 1y agoAnyone using GitLab have any insight on how well their operations are running these days? We originally left GitLab for GitHub after being bit by a major outage that resulted in data loss. Our code was saved, but we lost everything else. But that was almost 10 years ago at this point.
- colesantiago 1y agoNo issues on GitLab. Haven't seen any outage from GitLab in like, ever.
- philipwhiuk 1y agohttps://status.gitlab.com/pages/history/5b36dc6502d06804c08349f7 https://status.gitlab.com/pages/history/5b36dc6502d06804c083...
- colesantiago 1y agoNever had any problems really. GitHub on the other hand has outages more frequently.
- tux3 1y agoThat has definitely not been my experience. I like Gitlab, but they've had regular incidents all along. If a git push failed I wouldn't question it, it's almost never my network. I'd just open Gitlab's Gitlab and find the current active issue. To Gitlab's credit their observability seems to be good, and they do a good job communicating and resolving incidents quickly. Some companies that shall not be named have status pages that always show green and might as well be a static picture. Some use words like "some customers may have experienced partial service degradation" to mean "complete downtime". Gitlab also has incidents, but they're a lot more trustworthy. You can just open the issue tracker and there's the full incident complete with diagnosis.
- colesantiago 1y agoHmmm. You must be doing GitLab wrong.
- contravariant 1y agoAh that was why. Oh well, I just needed to get the code to the server, so I didn't really need Github anyway.
- ecshafer 1y agoGithub is owned by Microsoft, so this is a pretty small time indie operation, you need to give them a break.
- cube00 1y agoNot replacing the CEO suggests they aren't focusing on it as much as they were.
- lysace 1y agoJust your casual $3.8T company. There were so many severe Github Actions outages (10+ ?) in the past year. Cause: Migration to the disaster zone also known as Azure, I assume. Most of them happened during (morning) CET working hours, as to not inconvenience the americans and/or make headlines. Money doesn't buy competency. It's a long-term culture thing. You can never let go on maintaining competency in your organization. It rots if you do. I guess Microsoft did let go.
- conception 1y ago“guess Microsoft did let go” - are we thinking of the same Microsoft here?
- lysace 1y agoI am thinking of the atrophying one. Not MikeRoweSoft.
- mook 1y agoI thought GitHub Actions (in particular; not the rest of GitHub) was always Azure, because it was initially a fork of Azure Pipelines? GitHub as a whole, including the previously non-Azure bits, does seem flakier than a few years ago though, for sure.
- lysace 1y agoYou seem to be correct. Not that much visible from the outside, but yes it seems like they always ran on Azure, from the 2018 launch. (Apologies for the disinfo, although I qualified it with the "I assume".)
- arnvald 1y agoI’m old enough to remember when GitHub was on main page due to a cool feature they added, now they just end up here when it stops working
- alentred 1y agoIf I remember it well, every once in a while a new cool feature was also breaking stuff, doubling the chances of getting to the top page here. But truth being told, GitHub was fixing those at light speed too and it was very interesting to follow their progress. Their delivery pipeline (per branch, deliver when ready, etc.) sounded very much innovative by then and I think inspired many people.
- wavemode 1y agoIt's possible that Microsoft buying GitHub was a large-scale psyop intended to reduce the productivity of the competition. Any time their startup competitors are making too much progress they can just push the "GitHub incident" button and slow everyone down.
- grepfru_it 1y agoWe used to obsessively care about 500s. Like I would make a change that caused a 0.1% spike in 500s and I would silently say I'm sorry to the folks who got the unicorn page. I'm not sure the new school cares nearly as much. But then again this is how companies change as they mature. I saw this with StubHub as well.. The people who care the most are the initial employees, employee #7291 usually dgaf
- 0x1ch 1y agoI fall into the new school gen z category, and I think you're right. We don't care. We don't care about the problems started before us, and we owe nothing to no one (but our employers, must increase value for shareholders of course). I simply want to survive. I'll kiss ass where I have to, but not to people I don't work on behalf of.
- ares623 1y agoHell yeah
- kataklasm 1y agoCan't say that's entirely true for me ('02). If my [ employer, supervisor, ... ] provides me with logical, traceable tasks with their context properly laid out, I can totally put a ton of effort into providing meticulous, well thought out solutions, that are as good as it gets under the provided constraints. It's the non-sensical (be it actually non-sensical or just not understood enough because of unprovided context) tasks that make me not care.
- Arch485 1y agoI'll throw in my $0.02, as a fellow zoomer. I care about the things that are mine (as in, my code, my decisions, etc. etc.). But if management fucks up and tells me to fix it, there is no amount of money that will make me care. Especially if I advised management _not_ to do that in the first place.
- dustfinger 1y agoWhy does the main page show all green when there is an ongoing incident? All green here -> https://www.githubstatus.com/ https://www.githubstatus.com/
- gkoberger 1y agoIt's marked as resolved for some reason
- deleted 1y ago[deleted]
- blibble 1y agobecause then some mid-level manager gets a telling off and/or has to pay the SLA out of their budget
- dustfinger 1y agoahh, you are right. I am blind.
- deleted 1y ago[deleted]
- zamalek 1y agoThis is normal for Microsoft. It's as though status is owned and controlled by either marketing or accounting, not engineering.
- FrostKiwi 1y agoVery Microsofty. I'm still furious with how they handled the global ban on updating CDN (Frontdoor) files until last week and not properly communicating it. https://www.reddit.com/r/AZURE/comments/1on0ung/azure_status_is_not_honest/ https://www.reddit.com/r/AZURE/comments/1on0ung/azure_status...
- pfyra 1y agoCoincidentally, Azure Devops was also missing the ssh keys earlier today, both in the web ui and for ssh login.
- spockz 1y agoWell, github is moving to Azure and they are consolidating systems. No surprise there.
- carlyai 1y agothought i was going crazy
- prymitive 1y agoSpeaking of GitHub issues if you go to Insight->Traffic in your repo you’ll most likely see this banner: “ Referring sites and popular content are temporarily unavailable or may not display accurately. We're actively working to resolve the issue.” It’s been like that for months now with no sign of anyone working on it. They just don’t care about user experience anymore. https://github.com/orgs/community/discussions/173494 https://github.com/orgs/community/discussions/173494
- embedding-shape 1y agoSpeaking of "temporarily unavailable but it's actually forever", I've been wanting to get into Fallout and Starfield modding, so been waiting for their official wiki to come out of maintenance mode. I think I first tried to access it when Starfield launched (September 2023), and still today it is "currently down for backend maintenance". https://wiki.bethesda.net/ https://wiki.bethesda.net/
- vaindil 1y agoHey, I'm the GitHub employee who's working on fixing that right now. The service powering those stats is _ancient_, and it fell over back in September. It's taken longer than I hoped to get a replacement working, but it should be fixed within the next couple of weeks, fingers crossed.
- whoknowsidont 1y agoAnother outage brought to you by Azure.
- unit149 1y ago[dead]
- _heimdall 1y agoI really do feel for those hubbers that are still working on this years into the Microsoft era. GitHub was an excellent product and from what I hear it was an excellent culture too. I can only assume the culture has eroded similarly to the product itself as Microsoft has finally begun integrating the org into the slower moving machine that is MS.
- marak830 1y agoHuh could this be why I can't login and pushing packages says my account is banned?