8 ms·
Incident with multple GitHub services
- deleted 6mo ago[deleted]
- supakeen 6mo agoI mean; this is the normal mode of operation for GitHub at this point.
- argee 6mo agoI moved to Gitlab a while ago. It's a whole new level of freedom not having to pay for self-hosted CI runners.
- cjonas 6mo agoIt would be wild if they dropped below the "two 9's" metric. I think they would need an additional ~16hr of outage in the 90 day rolling period.
- waiwai933 6mo agohttps://mrshu.github.io/github-statuses/ https://mrshu.github.io/github-statuses/ suggests that their combined uptime doesn't even meet 1 nine, let alone 2.
- cjonas 6mo agoya i was just doing the math on their chart for the git operations. I added up 14.93 hours combined hours, which puts them WAY lower than the reported 99.7 metric they show right next to it. So based on their own reporting, the uptime number should be 99.31. Which means only like 6 additional hours and they'd fall below 99.0%
- cjonas 6mo agoalso I never had considered that breaking your up-time into a bunch of different components is just a strategy to make your SRE look better than it actually is. The combined up-time tells the real story (88%!). Thanks for the link
- femiagbabiaka 6mo agoThe number of nines assigned to a suite of services is not indicative of the quality of SRE at any given company, but rather a reflection of the tradeoffs a business has decided to make. Guaranteed there's a dashboard somewhere at Github looking at platform stickiness vs. reliability and deciding how hard to let teams push on various initiatives.
- cjonas 6mo agothis is fair. I should have just said "Site Reliability", as it's almost certainly out of the engineers control.
- CamouflagedKiwi 6mo agoThe intersection of uptime across every possible service they offer isn't a particularly great metric. I get the point that they are doing badly, but it makes it look worse than I think it really is. What I would like to see is a combined uptime for "code services", basically Git+Webhooks+API+Issues+PRs, which corresponds to a set of user workflows that really should be their bread & butter, without highlighting things you might not care about (Codespaces, Copilot).
- 6mo ago
- roblh 6mo agoGitHub is going for “eight 8’s” at this rate.
- fishgoesblub 6mo agoAt this point it'll be better to have alerts for when GitHub is online, rather than offline.
- josefritzishere 6mo agoMicroslop is destroying Github
- AnkerSkallebank 6mo agoSome of my jobs are completing, some are failing. Seems to be random. Kind of wish they would just fail outright, instead of running for 10 minutes and then failing.
- ossa-ma 6mo agoSeems like outages are increasingly more frequent nowadays. Obviously, this is not the best state of affairs, and developers should not be limited by services. In the meantime I've been experimenting with building third spaces for people to chill while they wait for the services they are dependent on to go back up. The first one I've built is a little ASCII hangout for Claude @ https://clawdpenguin.com https://clawdpenguin.com but threads like this make me want to build it for Github too.
- 0xbadcafebee 6mo agoI am this > < close to just running Gogs or Forgejo on some Hetzner boxes, quit my job, charge people for access. Why aren't there like 10 startups doing this yet? Please? I want to give you my money. Just give me a git host that doesn't suck. (All the current ones suck)
- pocksuppet 6mo agoCodeberg and Sourcehut are doing it for free, for open source. Corporate probably won't ever move off Github, because they need the prestige of using Github - the actual service quality is completely irrelevant. This is an aspect of the enshittocene epoch - I repeat, quality is irrelevant to corporates.
- apparatur 6mo agoCorporate will move off GitHub as soon as it loses prestige and they will move on to the next thing.
- 0xbadcafebee 6mo agoSourcehut isn't free and has weird UX, Codeberg is free but has poor performance and weirdly over-moderates discussions. I know corporate will always suck, I'm just talking about having something that approximates the "old GitHub" for personal/professional use
- gsliepen 5mo agoThe SourceHut UI looks weird compared to commercial offerings, but every time I use it I am pleasantly surprised how fast it is and how little clutter there is.
- apparatur 6mo agoGitea has a paid plan. On Forgejo forums you can find 3rd party offers of paid hosting as well.
- 6mo ago
- linhns 6mo agoBusiness as usual.
- ChrisArchitect 6mo agoMultple 9s
- embedding-shape 6mo agoI've been on a somewhat binge to move a bunch of stuff to self-hosting at home. Yesterday I finally completed my self-hosted Forgejo instance at home, together with Linux, Windows (via VM) and macOS (via Mac Mini) runners/workers for CI/CD, so everything finally lives in-house (literally), instead of all source code + Actions being on GitHub but the infrastructure actually living locally. This is probably the first time I felt vindicated with my self-hosting move literally the day after I finished the migration, very pleasant feeling. Usually it takes a month or two before I get here.
- shevy-java 6mo agoInteresting. I speculated not long ago that Microsoft is really taking a dive here, and other companies may look to provide better alternatives to GitHub, as one idea. Today I read your comment about self-hosting here; while that is not quite what I compared or had in mind, it is interesting to read about it, of people who go that route. Microsoft is really putting themselves into trouble in the last year or two. Some things no longer work, so much is clear here.
- yakattak 6mo agoI moved my forge to my home, outside of a little stress getting all the containers wrangled it was pretty effortless to setup Forgejo. I do need a good backup solution though, that’s one thing I’m missing.
- TranquilMarmot 6mo agoI use https://github.com/garethgeorge/backrest https://github.com/garethgeorge/backrest to manage nightly encrypted backups of my Forgejo instance to a Hetzner Storage Box. <$4/mo for 500GB of storage. It's also where I back up my Immich library to. Immich automatically dumps its DB every day, for Forgejo I have a little script that runs as part of the Backrest backup that does a pgdumb of the database before doing the backup. It works great, I even had to do disaster recovery on it once and it went smooth.
- lone-cloud 6mo ago
- dankobgd 6mo agoDon't worry, status page says that it's 100% working - green color, all good. even though i can't access a static page
- surya2006 6mo agowhat are the good alternatives available for github i find some alternative but as long as widely people use github i cant use other service right like i cant share my alternative to other developer and force him to use this for me. so i feel like i locked in even i want to move i can't
- embedding-shape 6mo agoHuh? Why not? Say "My git repository is here $URL" then if they want to visit and/or clone it, they'll do that, otherwise don't, why does it matter? Sure, if you're out after reaching the most people, gaining stars or otherwise try to attract "popularity" rather than just sharing and collaborate on code, then I'd understand what you mean. But then I'd begin with questioning your motivation first, that'll be a deeper issue than what SCM platform you use.
- mghackerlady 6mo agogitlab is about as close as you'll get
- pxc 6mo agoGitLab annoys me in tons of ways, but I feel it's generally better than GitHub in lots of ways.
- shimman 6mo agocodeberg.org is a thing, and it's perfectly suited for open source projects. Many neovim plugins and home lab tech I use are hosted on codeberg with no issues. If you just want to github as social media, you will never be happy.
- nosioptar 6mo agoI'm probably going to use source hut in the future. It allows contributions via email without an account requirement. https://sourcehut.org/ https://sourcehut.org/
- nerdypepper 6mo agogive tangled.org a go perhaps. its got the self-hostability that cgit/forgejo does and a the social bits that github does.
- shevy-java 6mo agoMicrosoft again. I think it is time that Microsoft lets go of GitHub. They are handling it too poorly.
- surya2006 6mo agoeven vercel also have more downtime nowadays
- buildbot 6mo agoAnyone also seeing Active Directory/Entra issues?
- throwatdem12311 6mo agoSeems like they just can’t deal with the absolute deluge of AI vomit being uploaded every day. Good riddance I hope it completely destroys them.
- mijoharas 6mo agoAre you taking about what they write to run the service? Because looking at the uptime, and considering it's Microslop, I wouldn't be surprised.
- throwatdem12311 6mo agoWhat they write and the extra demand from vibe coders.
- LorenDB 6mo agohttps://mrshu.github.io/github-statuses/ https://mrshu.github.io/github-statuses/ says they are down to 88.15% uptime. Even when you consider uptime of individual components, their best is 99.78%, so two nines.
- Ygg2 6mo agoI see Microsoft mandated AI is doing wonders. For self hosters and Linux enthusiasts.
- jpleger 6mo agoI wonder if there is any correlation between them moving towards Azure. https://thenewstack.io/github-will-prioritize-migrating-to-azure-over-feature-development/ https://thenewstack.io/github-will-prioritize-migrating-to-a...
- frakkingcylons 6mo agoThey are dealing with vastly more activity as a result of AI usage. It's that simple.
- jeffrallen 5mo agoThey have not (re-)engineered their system, even via temporary hacks, in the last 12 months to defend itself with soft failures, circuit breakers, hard quotas, etc. It's that simple.
- sharts 5mo agoThey’re pushing out AI slop as production services
- eddyg 5mo agoThe scale of growth they’re dealing with is insane. “There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.) GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week in 2025, and now 2.1B minutes so far this week.” Source: GitHub COO on April 3, 2026. https://x.com/kdaigle/status/2040164759836778878 https://x.com/kdaigle/status/2040164759836778878
- tomjen3 6mo agoAt this point it should almost be news when it works.
- jasoncartwright 6mo agoJust cancelled my GitHub Copilot Pro+ year subscription. Removal of Opus 4.6 stung, but the repeated continued downtime makes it unusable for me. Very disappointed. No fuss instant refund of my unused subscription (£160) appreciated.
- miltonlaxer 6mo agoWhat will you use now?
- jasoncartwright 6mo agoClaude Code
- tgrowazay 6mo agoDoesn’t GitHub Copilot Pro+ only have month-to-month payment option? Only Pro (without plus) can be paid annually for some reason.
- jasoncartwright 6mo agoI paid 390 USD for a year Pro+ subscription in November 2025. I used all the 'Premium Requests' every month on (mainly) Opus 4.5 & 4.6. From what I've read on here it seems I was probably a rather unprofitable customer - it felt like a steal.
- djeastm 6mo agoYes, it was definitely a good value for devs using those models. I was hoping since Github Copilot was rarely talked about compared to the Anthropic/OpenAI offerings, MS would continue to subsidize it to encourage people to move over, but maybe it just got too expensive.
- pkaye 6mo agoPro+ does have a annual plan but recently they paused or dropped the annual plans because they are trying to adjust the pricing model.
- bakies 6mo agoI definitely have better uptime hosting my own gitea instance. It's faster too. It's basically a knock off GitHub. Plus with privacy concerns, I'm just happier overall. Easy setup, all I did was deploy the helm chart.
- CamouflagedKiwi 6mo agoI wondered. We'd seen for most of today that Actions were slow to trigger, I had at least one that was just missed, it felt like something was definitely off but the status was green all day until this.
- nhhvhy 6mo agoIf the day ends in Y…
- jonnonz 6mo agoWell I suppose they are finding out if you lay off too many people the IP of how the system works goes out the door with them.
- poplarsol 6mo agoAzure webapp deploys are also trash right now. Microsoft needs to stop slathering h1b copilot slop and get basic things like Windows patches working.
- frereubu 6mo agoWe have pretty basic needs - git repos + actions - and a bit of downtime here and there doesn't really affect us too much because we're not constantly committing and deploying, but even we're looking around for alternatives now. Also, looks like people might be pummelling the SourceHut servers looking for an alternative: https://sr.ht/ https://sr.ht/ is down. (Edit: was down when I wrote that, back up now).
- nerdypepper 6mo agotangled.org maybe?
- delusional 6mo agoOnce that 10x developer velocity from AI kicks in, I'm sure github stability improves. Did you know AI finally makes it economical to fix all the little bugs?
- Groxx 6mo agoSo... three incidents today, all of them ~1h or longer, and everything's green for the day with "no recorded downtime". These don't really look any different than past incidents which have red bars on their respective days, except maybe that those tended to be several hours. What do the green bars even mean? Are they changed to non-green retroactively if people complain enough or something? So far as I can tell, literally none of the previous green days have any incident shown in the mouse-over, but there are multiple for today only, so I kinda have to assume the mouse-overs are conveniently "forgotten" or all incidents become non-green and they just don't bother informing anyone on the same day. Either seems intentionally misleading.
- agartner 6mo agoThere was another really bad incident today: https://www.githubstatus.com/incidents/zsg1lk7w13cf https://www.githubstatus.com/incidents/zsg1lk7w13cf > We have resolved a regression present when using merge queue with either squash merges or rebases. If you use merge queue in this configuration, some pull requests may have been merged incorrectly between 2026-04-23 16:05-20:43 UTC. We had ~8 commits get entirely reverted on our default branch during this time. I've never seen a github incident quite this bad.
- robertwt7 6mo agoyeah this is crazy we had many PRs reverted as well on many repos. downtime is 1 thing, but reverting PRs is failure on another level
- CoastalCoder 5mo agoReally makes one appreciate that concept of A.C.I.D. database transactions.
- nulltrace 6mo agoDowntime is one thing. Silently reverting commits on your default branch is something else entirely.
- darknavi 6mo agoSimilar here. Somewhat ironic that a tool that was supposed to be preventing merge conflicts was authoring completely mangled commits to our mainline branch.
- lucasqueiroz 6mo agoWe've also seen quite a few commits disappear from main, the status of the PRs continued on merged. Was stressful.
- olingern 5mo agoSame, we got an email with a PDF with the affected commits and how to fix. What an absolute clown show.
- deferredgrant 6mo ago[flagged]
- hedayet 6mo agoIs Github losing any significant business from all these outages? Curious because for a long time we as an industry maintained that reliability and brand value are business critical; but seems like they are cared very little now a days. Happy to be corrected about my perception too.
- braiamp 6mo agoThey are entrenched enough that it's wrote off as cost of doing business. Big business have their internal instances so they are "insulated", everyone else isn't as critical and have the resources to do an internal solution or move.
- ozarkerD 6mo agoOrg instances have been affected by these outages too. Including the one today
- datadrivenangel 6mo agoThey have on-prem / dedicated instances? I thought that microsoft only offered that through their Azure DevOps git offering.
- jamesfinlayson 6mo agoGitHub Enterprise has existed for a while: https://docs.github.com/en/enterprise-server@3.20/admin/overview/about-github-enterprise-server https://docs.github.com/en/enterprise-server@3.20/admin/over... I'm pretty sure it still does - I used it at a previous job and at somewhere that I interviewed recently they said they used GitHub (given their size and being a somewhat regulated industry I can't imagine they rely on github.com).
- bandrami 6mo agoAnd as recently as two or three years ago it was universally agreed that the only way to reliably and securely deliver software was via repeatable builds with an attested chain of custody and auditable bill of materials and everybody just gave up on that completely when the LLMs got somewhat better.
- chrisweekly 6mo agomods: title tupo "multple"
- bmd1905 6mo ago[dead]
- x0ruman 6mo agoWe lost about a day of git history across several repos on Bitbucket a while back-not an outage, a data issue on their side. Local clones saved most of it, but issues and PRs from that window were just gone. That’s roughly why I started building gitbacker as a side project. Turns out the ‘back up the repo’ part is easy; the metadata is where it gets interesting.
- BhavdeepSethi 6mo agoI'd really like to read the Post Mortem doc for this. I'm still shocked how they allowed this to happen in production.
- eshack94 6mo agoAt this point, I feel like there should be a HN post whenever there ISN'T an issue with some GitHub service. Otherwise, it's business as usual...
- redwood 5mo agoGuessing the sheer volume of pull requests related to AI code jitter is leading to instability of this Microsoft product.
- alansaber 5mo agoHow much of this github downtime is due to git actions?
- rvz 5mo agoIf any company had so many incidents like GitHub, they would be out of business by now. Another outage at GitHub thanks to the AI agents like (Tay.ai, GitHub Copilot) at the helm with no employees maintaining it as I said before [0] "Centralizing" everything to GitHub in 2026 is starting to look quite unthinkable. [0] https://news.ycombinator.com/item?id=47592294 https://news.ycombinator.com/item?id=47592294