7 ms·

An update on GitHub availability

421 points by salkahfi 5mo ago

TuxPowered 5mo ago
The availability of GitHub is still at 0% - it can't be reached over IPv6.
mijoharas 5mo ago
> we started working on path to multi cloud. Is this microsoft stating that they aren't able to get acceptable reliability from Azure? (I mean, I think a lot of us have heard that, but it's interesting to hear it from microsoft themselves).
- derwiki 5mo ago
  It’s pretty damning. But as someone who has used Azure, I buy it.
  - everfrustrated 5mo ago
    Pretty damming that two Microsoft subsidiaries - GitHub and LinkedIn - either shelved their forced migration to Azure or are looking at non-Azure options.
- jasoncartwright 5mo ago
  Seems pretty sensible to not rely on a single provider for their large complex system?
  - mijoharas 5mo ago
    I mean, amazon (shopping, along with prime video e.t.c.) runs on AWS.
    - jasoncartwright 5mo ago
      Prime video uses a non-AWS CDN when I watch football on it here in the UK
      - farfatched 5mo ago
        The BBC were unable to find a single CDN that could serve the UK during its peak football matches. https://www.bbc.co.uk/webarchive/https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Finternet%2Fentries%2F8c6c2414-df7a-4ad7-bd2e-dbe481da3633 https://www.bbc.co.uk/webarchive/https%3A%2F%2Fwww.bbc.co.uk...
      - jamesfinlayson 5mo ago
        I'd believe it - CloudFront always felt a bit like AWS ticking a box ("we have a CDN") rather than being a good to use product.
      - grogenaut 5mo ago
        that's called load balancing and regional availability. many companies do multi-cdn. in fact it's smart to use multiple CDNs so you can do better in contract time. Twitch uses IVS but we have failover to other CDNs for very large events.
    - ksimukka 5mo ago
      When I was at AWS, retail was not yet running on AWS. Has that changed? Prime video does use some AWS services, but live and on-demand are two entirely different beasts.
      - mijoharas 5mo ago
        Really? I thought retail was. It's been almost a decade since I worked at prime video but I think everything was running on AWS. (Some things didn't use brazil etc, but I think all the servers etc. were on AWS)
        malfist 5mo ago
  - cyanydeez 5mo ago
    This isn't a mom and pop shop. They have locations all over the world: https://datacenters.microsoft.com/ https://datacenters.microsoft.com/ There's no intrinsic reason they should be vulnerable to themselves.
    - jasoncartwright 5mo ago
      That website (for me) uses Cloudflare via WPEngine, which also isn't Azure
    - farfatched 5mo ago
      +1. Multi-cloud is typically done for vendor independence. But Github don't have that rationale.
    - tacticus 5mo ago
      and many of those azure regions are a single DC or single cage in a single dc that you can walk past to find the bigger netflix pop.
  - embedding-shape 5mo ago
    Man, you should have been there 6 months ago when they decided to start tearing down GitHub's own data centers and move everything exclusively to Azure. Seems they themselves realized this after they started moving, but imagine if you could have helped them realize this before they even started :)
    - benterix 5mo ago
      > Seems they themselves realized this after they started moving I guess most people at Github knew exactly it makes no sense but they didn't really have a choice. Maybe some voiced their statement, got "we hear you" in response and were told to proceed anyway.
      - embedding-shape 5mo ago
        Yeah, I don't know how it went down, but I also know exactly how it went down: Microsoft Execs: Everyone needs to move to Azure! GitHub developers: But Azure is not gonna be able to handle our load, we literally have our own data centers! Microsoft Execs: Sure, but you're Microsoft now, please publish blog post about how in half a year you'll be 100% on Azure. Few months later... GitHub Developer: We've tried our best, users are leaving in droves and Azure can't keep up! Microsoft Execs: Ok fine, you can use something else too, but only if you mainly use Azure and continue publishing blog posts about how great Azure is.
        alper 5mo ago
        Azure is the MS Teams of clouds.
    - nextaccountic 5mo ago
      Made me think. Why not convert Github datacenters into Azure datacenters that have Github as their sole customer? Then it's up to Azure how they will manage this
      - hobofan 5mo ago
- cbg0 5mo ago
  I think this is more tailored towards enterprise clients that lose money when Github is down, that would probably help with retention.
  - bombcar 5mo ago
    You’d think they could have had the existing GitHub on whatever continue as is (maybe for paying customers) while all the AI new inrush goes to the Azure setup.
  - jofzar 5mo ago
    Yeah that's a top tier enterprise plan feature if I have ever seen ut
- jansan 5mo ago
  The entire concept of multi cloud is amusing if you think what cloud originally was supposed to be. They could call them meta clouds (might infringe trademarks), and with the current growth trajectory of AI generated code eventually multi-meta-clouds, renamed to beyond-clouds, and then multi-beyond-clounds. I see no limits.
- deleted 5mo ago
  [deleted]
- youwangd 5mo ago
  Show HN timing matters more than people think. Monday-Thursday, 9-11am Pacific, is when the front page has the most engaged readers. Weekend posts get less competition but also less engagement.
- zamalek 5mo ago
  There was somewhat recently a post here about how priorities, pressure, and management subverted Dave Cutler's vision for Azure (which was to have near zero human involvement) - my Google fu isn't strong enough to find it. Supposedly, someone running over or opening a serial to a rack/VM is now typical operational procedure.
  - ok_dad 5mo ago
    This one? https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion https://isolveproblems.substack.com/p/how-microsoft-vaporize...
    - zamalek 5mo ago
      That's the one!
      - consumer451 5mo ago
        Amazing read. Thanks to both of you for finding that. > I later researched this further and found that no one at Microsoft, not a single soul, could articulate why up to 173 agents were needed to manage an Azure node, what they all did, how they interacted with one another, what their feature set was, or even why they existed in the first place. This reads like a description of the SLS-based (aka Senate Launch System) Artemis program, which somehow ended up deciding that the insane Lunar Gateway should be a thing. Destin (SmarterEveryDay on YouTube) [0] called out the entire nutball scheme to NASA, at NASA. This includes the SLS/Orion/Lunar Gateway insanity, and calling out the number of unknown, but very large number, of on-orbit refuelings that Starship would need to get to the moon. In that video's comments, I believe there is someone who worked on the Orion-related system, who says ~"Yeah, we thought the delta-v was too low, we could have increased it, but no one was speaking with each other at a whole system level." The mission drift at large orgs, gov and corp, is a huge problem that might one day be solved? [0] https://www.youtube.com/watch?v=OoJsPvmFixU https://www.youtube.com/watch?v=OoJsPvmFixU
        5mo ago
- tedd4u 5mo ago
  > multi-cloud XXXXL size project. May not ever deliver. But if it fails, it will only do so after years grinding through people, resources, etc.
pluc 5mo ago
There are no words that Microsoft can use that would make me trust Microsoft.
baq 5mo ago
openai, anthropic, google and a plethora of chinese models all end up pushing code into github. you can discuss whether gpt 5.5 is better than opus 4.7, but for github it doesn't matter: they'll be receiving the code no matter which llm spits it out. amazing on one hand, quite scary on the other for github and all other forges if this continues and there is no reason why it wouldn't.
- graemep 5mo ago
  Simple solution: charge all users. Charge more for higher usage.
  - gattr 5mo ago
    And/or provide a baseline free tier, corresponding to how much a typical human user would at most push/clone etc. They have pre-LLM statistics on that.
jcattle 5mo ago
When there's a gold rush invest in checks notes jewellery makers?
huijzer 5mo ago
I’m pretty sure my Forgejo instance on a Raspberry Pi is outperforming GitHub reliability. It’s faster that’s for sure.
- huijzer 5mo ago
  Why the downvotes? I’m serious. On GitHub I’ve experienced many downtimes. My Forgejo hasn’t gone down yet apart from reboots by me.
darkwater 5mo ago
Glad that they released some data about new repo/issues/commits over the last years. It confirms what everyone else already believed from the outside: agents are putting a lot of extra, sudden pressure on GitHub. It's like a startup that is growing exponentially, with the difference that they already have a large user base to serve - and that keeps them in the bullseye - and probably a not-so-fast-moving organization when it comes down to changes. On the other side of the coin, they also have a lot of talent, infra and money a startup might not have yet.
- maccard 5mo ago
  What data is that? There's an unlabelled graph and a number at the current peak.
  - ncruces 5mo ago
    Some previous numbers: https://x.com/kdaigle/status/2040164759836778878 https://x.com/kdaigle/status/2040164759836778878
    - maccard 5mo ago
      This is the data that should be in the blog post. Thanks for sharing.
  - deleted 5mo ago
    [deleted]
  - darkwater 5mo ago
    IMO it transmits the magnitude of the impact pretty well.
frangonf 5mo ago
What are we doing? Stop subsidizing tokens now that we extracted enough training data from you and we have enough agentic junkies business to keep the flywheel going up and cut on the loss leaders. [0] [0] https://news.ycombinator.com/item?id=47923357 https://news.ycombinator.com/item?id=47923357
guidoiaquinti 5mo ago
> While we were already in progress of migrating out of our smaller custom data centers into public cloud, we started working on path to multi cloud. This longer-term measure is necessary to achieve the level of resilience, low latency, and flexibility that will be needed in the future. Wild
maccard 5mo ago
It's kind of hard to read this with a straight face. The unlabelled graph with big numbers on top, the priorities that don't match with what we're experiencing, and a list of things that they're doing without a real acknowledgement of the _dire_ uptime over the last 12 months....
- ramon156 5mo ago
  "We hear you" in ~300 words, basically.
- ncruces 5mo ago
  More numbers: https://x.com/kdaigle/status/2040164759836778878 https://x.com/kdaigle/status/2040164759836778878 What's the question here, you don't believe growth is currently exponential, or do you think it shouldn't be hard to scale, when 10x YoY is not enough?
  - OtherShrezzing 5mo ago
    As a business user, our costs have gone up while service has gone down dramatically. Meanwhile our marginal cost to GitHub has hardly changed. Where our costs to them have increased, they mostly charge us per cpu minute, so obviously aren’t making any kind of loss on our account. I’m sure they’re experiencing scaling issues across the platform, but it’s unacceptable for that to have a negative impact on us when we're sending them $250/dev/yr for (what is in all honesty) hosting a bunch of static text files.
    - rdevilla 5mo ago
      > we're sending them $250/dev/yr for (what is in all honesty) hosting a bunch of static text files. You know, you can just host your own code forge. Or you can just drop gitolite on a server. Or pull directly from each others' dev machines on a LAN. GitHub is not git.
      - OtherShrezzing 5mo ago
nraynaud 5mo ago
So I gather that nobody is working on a search that stays on the current branch?
fontain 5mo ago
Personally, I’m sympathetic. We know that GitHub did a huge amount of work over the last decade to make Git scale, which has benefited us all. These new scaling challenges are real challenges, 30x growth would be a nightmare for any system that was already pushing the limits of what was possible, I think we are being far too hard on GitHub, they deserve a little grace.
- someone_eu 5mo ago
  GitHub's scaling issues are caused by their own vendor-lock approach and monopoly. Yes, of course _their_ goal is to be even bigger and even more all-consuming, so _they_ have to deal with the scale. Why a user would be sympathetic to that? The user (and not a big tech monopoly) answer to scaling issues is almost always to stop scaling and start federating and interoperating.
- remus 5mo ago
  For all the negatives about github I agree. They offer a lot of free stuff, and LLMs seem likely to put massively increase their costs with no guarantee they'll be making money off it. I can't think of many (any?) large businesses which could scale up to meet so much new demand without some significant growing pains along the way.
icy 5mo ago
I'm biased (founder of tangled.org), but the future really should be federated forges. Host repositories on sovereign infra with global identity + federated "metadata" (issues, pulls, etc.). Global indices for this should be trivial to spin up so availability is never a concern (we're working towards this!).
- ramon156 5mo ago
  Love the idea, would replace the LLM generated content ony our site, though. I recently migrated to codeberg because I'm okay with self-hosting big runners, while using codeberg's available runners for smaller cron-based things (they even have lazy runners for this).
  - icy 5mo ago
    It’s… all hand written? We just sound “professional”.
- ArcHound 5mo ago
  But, there are? I can host a repo on GitHub, Codeberg and self host it too. Then I need to watch over main to keep it consistent between those. After that's established, I can do updates from wherever. Link'em in the README.
  - nibbleyou 5mo ago
    There's also a tool to automatically push it to multiple repos: https://github.com/prashantsengar/GitEcho https://github.com/prashantsengar/GitEcho Disclaimer: the author is a colleague of mine Though to be fair, what the parent meant by federated forges is different than this approach.
    - pabs3 5mo ago
      git itself can push to multiple URLs btw: https://stackoverflow.com/questions/849308/how-can-i-pull-push-from-multiple-remote-locations https://stackoverflow.com/questions/849308/how-can-i-pull-pu...
jftuga 5mo ago
Some interesting tid bits: * we had to resolve a variety of bottlenecks that appeared faster than expected from moving webhooks to a different backend (out of MySQL) * * redesigning user session cache to redoing authentication and authorization flows to substantially reduce database load. * we accelerated parts of migrating performance or scale sensitive code out of Ruby monolith into Go. I'd like to know what database backend they migrated to. I was also surprised to read that the migration from Ruby to a more performant language had not already been completed. I assume this is because it a large code base with many moving parts, etc.
- mohsen1 5mo ago
  Another interesting bit: they are hitting performance issues due to the rise of monorepos. GitHub and frankly Git were not designed for monorepos
  - ghthor 5mo ago
    Yet the Linux kernel is a monorepo
    - mohsen1 5mo ago
      Try google3
    - guipsp 5mo ago
      The Linux kernel is pretty small
deleted 5mo ago
[deleted]
rootnod3 5mo ago
> Our priorities are clear: availability first That's a delayed April fool's right?
- embedding-shape 5mo ago
  No, just a 6 month old memo that was first opened today, as they said literally the same 6 months ago.
embedding-shape 5mo ago
Hah, love that now they say "Our priorities are clear: availability first, then capacity, then new features" when 6 months ago, it was seemingly exactly the same except Azure supposedly was gonna save them: > GitHub Will Prioritize Migrating to Azure Over Feature Development - GitHub is working on migrating all of its infrastructure to Azure, even though this means it'll have to delay some feature development. > In a message to GitHub’s staff, CTO Vladimir Fedorov notes that GitHub is constrained on capacity in its Virginia data center. “It’s existential for us to keep up with the demands of AI and Copilot, which are changing how people use GitHub,” he writes. https://thenewstack.io/github-will-prioritize-migrating-to-azure-over-feature-development/ https://thenewstack.io/github-will-prioritize-migrating-to-a... So the currently delayed feature development is now gonna be further delayed, yet almost every week we see new features and changes, just the other day the single issues view was changed, as just one example. And it was "existential" 6 months ago yet they keep stumbling on the exact same issue today? Even if they're focused exclusively on reliability and uptime, we get the experience that we have today, kind of incredible how a company with the resources of Microsoft seemingly are unable to stop continuously shot themselves in the foot. It's kind of impressive actually. As icing on the cake, they've decided to buy up all popular developer services then migrate them all to the same platform, great idea too.
- ncruces 5mo ago
  > So the currently delayed feature development is now gonna be further delayed, yet almost every week we see new features and changes, just the other day the single issues view was changed, as just one example. They did that as a panic mode hack to mitigate performance: https://news.ycombinator.com/item?id=47912521 https://news.ycombinator.com/item?id=47912521
- madeofpalk 5mo ago
  This seems uncharitable. Priorities aren't exclusive, especially at scale across large engineering orgs like GitHub. It could be that these are the top level priorities, but teams or individuals who aren't able to contribute to these priorities will work on other things like new features.
deleted 5mo ago
[deleted]
cedws 5mo ago
I wonder if they’ll end the free lunch we’ve been having since the MS takeover. There’s been a deluge of spam and crapware projects due to the LLM wave which is visible in that graph. Can’t see them sustaining being a public dustbin for low value projects forever.
- sbarre 5mo ago
  I could see them expiring/archiving/deleting inactive projects after some time. I feel like this would have negative impacts (lots of interesting historical archives on Github) but maybe if a project hasn't been touched, or cloned, in some time, it just gets deleted with some notice.
  - deleted 5mo ago
    [deleted]
  - rmunn 5mo ago
    Thing is, projects that don't get touched for months and months are the least costly. Disk space is cheap; what's costly is compute time to process new commits, new/updated/closed issues, new/reviewed/merged PRs, and so on. Inactive projects just sit there taking up disk space but basically zero compute time. So it would make no sense at all for them to delete old, inactive projects. (Which doesn't mean they won't do it: they might have hidden costs I'm unaware of, or they might make stupid decisions. People do make stupid decisions sometimes).
    - ifwinterco 5mo ago
      Also creates a perverse incentive to automatically push random commits to make sure your repos stay “active” and don’t get deleted, creating more load
    - sbarre 5mo ago
      Yeah that's a fair point!
  - jamesfinlayson 5mo ago
s_ting765 5mo ago
> Vladimir Fedorov is GitHub's Chief Technology Officer .... He currently serves on the board of Codepath.org, an organization dedicated to reprogramming higher education to create the first AI-native generation of engineers, CTOs, and founders. I think I found the issue.
- dude250711 5mo ago
  Sounds like a bet that AGI is not achievable.
latexr 5mo ago
> The main driver is a rapid change in how software is being built. Since the second half of December 2025, agentic development workflows have accelerated sharply. GitHub instability has started way before that. I understand it’s too much to ask of a trillion-dollar corporation to consider the impact of their own actions, but perhaps they should’ve thought of that before forcing LLM development down everyone’s throats.
- mathgeek 5mo ago
  While they contributed, they were still following the market trend anyway. If they weren't letting folks use it directly, other companies would have (and are).
  - latexr 5mo ago
    > they were still following the market trend anyway. They started the trend with Copilot. > If they weren't letting folks use it directly There is a chasm of difference between “letting you use it” and “forcing it down your throat”. Microsoft is doing the latter, not the former. Copilot is annoyingly present by default at every step on GitHub.
    - jcgrillo 5mo ago
      They should have just called it Clippy and revived the animated avatar.
steve1977 5mo ago
I know that I'm simplifying (probably too much), but it seems like things were fine when GitHub was still a Ruby on Rails monolith and all the rigmarole with microservices etc. only made things worse.
- tankenmate 5mo ago
  This sounds more like a belief, based on little more than "correlation is causation", than analysis that controls for macro-trends backed by evidence.
  - embedding-shape 5mo ago
    GitHub been oscillating between long phases of "Never any new features but rock-solid and no downtime" and "New features every week but also unicorns (used to be the "service unavailable page") every week" for as long as I can remember. Seems they're on some interval switching between the two.
  - sgarland 5mo ago
    It is, but everyone is entitled to beliefs. Anecdotally, I feel the same way. Everywhere I’ve been, there has always been a legacy monolith that was stable as a rock, with dozens of new microservices scattered around it in an attempt to exit the monolith. The microservices have never once been stable. People fail to take the most basic things into consideration, like “you can’t have Consistency and Availability when everything is a network call.” I’m sure survivor bias is at play here, but when I look through the older code bases - especially the data model - it’s an entirely different world than the newer stuff, and it’s clear which of the two was written by people who understand systems.
    - steve1977 5mo ago
      I mean one of the selling points of microservices is, that the developer can only be concerned with their slice of the whole and does not need to understand systems. Maybe this is not so clever after all. Or, it would require an architect who has a very good understanding of the system. Which in reality seems to be rare.
sikozu 5mo ago
This latest incident was the nail in the coffin for me. I've been on GitHub since 2012 but I'm feeling the pull to migrate out to Gitea/Forgejo. Has anybody done this recently? How'd it go?
- deleted 5mo ago
  [deleted]
- embedding-shape 5mo ago
  When one of the incident they write about here happened, I wrote about my experience moving from GitHub to Forgejo which I happened to complete just the night before that happened: https://news.ycombinator.com/item?id=47878192 https://news.ycombinator.com/item?id=47878192 (lots of other people sharing their experience as replies too) I was thinking of maybe doing a proper write up about how to host your own Forgejo + Action runners on Linux, Windows and macOS, not sure if there is enough interest. What would people for sure want to know in a guide/explanation of this?
- sltr 5mo ago
  I moved over back when GitHub was planning to charge per minute to use my own runner. It was easy with Claude, the gh API, and forgejo web API. I even set up daily backups to my S3 clone of choice. The only repos I left on GitHub are forks and one with a bit of public engagement.
bananapub 5mo ago
anyone who's actually worked there, could you explain why they're finding scalability and reliability so hard? naively it seems like 'repo groups', ie clusters of repositories linked by being mutual forks, would be fairly isolated for the whole git storage layer, and everything else feels pretty easily parallelisable (issues, actions, etc, modulo taking locks now and then to submit results or whatever). and given that, surely you can incrementally deploy changes across those many shards to avoid most big outages? are there big conceptual serialisations that I've missed? is it just not well factored? was the move to Azure just a catastrophically bad idea? some other thing?
- dist-epoch 5mo ago
  recently there was a twit how GitHub PR diffs had 10 React components PER LINE. And how they optimized that to only 2 React components per line or something. > To summarize, for every v1 diff line there would be: > - Minimum of 10-15 DOM tree elements > - Minimum of 8-13 React Components > - Minimum of 20 React Event Handlers > - Lots of small re-usable React Components https://github.blog/engineering/architecture-optimization/the-uphill-climb-of-making-diff-lines-performant/ https://github.blog/engineering/architecture-optimization/th...
  - bananapub 5mo ago
    I'm asking about the infrastructure, obviously they chose for some reason to make my computer fans turn on to show some red and green lines on a text file.
    - dist-epoch 5mo ago
      terrible frontend architecture suggests poor engineering culture which typically spreads to all teams, including the infrastructure team
- fontain 5mo ago
  Almost every high volume service on the internet is write a little, read a lot, and when there are writes, they're relatively small, a few bytes into a database that can fan out. GitHub is very different: constant writes, large files, it is under far more pressure than the systems the rest of us build. And then, as the article says, vibecoding happens, and suddenly they're receiving 30x the volume of expensive operations. GitHub are responsible for many of the performance improvements made to Git over the years, Git scales today because of work GitHub did, but that work was never intended to scale to volume of today. Even as recently as 18 months ago, Lovable appeared, seemingly overnight, and caused huge problems for GitHub because they were creating repositories on GitHub for every single Lovable project, offloading the very high cost onto GitHub, hundreds of thousands of repositories. A couple of years before that, Homebrew used GitHub as a de facto CDN and that was a huge problem, too. Nowadays it is easy to imagine how we can scale out a service like Twitter or YouTube or Facebook because everything has been done before, but that's not true of Git, Git hasn't ever scaled like this before, there are very few examples of service with GitHub's characteristics. https://lovable.dev/blog/incident-github-outage https://lovable.dev/blog/incident-github-outage https://news.ycombinator.com/item?id=42659111 https://news.ycombinator.com/item?id=42659111
torben-friis 5mo ago
Not enough attention is being put in the production/delivery mismatch. GitHub is claiming they require 30x scale due to the giant increase in repository creation, PRs, commits, etc. I have not seen a single product increase in features or quality as an end user, nor new significant products have come out in this period (other than the LLMs themselves). Where is all this code going?
- whstl 5mo ago
  I for one believe Microsoft when they say this code is going to Github... to die. Half of my friends is vibe-coding something but they can barely get the rest of the group chat to use it once. In companies, I see people vibe-coding "miracle apps" that fall under the smallest amount of scrutiny. Basically people are doing the same developers do when they say "I can do this in a weekend", which is getting a prototype sort of running and then immediately losing energy (or in this case lacking ability) to push it forward.
  - jansan 5mo ago
    > Half of my friends is vibe-coding something but they can barely get the rest of the group chat to use it once. Some people I know can't even explain what they are trying to create.
  - jamesfinlayson 5mo ago
    Yeah I was talking to someone recently that needed some feature in a long-abandoned tool. They vibe-coded the feature and it worked, so good for them, but then they added up vibe-coding a bunch of extra features that they didn't need, just because.
- jmbwell 5mo ago
  I understood it to mean, GitHub is being crushed by LLM/AI/Agentic code review and submission, not GitHub’s code itself What I’m not seeing here but I am seeing with the Linux kernel is, most of the automatically submitted code is irrelevant or not useful (Maybe that’s what you were getting at, apologies)
LiamPowell 5mo ago
I can not figure out what on Earth they've done with these graphs, it almost seems like these are an artists impression of a graph. Looking at the commit graph: Why do commits have big steps followed by slow rolloffs? Why do the steps not happen at uniform points Why do larger steps sometimes have less of a slope than smaller steps but not all the time? Then looking at the other graphs there's completely different effects going on.
- jospeh554 5mo ago
  It's because they are your standard PowerPoint graph that just shows "thing goes up" rather than actual data, or the meaning of the data.
  - arnitdo 5mo ago
    They seem to be the result of an image-gen model to me If this is the unvetted and unbased information they are putting out in public facing-blogs, only the stars would know what data is being "presented" in their boardrooms
himata4113 5mo ago
so what they're saying is that Co-Authored-By claude@anthropic.com is overloading their systems? and that azure cannot scale fast enough to handle the load so they're embracing multi-cloud as a company... owned by microsoft? woah. what am I reading.
- 2ndorderthought 5mo ago
  AI is the new DNS when it comes to service failure.
  - cindyllm 5mo ago
    [dead]
yieldcrv 5mo ago
Ruby catching strays Good chuckle out of this post, it’s crazy that neither Atlassian (Bitbucket) or Gitlab are capturing value out of this same agentic coding boom. I wish github was separately publicly traded outside of Microsoft. Nowhere to get exposure to this
BlackFingolfin 5mo ago
GitHub stability has been bad for me. And recently even the data they show me in the web has been unreliably. Since yesterday, me and several colleagues noticed that the pull request lists on the website are incomplete, across many repositories. For example, on https://github.com/gap-system/gap/pulls https://github.com/gap-system/gap/pulls it says "Pull requests 78" in the "tab list", but the PR list view reports "35 open" (the number 78 is correct, and confirmed by e.g. `gh pr list`) And that despite <https://www.githubstatus.com https://www.githubstatus.com> reporting "all systems operational".
- embedding-shape 5mo ago
  > For example, on https://github.com/gap-system/gap/pulls https://github.com/gap-system/gap/pulls it says "Pull requests 78" in the "tab list", but the PR list view reports "35 open" (the number 78 is correct, and confirmed by e.g. `gh pr list`) Surely a scaling hack where they use "estimation" queries that return "kind of right" results instead of 100% correct data, as it's less load on the infrastructure. Not necessarily a bug as much a shit choice from product perspective.
  - BlackFingolfin 5mo ago
    If the numbers were all that is wrong, that'd be OK. But it fails to list all data -- so the only way to navigate to the missing PRs is to know their number, and manually inserting the right URL (or to go to another PR, and then edit the URL in the navigation). Sorry, but I don't think there is any way this can be classified as "not actually a bug"
- matharmin 5mo ago
  In many of my projects don't show any closed pull requests for the last 6 days. The CLI can list them, but anything going through search shows nothing. Their support acknowledged the issue, but has been silent since then, and the status page still shows nothing other than the potentially-related issue on the 27th. It looks like it has been resolved on some repositories in the meantime, but I still have the issue across multiple orgs and repositories. https://github.com/orgs/community/discussions/193388 https://github.com/orgs/community/discussions/193388
imrozim 5mo ago
As a solo dev GitHub going down is scary all my code, all my history, one platform. This makes me want to keep local backups more seriously.
- 2ndorderthought 5mo ago
  Yea or use another provider like codeberg
  - imrozim 5mo ago
    True but switching is not that easy when all your ci pipelines and integration on in GitHub.
    - embedding-shape 5mo ago
      I don't think it's 100% compatible, but Gitea's/Forgejo's (which Codeberg runs on) own Action implementation is pretty much the same as GitHub Actions, with minor differences.
      - imrozim 5mo ago
        Good to know might actually try it for one project 1 st before switching
  - maccard 5mo ago
    Personally I'd never use codeberg. Their FAQ on licensing [0] is basically everything that anyone who supports free software should abhor - it's "we might allow you to do what you want to". [0] https://docs.codeberg.org/getting-started/faq/#how-about-private-repositories%3F https://docs.codeberg.org/getting-started/faq/#how-about-pri...
- deleted 5mo ago
  [deleted]
- tosti 5mo ago
  Sorry to ask but... Do you have any idea how git works???
sltr 5mo ago
One thing is clear: an LLM wrote this.
everfrustrated 5mo ago
So they haven't even finished migrating from their datacenters to Azure and have now started a project to add another cloud provider ("multi cloud")? Madness.
otar 5mo ago
I had to postpone a call with developers (in 2 different countries) because I didn't had access to the issues board, which is a single source of truth for us. I understand the rapid growth (because of AI agents), but if such critical software service becomes unstable then it's time to migrate? Thinking about self-hosting GitLab.
- embedding-shape 5mo ago
  > but if such critical software service becomes unstable then it's time to migrate? Right way to think about this: > If things we need/see as critical for our work are hosted on a platform with really bad reliability, it's time for us to migrate My internet connection at home is really shit, and almost every week there is a multi-hour downtime for some reason, not to mention when La Liga games are on TV anything using Cloudflare is unavailable, so I've had to spend extra energy and time to setup things in a way so I can still work whenever this happens.
throwatdem12311 5mo ago
> The main driver is a rapid change in how software is being built. Leopard, meet face. Too little too late, yesterday was the straw that broke the camel’s back for us and we’ve started a migration to a self-hosted GitLab.
lousken 5mo ago
Availability is priority? Does not seem like it is https://mrshu.github.io/github-statuses/ https://mrshu.github.io/github-statuses/
eolgun 5mo ago
The AI agent growth explanation is interesting but also a bit of a deflection. If a meaningful portion of your traffic is now automated agents, your capacity planning model is fundamentally different, you're no longer scaling for human paced workflows but for burst patterns that look nothing like historical load. The unlabeled graphs don't help the credibility case. When you are already in the hole on trust, shipping a post that requires readers to assume favorable baselines is exactly the wrong move.
jameskilton 5mo ago
Nice, they have availability numbers now on their status page, but they aren't aggregating. If you multiply all current numbers together (as of Apr 28), you find out that GitHub has a 97.26% uptime. One ... single ... 9. They can do better.
- embedding-shape 5mo ago
  Kind of unfair though, do the same for any platform with multiple services and you'd probably get <99% for most of them. > you find out that GitHub has a 97.26% uptime Calculating that to "Downtime per day" you get ~40 minutes of downtime per day, almost a week per year. Crazy stuff for something essential like this.
deleted 5mo ago
[deleted]
Waterluvian 5mo ago
I have a hard time believing anything what's said in a blog post where a graph lacks axes labels/scale. It tells me that nobody who cares about correctness had any say on the content of the post. Maybe I'm being 8am cranky and pedantic, but I'm sticking with it. > availability first, then capacity, then new features. I'd love to experience first-hand a leadership team who says, "stop accepting new paying customers until we've got availability sorted out!"
- madeofpalk 5mo ago
  Like they did with Copilot last week? https://github.blog/news-insights/company-news/changes-to-github-copilot-individual-plans/ https://github.blog/news-insights/company-news/changes-to-gi... > New sign-ups for GitHub Copilot Pro, Pro+, and Student plans are paused. Pausing sign-ups allows us to serve existing customers more effectively.
GS_Projects 5mo ago
The bit nobody covers in these write-ups: small teams without dual-cloud failover budget. Last big GitHub outage cost me a deploy day. Not catastrophic but the kind of thing you don't budget for when GitHub is your single source of truth. Status page is also still doing that thing where every component is green but in practice clone is hanging, push is timing out, actions are stuck. Per-service uptime is a managed number. The user-experience number is the one that matters and it's not in the post-mortem.
- AlexeyBelov 5mo ago
  No LLM comments.
jimmypk 5mo ago
[flagged]
dzonga 5mo ago
blame MySQL. Blame Ruby. on another note - is the exponential growth from 'agentic' workflows actually resulting in productive software in the wild. Or it is just noise. On my end I haven't seen the software I use getting better.
dangoodmanUT 5mo ago
Two incidents? Just two? In seriousness, looking at their scale, this is an insane engineering challenge. Especially if they’re moving databases, not easy ever, and certainly not at that scale
mendyberger 5mo ago
I wonder if this mess has anything to do with talent loss resulting from layoffs after the pandemic
- pointlessone 5mo ago
  I’d guess it has much more to do with the extra load agentic ai generates. If we take the charts in the OP at face value, do you think gh suddenly exploded in popularity? At this point I think almost everyone who has any use for gh already has an account and use it as much as they ever would. But all the charts go to the moon. Gh obviously didn’t take into account that ai agents can generate a lot of activity they don’t have capacity for.
OutOfHere 5mo ago
> we accelerated parts of migrating performance or scale sensitive code out of Ruby monolith into Go. I am surprised that Microsoft is allowed to use Go. How long will it be before a bean counter forces a rewrite to a Microsoft favored language?
- senderista 5mo ago
  They used Go for the new TypeScript compiler!
agluszak 5mo ago
Regarding their image with stats (https://github.blog/wp-content/uploads/2026/04/record-accelleration-1920x1080-2.png https://github.blog/wp-content/uploads/2026/04/record-accell...) - what exactly are the ranges on y-axes? I doubt they had close to 0 PRs merged in 2023 ;)
000ooo000 5mo ago
Load from paying customers vs. load from nonpaying users would be interesting to know. No doubt omitted deliberately.
clvx 5mo ago
With this prioritization Github IPv6 support is gonna happen the next decade.
- snihalani 5mo ago
  IPv6 doesn't sound like a huge lift at the entrypoints. Internal networking to IPv6 only sounds like an impossible lift
perbu 5mo ago
fwiw, I've had good luck scaling git, specially doing clones, in the HTTP layer, using Varnish. this was CI bringing Github Enterprise to it's knees.
JimmaDaRustla 5mo ago
AS IF THEY POST THIS WHILE THEIR SEARCH IS BROKEN, what a circus
pier25 5mo ago
Github has been having availability issues for years now.
zinodaur 5mo ago
> posts graphs without way to determine scale of y axis Now that’s the kind of excellence I expect from the GitHub engineering team
BigTTYGothGF 5mo ago
LLMs and vibe coding ruining it for the rest of us.
devmor 5mo ago
Microsoft has been an abysmal steward of Github - the few nice features it has over self-hosting just aren't worth losing an hour or more of CI/CD downtime during daylight hours every week. Yesterday was the last straw for me - I've begun migrating my personal private projects and my contracting firm's projects off of github.
mrhottakes 5mo ago
LLMs have helped us invent websites that only work sometimes. We're truly living in the future.
zamalek 5mo ago
> Our priorities are clear: availability first, then capacity, then new features. No mention of Copilot/slopiffication. Probably an intentional omission as Microsoft only has one true priority across all of its products.
saghm 5mo ago
Given what "An Update on <XYZ>" usually means, I can only assume this means that Github has decided to no longer provide availability. Not particularly surprising given current trends I guess
dangus 5mo ago
Notice how the graphs have no Y axis. That's how you know it's manipulative. This company is owned by one of the major causes of the AI boom and is hiding behind difficulty scaling, despite its parent company also being a premier source of scaling solutions. GitHub: don't gaslight your customers. It is not your customers' problem that you're having trouble scaling. Nobody cares. Give us the service we are paying you for and make it reliable, or else we'll choose something else. After the words "Both of those incidents are not acceptable" the blog post should have been over. Nobody needs to hear a sob story about how your service is too popular.
danra 5mo ago
When it's down to brass tacks, the most common GitHub action, actions/checkout, is not taking contributions due to "focus [...] on strategic areas" [0] despite having years-old issues - here's one[1] that soon celebrates its sixth birthday, despite having an available PR! [0] https://github.com/actions/checkout#note https://github.com/actions/checkout#note [1] https://github.com/actions/checkout/issues/270#issue-628967704 https://github.com/actions/checkout/issues/270#issue-6289677...
bartread 5mo ago
> I wanted to give an update on GitHub’s availability in light of two recent incidents. [Emphasis mine] Vlad, you are living in a very different world to me. GitHub has suffered dozens and dozens of outages since the beginning of the year. It is notably less available and reliable than it was even as recently as last year. People have created dashboards and heatmaps showing how bad GitHub has become. At least one of those has made it to the front page of Hacker News. In fact its unreliability and persistent availability issues have become a frequent topic of conversation across sites and communities frequented its users - of which HN and Reddit are two obvious examples. At this point GitHub's unreliability risks becoming a meme, if it hasn't already done so. The only thing your post makes clear is that your priorities ARE NOT clear. > Our priorities are clear: availability first, then capacity, then new features. WRONG! Your priorities are: 1. Availability 2. Availability 3. Availability You have NO OTHER PRIORITIES. If you want other priorities, focus on AVAILABILITY for 6 months and then come back and we can all have a serious conversation about something else. In the meantime, you need to understand that GitHub's reliability over months and months - not just in April - has been completely unacceptable. Focus on fixing that and on nothing else.
- dminik 5mo ago
  I've recently built a script that periodically (every 25 minutes) fetches the latest merged PRs to check for some potential rule violations. I'm not an admin and couldn't get the events API working, so I just resorted to polling. On an average ~8 hour working day, there's at least one failed request. In fact, looking over the logs, I can't spot a single day that did not have a failed request. Now, I can't guarantee that these are all caused by GitHub (as opposed to my connection), but it is pretty funny.
- dude250711 5mo ago
  Microsoft board and shareholders: "LOL, nah! More vibe coding inside and outside plz".
- baobun 5mo ago
  Wow. Security or trust not even making the list.
gamerslexus 5mo ago
> The main driver is a rapid change in how software is being built. Since the second half of December 2025, agentic development workflows have accelerated sharply. So, it's because of LLMs guys.
russellthehippo 5mo ago
Reading the capacity crunch idea made me a little more empathetic to their issues - 30x in one year is a lot when you're starting from a high baseline. Now that being said...I'd really appreciate more availability.
mw888 5mo ago
Their ostensible troubles are fueled by "exponential usage growth", demonstrated by three graphs which exclude axis labels and are aggressively cropped.
twobitshifter 5mo ago
let’s do git without github again
init_31 5mo ago
GitHub wants everyone to use AI agents but if you try to register a new account today you are greeted by an endless gauntlet of absolutely ridiculous captchas that reject you because they think you're an AI agent and nobody should be using AI agents on GitHub.
chanux 5mo ago
I have a feeling that this post is good enough reassurance for big money corps. Long live Github under MS!
JackSlateur 5mo ago
I do not understand how github can still have issues .. I mean: obviously, in the old world, it would have taken lot of work to improve the situation. But now, with AI, just plug copilot against the code base and everything is fixed in a week, no ?

421 points by salkahfi 5mo ago

The availability of GitHub is still at 0% - it can't be reached over IPv6.

> we started working on path to multi cloud. Is this microsoft stating that they aren't able to get acceptable reliability from Azure? (I mean, I think a lot of us have heard that, but it's interesting to hear it from microsoft themselves).

derwiki 5mo ago
It’s pretty damning. But as someone who has used Azure, I buy it.
- everfrustrated 5mo ago
  Pretty damming that two Microsoft subsidiaries - GitHub and LinkedIn - either shelved their forced migration to Azure or are looking at non-Azure options.
jasoncartwright 5mo ago
Seems pretty sensible to not rely on a single provider for their large complex system?
- mijoharas 5mo ago
  I mean, amazon (shopping, along with prime video e.t.c.) runs on AWS.
  - jasoncartwright 5mo ago
    Prime video uses a non-AWS CDN when I watch football on it here in the UK
    - farfatched 5mo ago
      The BBC were unable to find a single CDN that could serve the UK during its peak football matches. https://www.bbc.co.uk/webarchive/https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Finternet%2Fentries%2F8c6c2414-df7a-4ad7-bd2e-dbe481da3633 https://www.bbc.co.uk/webarchive/https%3A%2F%2Fwww.bbc.co.uk...
    - jamesfinlayson 5mo ago
      I'd believe it - CloudFront always felt a bit like AWS ticking a box ("we have a CDN") rather than being a good to use product.
    - grogenaut 5mo ago
      that's called load balancing and regional availability. many companies do multi-cdn. in fact it's smart to use multiple CDNs so you can do better in contract time. Twitch uses IVS but we have failover to other CDNs for very large events.
  - ksimukka 5mo ago
    When I was at AWS, retail was not yet running on AWS. Has that changed? Prime video does use some AWS services, but live and on-demand are two entirely different beasts.
    - mijoharas 5mo ago
      Really? I thought retail was. It's been almost a decade since I worked at prime video but I think everything was running on AWS. (Some things didn't use brazil etc, but I think all the servers etc. were on AWS)
      - malfist 5mo ago
- cyanydeez 5mo ago
  This isn't a mom and pop shop. They have locations all over the world: https://datacenters.microsoft.com/ https://datacenters.microsoft.com/ There's no intrinsic reason they should be vulnerable to themselves.
  - jasoncartwright 5mo ago
    That website (for me) uses Cloudflare via WPEngine, which also isn't Azure
  - farfatched 5mo ago
    +1. Multi-cloud is typically done for vendor independence. But Github don't have that rationale.
  - tacticus 5mo ago
    and many of those azure regions are a single DC or single cage in a single dc that you can walk past to find the bigger netflix pop.
- embedding-shape 5mo ago
  Man, you should have been there 6 months ago when they decided to start tearing down GitHub's own data centers and move everything exclusively to Azure. Seems they themselves realized this after they started moving, but imagine if you could have helped them realize this before they even started :)
  - benterix 5mo ago
    > Seems they themselves realized this after they started moving I guess most people at Github knew exactly it makes no sense but they didn't really have a choice. Maybe some voiced their statement, got "we hear you" in response and were told to proceed anyway.
    - embedding-shape 5mo ago
      Yeah, I don't know how it went down, but I also know exactly how it went down: Microsoft Execs: Everyone needs to move to Azure! GitHub developers: But Azure is not gonna be able to handle our load, we literally have our own data centers! Microsoft Execs: Sure, but you're Microsoft now, please publish blog post about how in half a year you'll be 100% on Azure. Few months later... GitHub Developer: We've tried our best, users are leaving in droves and Azure can't keep up! Microsoft Execs: Ok fine, you can use something else too, but only if you mainly use Azure and continue publishing blog posts about how great Azure is.
      - alper 5mo ago
        Azure is the MS Teams of clouds.
  - nextaccountic 5mo ago
    Made me think. Why not convert Github datacenters into Azure datacenters that have Github as their sole customer? Then it's up to Azure how they will manage this
    - hobofan 5mo ago
cbg0 5mo ago
I think this is more tailored towards enterprise clients that lose money when Github is down, that would probably help with retention.
- bombcar 5mo ago
  You’d think they could have had the existing GitHub on whatever continue as is (maybe for paying customers) while all the AI new inrush goes to the Azure setup.
- jofzar 5mo ago
  Yeah that's a top tier enterprise plan feature if I have ever seen ut
jansan 5mo ago
The entire concept of multi cloud is amusing if you think what cloud originally was supposed to be. They could call them meta clouds (might infringe trademarks), and with the current growth trajectory of AI generated code eventually multi-meta-clouds, renamed to beyond-clouds, and then multi-beyond-clounds. I see no limits.
deleted 5mo ago
[deleted]
youwangd 5mo ago
Show HN timing matters more than people think. Monday-Thursday, 9-11am Pacific, is when the front page has the most engaged readers. Weekend posts get less competition but also less engagement.
zamalek 5mo ago
There was somewhat recently a post here about how priorities, pressure, and management subverted Dave Cutler's vision for Azure (which was to have near zero human involvement) - my Google fu isn't strong enough to find it. Supposedly, someone running over or opening a serial to a rack/VM is now typical operational procedure.
- ok_dad 5mo ago
  This one? https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion https://isolveproblems.substack.com/p/how-microsoft-vaporize...
  - zamalek 5mo ago
    That's the one!
    - consumer451 5mo ago
      Amazing read. Thanks to both of you for finding that. > I later researched this further and found that no one at Microsoft, not a single soul, could articulate why up to 173 agents were needed to manage an Azure node, what they all did, how they interacted with one another, what their feature set was, or even why they existed in the first place. This reads like a description of the SLS-based (aka Senate Launch System) Artemis program, which somehow ended up deciding that the insane Lunar Gateway should be a thing. Destin (SmarterEveryDay on YouTube) [0] called out the entire nutball scheme to NASA, at NASA. This includes the SLS/Orion/Lunar Gateway insanity, and calling out the number of unknown, but very large number, of on-orbit refuelings that Starship would need to get to the moon. In that video's comments, I believe there is someone who worked on the Orion-related system, who says ~"Yeah, we thought the delta-v was too low, we could have increased it, but no one was speaking with each other at a whole system level." The mission drift at large orgs, gov and corp, is a huge problem that might one day be solved? [0] https://www.youtube.com/watch?v=OoJsPvmFixU https://www.youtube.com/watch?v=OoJsPvmFixU
      - 5mo ago
tedd4u 5mo ago
> multi-cloud XXXXL size project. May not ever deliver. But if it fails, it will only do so after years grinding through people, resources, etc.

pluc 5mo ago

There are no words that Microsoft can use that would make me trust Microsoft.

baq 5mo ago

openai, anthropic, google and a plethora of chinese models all end up pushing code into github. you can discuss whether gpt 5.5 is better than opus 4.7, but for github it doesn't matter: they'll be receiving the code no matter which llm spits it out. amazing on one hand, quite scary on the other for github and all other forges if this continues and there is no reason why it wouldn't.

graemep 5mo ago
Simple solution: charge all users. Charge more for higher usage.
- gattr 5mo ago
  And/or provide a baseline free tier, corresponding to how much a typical human user would at most push/clone etc. They have pre-LLM statistics on that.

jcattle 5mo ago

When there's a gold rush invest in checks notes jewellery makers?

huijzer 5mo ago

I’m pretty sure my Forgejo instance on a Raspberry Pi is outperforming GitHub reliability. It’s faster that’s for sure.

huijzer 5mo ago
Why the downvotes? I’m serious. On GitHub I’ve experienced many downtimes. My Forgejo hasn’t gone down yet apart from reboots by me.

darkwater 5mo ago

Glad that they released some data about new repo/issues/commits over the last years. It confirms what everyone else already believed from the outside: agents are putting a lot of extra, sudden pressure on GitHub. It's like a startup that is growing exponentially, with the difference that they already have a large user base to serve - and that keeps them in the bullseye - and probably a not-so-fast-moving organization when it comes down to changes. On the other side of the coin, they also have a lot of talent, infra and money a startup might not have yet.

maccard 5mo ago
What data is that? There's an unlabelled graph and a number at the current peak.
- ncruces 5mo ago
  Some previous numbers: https://x.com/kdaigle/status/2040164759836778878 https://x.com/kdaigle/status/2040164759836778878
  - maccard 5mo ago
    This is the data that should be in the blog post. Thanks for sharing.
- deleted 5mo ago
  [deleted]
- darkwater 5mo ago
  IMO it transmits the magnitude of the impact pretty well.

frangonf 5mo ago

What are we doing? Stop subsidizing tokens now that we extracted enough training data from you and we have enough agentic junkies business to keep the flywheel going up and cut on the loss leaders. [0] [0] https://news.ycombinator.com/item?id=47923357 https://news.ycombinator.com/item?id=47923357

guidoiaquinti 5mo ago

> While we were already in progress of migrating out of our smaller custom data centers into public cloud, we started working on path to multi cloud. This longer-term measure is necessary to achieve the level of resilience, low latency, and flexibility that will be needed in the future. Wild

maccard 5mo ago

It's kind of hard to read this with a straight face. The unlabelled graph with big numbers on top, the priorities that don't match with what we're experiencing, and a list of things that they're doing without a real acknowledgement of the _dire_ uptime over the last 12 months....

ramon156 5mo ago
"We hear you" in ~300 words, basically.
ncruces 5mo ago
More numbers: https://x.com/kdaigle/status/2040164759836778878 https://x.com/kdaigle/status/2040164759836778878 What's the question here, you don't believe growth is currently exponential, or do you think it shouldn't be hard to scale, when 10x YoY is not enough?
- OtherShrezzing 5mo ago
  As a business user, our costs have gone up while service has gone down dramatically. Meanwhile our marginal cost to GitHub has hardly changed. Where our costs to them have increased, they mostly charge us per cpu minute, so obviously aren’t making any kind of loss on our account. I’m sure they’re experiencing scaling issues across the platform, but it’s unacceptable for that to have a negative impact on us when we're sending them $250/dev/yr for (what is in all honesty) hosting a bunch of static text files.
  - rdevilla 5mo ago
    > we're sending them $250/dev/yr for (what is in all honesty) hosting a bunch of static text files. You know, you can just host your own code forge. Or you can just drop gitolite on a server. Or pull directly from each others' dev machines on a LAN. GitHub is not git.
    - OtherShrezzing 5mo ago

nraynaud 5mo ago

So I gather that nobody is working on a search that stays on the current branch?

fontain 5mo ago

Personally, I’m sympathetic. We know that GitHub did a huge amount of work over the last decade to make Git scale, which has benefited us all. These new scaling challenges are real challenges, 30x growth would be a nightmare for any system that was already pushing the limits of what was possible, I think we are being far too hard on GitHub, they deserve a little grace.

someone_eu 5mo ago
GitHub's scaling issues are caused by their own vendor-lock approach and monopoly. Yes, of course _their_ goal is to be even bigger and even more all-consuming, so _they_ have to deal with the scale. Why a user would be sympathetic to that? The user (and not a big tech monopoly) answer to scaling issues is almost always to stop scaling and start federating and interoperating.
remus 5mo ago
For all the negatives about github I agree. They offer a lot of free stuff, and LLMs seem likely to put massively increase their costs with no guarantee they'll be making money off it. I can't think of many (any?) large businesses which could scale up to meet so much new demand without some significant growing pains along the way.

icy 5mo ago

I'm biased (founder of tangled.org), but the future really should be federated forges. Host repositories on sovereign infra with global identity + federated "metadata" (issues, pulls, etc.). Global indices for this should be trivial to spin up so availability is never a concern (we're working towards this!).

ramon156 5mo ago
Love the idea, would replace the LLM generated content ony our site, though. I recently migrated to codeberg because I'm okay with self-hosting big runners, while using codeberg's available runners for smaller cron-based things (they even have lazy runners for this).
- icy 5mo ago
  It’s… all hand written? We just sound “professional”.
ArcHound 5mo ago
But, there are? I can host a repo on GitHub, Codeberg and self host it too. Then I need to watch over main to keep it consistent between those. After that's established, I can do updates from wherever. Link'em in the README.
- nibbleyou 5mo ago
  There's also a tool to automatically push it to multiple repos: https://github.com/prashantsengar/GitEcho https://github.com/prashantsengar/GitEcho Disclaimer: the author is a colleague of mine Though to be fair, what the parent meant by federated forges is different than this approach.
  - pabs3 5mo ago
    git itself can push to multiple URLs btw: https://stackoverflow.com/questions/849308/how-can-i-pull-push-from-multiple-remote-locations https://stackoverflow.com/questions/849308/how-can-i-pull-pu...

jftuga 5mo ago

Some interesting tid bits: * we had to resolve a variety of bottlenecks that appeared faster than expected from moving webhooks to a different backend (out of MySQL) * * redesigning user session cache to redoing authentication and authorization flows to substantially reduce database load. * we accelerated parts of migrating performance or scale sensitive code out of Ruby monolith into Go. I'd like to know what database backend they migrated to. I was also surprised to read that the migration from Ruby to a more performant language had not already been completed. I assume this is because it a large code base with many moving parts, etc.

mohsen1 5mo ago
Another interesting bit: they are hitting performance issues due to the rise of monorepos. GitHub and frankly Git were not designed for monorepos
- ghthor 5mo ago
  Yet the Linux kernel is a monorepo
  - mohsen1 5mo ago
    Try google3
  - guipsp 5mo ago
    The Linux kernel is pretty small

deleted 5mo ago

[deleted]

rootnod3 5mo ago

> Our priorities are clear: availability first That's a delayed April fool's right?

embedding-shape 5mo ago
No, just a 6 month old memo that was first opened today, as they said literally the same 6 months ago.

embedding-shape 5mo ago

Hah, love that now they say "Our priorities are clear: availability first, then capacity, then new features" when 6 months ago, it was seemingly exactly the same except Azure supposedly was gonna save them: > GitHub Will Prioritize Migrating to Azure Over Feature Development - GitHub is working on migrating all of its infrastructure to Azure, even though this means it'll have to delay some feature development. > In a message to GitHub’s staff, CTO Vladimir Fedorov notes that GitHub is constrained on capacity in its Virginia data center. “It’s existential for us to keep up with the demands of AI and Copilot, which are changing how people use GitHub,” he writes. https://thenewstack.io/github-will-prioritize-migrating-to-azure-over-feature-development/ https://thenewstack.io/github-will-prioritize-migrating-to-a... So the currently delayed feature development is now gonna be further delayed, yet almost every week we see new features and changes, just the other day the single issues view was changed, as just one example. And it was "existential" 6 months ago yet they keep stumbling on the exact same issue today? Even if they're focused exclusively on reliability and uptime, we get the experience that we have today, kind of incredible how a company with the resources of Microsoft seemingly are unable to stop continuously shot themselves in the foot. It's kind of impressive actually. As icing on the cake, they've decided to buy up all popular developer services then migrate them all to the same platform, great idea too.

ncruces 5mo ago
> So the currently delayed feature development is now gonna be further delayed, yet almost every week we see new features and changes, just the other day the single issues view was changed, as just one example. They did that as a panic mode hack to mitigate performance: https://news.ycombinator.com/item?id=47912521 https://news.ycombinator.com/item?id=47912521
madeofpalk 5mo ago
This seems uncharitable. Priorities aren't exclusive, especially at scale across large engineering orgs like GitHub. It could be that these are the top level priorities, but teams or individuals who aren't able to contribute to these priorities will work on other things like new features.

deleted 5mo ago

[deleted]

cedws 5mo ago

I wonder if they’ll end the free lunch we’ve been having since the MS takeover. There’s been a deluge of spam and crapware projects due to the LLM wave which is visible in that graph. Can’t see them sustaining being a public dustbin for low value projects forever.

sbarre 5mo ago
I could see them expiring/archiving/deleting inactive projects after some time. I feel like this would have negative impacts (lots of interesting historical archives on Github) but maybe if a project hasn't been touched, or cloned, in some time, it just gets deleted with some notice.
- deleted 5mo ago
  [deleted]
- rmunn 5mo ago
  Thing is, projects that don't get touched for months and months are the least costly. Disk space is cheap; what's costly is compute time to process new commits, new/updated/closed issues, new/reviewed/merged PRs, and so on. Inactive projects just sit there taking up disk space but basically zero compute time. So it would make no sense at all for them to delete old, inactive projects. (Which doesn't mean they won't do it: they might have hidden costs I'm unaware of, or they might make stupid decisions. People do make stupid decisions sometimes).
  - ifwinterco 5mo ago
    Also creates a perverse incentive to automatically push random commits to make sure your repos stay “active” and don’t get deleted, creating more load
  - sbarre 5mo ago
    Yeah that's a fair point!
- jamesfinlayson 5mo ago

s_ting765 5mo ago

> Vladimir Fedorov is GitHub's Chief Technology Officer .... He currently serves on the board of Codepath.org, an organization dedicated to reprogramming higher education to create the first AI-native generation of engineers, CTOs, and founders. I think I found the issue.

dude250711 5mo ago
Sounds like a bet that AGI is not achievable.

latexr 5mo ago

> The main driver is a rapid change in how software is being built. Since the second half of December 2025, agentic development workflows have accelerated sharply. GitHub instability has started way before that. I understand it’s too much to ask of a trillion-dollar corporation to consider the impact of their own actions, but perhaps they should’ve thought of that before forcing LLM development down everyone’s throats.

mathgeek 5mo ago
While they contributed, they were still following the market trend anyway. If they weren't letting folks use it directly, other companies would have (and are).
- latexr 5mo ago
  > they were still following the market trend anyway. They started the trend with Copilot. > If they weren't letting folks use it directly There is a chasm of difference between “letting you use it” and “forcing it down your throat”. Microsoft is doing the latter, not the former. Copilot is annoyingly present by default at every step on GitHub.
  - jcgrillo 5mo ago
    They should have just called it Clippy and revived the animated avatar.

steve1977 5mo ago

I know that I'm simplifying (probably too much), but it seems like things were fine when GitHub was still a Ruby on Rails monolith and all the rigmarole with microservices etc. only made things worse.

tankenmate 5mo ago
This sounds more like a belief, based on little more than "correlation is causation", than analysis that controls for macro-trends backed by evidence.
- embedding-shape 5mo ago
  GitHub been oscillating between long phases of "Never any new features but rock-solid and no downtime" and "New features every week but also unicorns (used to be the "service unavailable page") every week" for as long as I can remember. Seems they're on some interval switching between the two.
- sgarland 5mo ago
  It is, but everyone is entitled to beliefs. Anecdotally, I feel the same way. Everywhere I’ve been, there has always been a legacy monolith that was stable as a rock, with dozens of new microservices scattered around it in an attempt to exit the monolith. The microservices have never once been stable. People fail to take the most basic things into consideration, like “you can’t have Consistency and Availability when everything is a network call.” I’m sure survivor bias is at play here, but when I look through the older code bases - especially the data model - it’s an entirely different world than the newer stuff, and it’s clear which of the two was written by people who understand systems.
  - steve1977 5mo ago
    I mean one of the selling points of microservices is, that the developer can only be concerned with their slice of the whole and does not need to understand systems. Maybe this is not so clever after all. Or, it would require an architect who has a very good understanding of the system. Which in reality seems to be rare.

sikozu 5mo ago

This latest incident was the nail in the coffin for me. I've been on GitHub since 2012 but I'm feeling the pull to migrate out to Gitea/Forgejo. Has anybody done this recently? How'd it go?

deleted 5mo ago
[deleted]
embedding-shape 5mo ago
When one of the incident they write about here happened, I wrote about my experience moving from GitHub to Forgejo which I happened to complete just the night before that happened: https://news.ycombinator.com/item?id=47878192 https://news.ycombinator.com/item?id=47878192 (lots of other people sharing their experience as replies too) I was thinking of maybe doing a proper write up about how to host your own Forgejo + Action runners on Linux, Windows and macOS, not sure if there is enough interest. What would people for sure want to know in a guide/explanation of this?
sltr 5mo ago
I moved over back when GitHub was planning to charge per minute to use my own runner. It was easy with Claude, the gh API, and forgejo web API. I even set up daily backups to my S3 clone of choice. The only repos I left on GitHub are forks and one with a bit of public engagement.

bananapub 5mo ago

anyone who's actually worked there, could you explain why they're finding scalability and reliability so hard? naively it seems like 'repo groups', ie clusters of repositories linked by being mutual forks, would be fairly isolated for the whole git storage layer, and everything else feels pretty easily parallelisable (issues, actions, etc, modulo taking locks now and then to submit results or whatever). and given that, surely you can incrementally deploy changes across those many shards to avoid most big outages? are there big conceptual serialisations that I've missed? is it just not well factored? was the move to Azure just a catastrophically bad idea? some other thing?

dist-epoch 5mo ago
recently there was a twit how GitHub PR diffs had 10 React components PER LINE. And how they optimized that to only 2 React components per line or something. > To summarize, for every v1 diff line there would be: > - Minimum of 10-15 DOM tree elements > - Minimum of 8-13 React Components > - Minimum of 20 React Event Handlers > - Lots of small re-usable React Components https://github.blog/engineering/architecture-optimization/the-uphill-climb-of-making-diff-lines-performant/ https://github.blog/engineering/architecture-optimization/th...
- bananapub 5mo ago
  I'm asking about the infrastructure, obviously they chose for some reason to make my computer fans turn on to show some red and green lines on a text file.
  - dist-epoch 5mo ago
    terrible frontend architecture suggests poor engineering culture which typically spreads to all teams, including the infrastructure team
fontain 5mo ago
Almost every high volume service on the internet is write a little, read a lot, and when there are writes, they're relatively small, a few bytes into a database that can fan out. GitHub is very different: constant writes, large files, it is under far more pressure than the systems the rest of us build. And then, as the article says, vibecoding happens, and suddenly they're receiving 30x the volume of expensive operations. GitHub are responsible for many of the performance improvements made to Git over the years, Git scales today because of work GitHub did, but that work was never intended to scale to volume of today. Even as recently as 18 months ago, Lovable appeared, seemingly overnight, and caused huge problems for GitHub because they were creating repositories on GitHub for every single Lovable project, offloading the very high cost onto GitHub, hundreds of thousands of repositories. A couple of years before that, Homebrew used GitHub as a de facto CDN and that was a huge problem, too. Nowadays it is easy to imagine how we can scale out a service like Twitter or YouTube or Facebook because everything has been done before, but that's not true of Git, Git hasn't ever scaled like this before, there are very few examples of service with GitHub's characteristics. https://lovable.dev/blog/incident-github-outage https://lovable.dev/blog/incident-github-outage https://news.ycombinator.com/item?id=42659111 https://news.ycombinator.com/item?id=42659111

torben-friis 5mo ago

Not enough attention is being put in the production/delivery mismatch. GitHub is claiming they require 30x scale due to the giant increase in repository creation, PRs, commits, etc. I have not seen a single product increase in features or quality as an end user, nor new significant products have come out in this period (other than the LLMs themselves). Where is all this code going?

whstl 5mo ago
I for one believe Microsoft when they say this code is going to Github... to die. Half of my friends is vibe-coding something but they can barely get the rest of the group chat to use it once. In companies, I see people vibe-coding "miracle apps" that fall under the smallest amount of scrutiny. Basically people are doing the same developers do when they say "I can do this in a weekend", which is getting a prototype sort of running and then immediately losing energy (or in this case lacking ability) to push it forward.
- jansan 5mo ago
  > Half of my friends is vibe-coding something but they can barely get the rest of the group chat to use it once. Some people I know can't even explain what they are trying to create.
- jamesfinlayson 5mo ago
  Yeah I was talking to someone recently that needed some feature in a long-abandoned tool. They vibe-coded the feature and it worked, so good for them, but then they added up vibe-coding a bunch of extra features that they didn't need, just because.
jmbwell 5mo ago
I understood it to mean, GitHub is being crushed by LLM/AI/Agentic code review and submission, not GitHub’s code itself What I’m not seeing here but I am seeing with the Linux kernel is, most of the automatically submitted code is irrelevant or not useful (Maybe that’s what you were getting at, apologies)

LiamPowell 5mo ago

I can not figure out what on Earth they've done with these graphs, it almost seems like these are an artists impression of a graph. Looking at the commit graph: Why do commits have big steps followed by slow rolloffs? Why do the steps not happen at uniform points Why do larger steps sometimes have less of a slope than smaller steps but not all the time? Then looking at the other graphs there's completely different effects going on.

jospeh554 5mo ago
It's because they are your standard PowerPoint graph that just shows "thing goes up" rather than actual data, or the meaning of the data.
- arnitdo 5mo ago
  They seem to be the result of an image-gen model to me If this is the unvetted and unbased information they are putting out in public facing-blogs, only the stars would know what data is being "presented" in their boardrooms

himata4113 5mo ago

so what they're saying is that Co-Authored-By claude@anthropic.com is overloading their systems? and that azure cannot scale fast enough to handle the load so they're embracing multi-cloud as a company... owned by microsoft? woah. what am I reading.

2ndorderthought 5mo ago
AI is the new DNS when it comes to service failure.
- cindyllm 5mo ago
  [dead]

yieldcrv 5mo ago

Ruby catching strays Good chuckle out of this post, it’s crazy that neither Atlassian (Bitbucket) or Gitlab are capturing value out of this same agentic coding boom. I wish github was separately publicly traded outside of Microsoft. Nowhere to get exposure to this

BlackFingolfin 5mo ago

GitHub stability has been bad for me. And recently even the data they show me in the web has been unreliably. Since yesterday, me and several colleagues noticed that the pull request lists on the website are incomplete, across many repositories. For example, on https://github.com/gap-system/gap/pulls https://github.com/gap-system/gap/pulls it says "Pull requests 78" in the "tab list", but the PR list view reports "35 open" (the number 78 is correct, and confirmed by e.g. `gh pr list`) And that despite <https://www.githubstatus.com https://www.githubstatus.com> reporting "all systems operational".

embedding-shape 5mo ago
> For example, on https://github.com/gap-system/gap/pulls https://github.com/gap-system/gap/pulls it says "Pull requests 78" in the "tab list", but the PR list view reports "35 open" (the number 78 is correct, and confirmed by e.g. `gh pr list`) Surely a scaling hack where they use "estimation" queries that return "kind of right" results instead of 100% correct data, as it's less load on the infrastructure. Not necessarily a bug as much a shit choice from product perspective.
- BlackFingolfin 5mo ago
  If the numbers were all that is wrong, that'd be OK. But it fails to list all data -- so the only way to navigate to the missing PRs is to know their number, and manually inserting the right URL (or to go to another PR, and then edit the URL in the navigation). Sorry, but I don't think there is any way this can be classified as "not actually a bug"
matharmin 5mo ago
In many of my projects don't show any closed pull requests for the last 6 days. The CLI can list them, but anything going through search shows nothing. Their support acknowledged the issue, but has been silent since then, and the status page still shows nothing other than the potentially-related issue on the 27th. It looks like it has been resolved on some repositories in the meantime, but I still have the issue across multiple orgs and repositories. https://github.com/orgs/community/discussions/193388 https://github.com/orgs/community/discussions/193388

imrozim 5mo ago

As a solo dev GitHub going down is scary all my code, all my history, one platform. This makes me want to keep local backups more seriously.

2ndorderthought 5mo ago
Yea or use another provider like codeberg
- imrozim 5mo ago
  True but switching is not that easy when all your ci pipelines and integration on in GitHub.
  - embedding-shape 5mo ago
    I don't think it's 100% compatible, but Gitea's/Forgejo's (which Codeberg runs on) own Action implementation is pretty much the same as GitHub Actions, with minor differences.
    - imrozim 5mo ago
      Good to know might actually try it for one project 1 st before switching
- maccard 5mo ago
  Personally I'd never use codeberg. Their FAQ on licensing [0] is basically everything that anyone who supports free software should abhor - it's "we might allow you to do what you want to". [0] https://docs.codeberg.org/getting-started/faq/#how-about-private-repositories%3F https://docs.codeberg.org/getting-started/faq/#how-about-pri...
deleted 5mo ago
[deleted]
tosti 5mo ago
Sorry to ask but... Do you have any idea how git works???

sltr 5mo ago

One thing is clear: an LLM wrote this.

everfrustrated 5mo ago

So they haven't even finished migrating from their datacenters to Azure and have now started a project to add another cloud provider ("multi cloud")? Madness.

otar 5mo ago

I had to postpone a call with developers (in 2 different countries) because I didn't had access to the issues board, which is a single source of truth for us. I understand the rapid growth (because of AI agents), but if such critical software service becomes unstable then it's time to migrate? Thinking about self-hosting GitLab.

embedding-shape 5mo ago
> but if such critical software service becomes unstable then it's time to migrate? Right way to think about this: > If things we need/see as critical for our work are hosted on a platform with really bad reliability, it's time for us to migrate My internet connection at home is really shit, and almost every week there is a multi-hour downtime for some reason, not to mention when La Liga games are on TV anything using Cloudflare is unavailable, so I've had to spend extra energy and time to setup things in a way so I can still work whenever this happens.

throwatdem12311 5mo ago

> The main driver is a rapid change in how software is being built. Leopard, meet face. Too little too late, yesterday was the straw that broke the camel’s back for us and we’ve started a migration to a self-hosted GitLab.

lousken 5mo ago

Availability is priority? Does not seem like it is https://mrshu.github.io/github-statuses/ https://mrshu.github.io/github-statuses/

eolgun 5mo ago

The AI agent growth explanation is interesting but also a bit of a deflection. If a meaningful portion of your traffic is now automated agents, your capacity planning model is fundamentally different, you're no longer scaling for human paced workflows but for burst patterns that look nothing like historical load. The unlabeled graphs don't help the credibility case. When you are already in the hole on trust, shipping a post that requires readers to assume favorable baselines is exactly the wrong move.

jameskilton 5mo ago

Nice, they have availability numbers now on their status page, but they aren't aggregating. If you multiply all current numbers together (as of Apr 28), you find out that GitHub has a 97.26% uptime. One ... single ... 9. They can do better.

embedding-shape 5mo ago
Kind of unfair though, do the same for any platform with multiple services and you'd probably get <99% for most of them. > you find out that GitHub has a 97.26% uptime Calculating that to "Downtime per day" you get ~40 minutes of downtime per day, almost a week per year. Crazy stuff for something essential like this.

deleted 5mo ago

[deleted]

Waterluvian 5mo ago

I have a hard time believing anything what's said in a blog post where a graph lacks axes labels/scale. It tells me that nobody who cares about correctness had any say on the content of the post. Maybe I'm being 8am cranky and pedantic, but I'm sticking with it. > availability first, then capacity, then new features. I'd love to experience first-hand a leadership team who says, "stop accepting new paying customers until we've got availability sorted out!"

madeofpalk 5mo ago
Like they did with Copilot last week? https://github.blog/news-insights/company-news/changes-to-github-copilot-individual-plans/ https://github.blog/news-insights/company-news/changes-to-gi... > New sign-ups for GitHub Copilot Pro, Pro+, and Student plans are paused. Pausing sign-ups allows us to serve existing customers more effectively.

GS_Projects 5mo ago

The bit nobody covers in these write-ups: small teams without dual-cloud failover budget. Last big GitHub outage cost me a deploy day. Not catastrophic but the kind of thing you don't budget for when GitHub is your single source of truth. Status page is also still doing that thing where every component is green but in practice clone is hanging, push is timing out, actions are stuck. Per-service uptime is a managed number. The user-experience number is the one that matters and it's not in the post-mortem.

AlexeyBelov 5mo ago
No LLM comments.

jimmypk 5mo ago

[flagged]

dzonga 5mo ago

blame MySQL. Blame Ruby. on another note - is the exponential growth from 'agentic' workflows actually resulting in productive software in the wild. Or it is just noise. On my end I haven't seen the software I use getting better.

dangoodmanUT 5mo ago

Two incidents? Just two? In seriousness, looking at their scale, this is an insane engineering challenge. Especially if they’re moving databases, not easy ever, and certainly not at that scale

mendyberger 5mo ago

I wonder if this mess has anything to do with talent loss resulting from layoffs after the pandemic

pointlessone 5mo ago
I’d guess it has much more to do with the extra load agentic ai generates. If we take the charts in the OP at face value, do you think gh suddenly exploded in popularity? At this point I think almost everyone who has any use for gh already has an account and use it as much as they ever would. But all the charts go to the moon. Gh obviously didn’t take into account that ai agents can generate a lot of activity they don’t have capacity for.

OutOfHere 5mo ago

> we accelerated parts of migrating performance or scale sensitive code out of Ruby monolith into Go. I am surprised that Microsoft is allowed to use Go. How long will it be before a bean counter forces a rewrite to a Microsoft favored language?

senderista 5mo ago
They used Go for the new TypeScript compiler!

agluszak 5mo ago

Regarding their image with stats (https://github.blog/wp-content/uploads/2026/04/record-accelleration-1920x1080-2.png https://github.blog/wp-content/uploads/2026/04/record-accell...) - what exactly are the ranges on y-axes? I doubt they had close to 0 PRs merged in 2023 ;)

000ooo000 5mo ago

Load from paying customers vs. load from nonpaying users would be interesting to know. No doubt omitted deliberately.

clvx 5mo ago

With this prioritization Github IPv6 support is gonna happen the next decade.

snihalani 5mo ago
IPv6 doesn't sound like a huge lift at the entrypoints. Internal networking to IPv6 only sounds like an impossible lift

perbu 5mo ago

fwiw, I've had good luck scaling git, specially doing clones, in the HTTP layer, using Varnish. this was CI bringing Github Enterprise to it's knees.

JimmaDaRustla 5mo ago

AS IF THEY POST THIS WHILE THEIR SEARCH IS BROKEN, what a circus

pier25 5mo ago

Github has been having availability issues for years now.

zinodaur 5mo ago

> posts graphs without way to determine scale of y axis Now that’s the kind of excellence I expect from the GitHub engineering team

BigTTYGothGF 5mo ago

LLMs and vibe coding ruining it for the rest of us.

devmor 5mo ago

Microsoft has been an abysmal steward of Github - the few nice features it has over self-hosting just aren't worth losing an hour or more of CI/CD downtime during daylight hours every week. Yesterday was the last straw for me - I've begun migrating my personal private projects and my contracting firm's projects off of github.

mrhottakes 5mo ago

LLMs have helped us invent websites that only work sometimes. We're truly living in the future.

zamalek 5mo ago

> Our priorities are clear: availability first, then capacity, then new features. No mention of Copilot/slopiffication. Probably an intentional omission as Microsoft only has one true priority across all of its products.

saghm 5mo ago

Given what "An Update on <XYZ>" usually means, I can only assume this means that Github has decided to no longer provide availability. Not particularly surprising given current trends I guess

dangus 5mo ago

Notice how the graphs have no Y axis. That's how you know it's manipulative. This company is owned by one of the major causes of the AI boom and is hiding behind difficulty scaling, despite its parent company also being a premier source of scaling solutions. GitHub: don't gaslight your customers. It is not your customers' problem that you're having trouble scaling. Nobody cares. Give us the service we are paying you for and make it reliable, or else we'll choose something else. After the words "Both of those incidents are not acceptable" the blog post should have been over. Nobody needs to hear a sob story about how your service is too popular.

danra 5mo ago

When it's down to brass tacks, the most common GitHub action, actions/checkout, is not taking contributions due to "focus [...] on strategic areas" [0] despite having years-old issues - here's one[1] that soon celebrates its sixth birthday, despite having an available PR! [0] https://github.com/actions/checkout#note https://github.com/actions/checkout#note [1] https://github.com/actions/checkout/issues/270#issue-628967704 https://github.com/actions/checkout/issues/270#issue-6289677...

bartread 5mo ago

> I wanted to give an update on GitHub’s availability in light of two recent incidents. [Emphasis mine] Vlad, you are living in a very different world to me. GitHub has suffered dozens and dozens of outages since the beginning of the year. It is notably less available and reliable than it was even as recently as last year. People have created dashboards and heatmaps showing how bad GitHub has become. At least one of those has made it to the front page of Hacker News. In fact its unreliability and persistent availability issues have become a frequent topic of conversation across sites and communities frequented its users - of which HN and Reddit are two obvious examples. At this point GitHub's unreliability risks becoming a meme, if it hasn't already done so. The only thing your post makes clear is that your priorities ARE NOT clear. > Our priorities are clear: availability first, then capacity, then new features. WRONG! Your priorities are: 1. Availability 2. Availability 3. Availability You have NO OTHER PRIORITIES. If you want other priorities, focus on AVAILABILITY for 6 months and then come back and we can all have a serious conversation about something else. In the meantime, you need to understand that GitHub's reliability over months and months - not just in April - has been completely unacceptable. Focus on fixing that and on nothing else.

dminik 5mo ago
I've recently built a script that periodically (every 25 minutes) fetches the latest merged PRs to check for some potential rule violations. I'm not an admin and couldn't get the events API working, so I just resorted to polling. On an average ~8 hour working day, there's at least one failed request. In fact, looking over the logs, I can't spot a single day that did not have a failed request. Now, I can't guarantee that these are all caused by GitHub (as opposed to my connection), but it is pretty funny.
dude250711 5mo ago
Microsoft board and shareholders: "LOL, nah! More vibe coding inside and outside plz".
baobun 5mo ago
Wow. Security or trust not even making the list.

gamerslexus 5mo ago

> The main driver is a rapid change in how software is being built. Since the second half of December 2025, agentic development workflows have accelerated sharply. So, it's because of LLMs guys.

russellthehippo 5mo ago

Reading the capacity crunch idea made me a little more empathetic to their issues - 30x in one year is a lot when you're starting from a high baseline. Now that being said...I'd really appreciate more availability.

mw888 5mo ago

Their ostensible troubles are fueled by "exponential usage growth", demonstrated by three graphs which exclude axis labels and are aggressively cropped.

twobitshifter 5mo ago

let’s do git without github again

init_31 5mo ago

GitHub wants everyone to use AI agents but if you try to register a new account today you are greeted by an endless gauntlet of absolutely ridiculous captchas that reject you because they think you're an AI agent and nobody should be using AI agents on GitHub.

chanux 5mo ago

I have a feeling that this post is good enough reassurance for big money corps. Long live Github under MS!

JackSlateur 5mo ago

I do not understand how github can still have issues .. I mean: obviously, in the old world, it would have taken lot of work to improve the situation. But now, with AI, just plug copilot against the code base and everything is fixed in a week, no ?