5 ms·

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then the

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figured out, but a little sympathy goes a long way. I personally wish them the best and hope their on-call people can go back to getting normal amounts of sleep soon.

herpdyderp 1mo ago
> I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. I don't think this is commendable at all. I give GitHub a lot of money and I'm tired of it being wasted with downtime.
- jjice 1mo ago
  It's a shame that the GitHub org that we use at my job that we pay a lot of money for gets affected the same way my personal nonsense does. I don't know the architecture or any of that, but I feel like there could be (and it's not like they would've really known this until the last year or two with the massive spike) separate infrastructure for paid users/orgs vs free the same way they make the distinction with enterprise. I get the massive load changes that they are under over the last two years, but why does a bunch of vibe coded slop take down the same resources that my company pays for every single month and has for years? I imagine properly splitting that out would be an absolute headache and not worthwhile for them vs stabilizing the rest of the service, but damn it sucks when I get blocked at work because GH is down.
  - tempest_ 1mo ago
    Our GitlabCE instance has been sitting in the racks for nearly two years with almost 100% uptime running on 10 year old xeons that have long since paid for themselves. Of course it is not free of all management but for our use case it is working. There are hiccups with the CI runners from time time but nothing major and we have another machine in another rack that serves as a backup which can be brought up ~< 20 minutes. I know companies have long since tossed their expertise for hosting their own stuff in favour of SaaS but at some point its hard to beat the up time of a single machine.
    - Uvix 1mo ago
      We have the expertise. We don’t have the interest in fighting our internal “security” teams to keep running VMs.
      - snovv_crash 1mo ago
        Potato potato. Expertise isn't just "Do we have a sysadmin?", it's also "Does the org enable them to get their job done?".
        a012 1mo ago
        You are missing the compliance. Enterprise is mostly compliance, making job done is just second
  - vel0city 1mo ago
    You can pay for GitHub Enterprise Cloud which is essentially their private GitHub deployment stack that they'll host for you. It's theoretically isolated, supposed to be completely separate from these outages. It's a lot more expensive and has a good bit more limitations to the regular SaaS product.
    - herpdyderp 1mo ago
      As a subscriber to GitHub Enterprise Cloud I assure you that it is not isolated from outages, not in any meaningful way that I have ever noticed.
padjo 1mo ago
I don't typically have sympathy for businesses that fail to deliver a service as advertised. I can have sympathy for the humans caught in the crossfire but only managing one nine of availability on a commercial service is not acceptable.
gogobio 1mo ago
Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.
- __MatrixMan__ 1mo ago
  Sympathy negated by their 2026 Pwnie award: "Lamest Vendor" https://this.weekinsecurity.com/microsoft-wins-lamest-vendor-at-pwnie-awards-2026-for-threatening-security-researchers-with-legal-action/ https://this.weekinsecurity.com/microsoft-wins-lamest-vendor...
- javier2 1mo ago
  and it has been so damn slow for a number of years now, it drives me insane.
- zvmaz 1mo ago
  > Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime. I gather that you have intimate and deep knowledge on the teams and the problems they try to solve there.
  - californical 1mo ago
    If I own a restaurant and buy bread from a supplier — BreadHub. And 99% of the bread that I get is good but 1% of the loaves, they forgot to add flour. Consistently, for years, they always have loaves missing a key ingredient that I still end up paying for. I can be pretty sure that BreadHub have a pretty major internal issue, and should probably be questioning their competence, regardless of their “scale”, and without any knowledge of the “problems they’re solving”
    - tyre 1mo ago
      I’m not saying users don’t have a right to be pissed or aren’t justified in looking at other options. I’m saying that GH is operating at a huge scale with (probably) lots of technical debt and a forced migration to new infrastructure. I would be (and am) highly critical of leadership. I’m not going to make strong assertions about ICs without knowing their context. I’ve worked at a company with a sterling reputation for engineering excellence where brilliant ICs were kneecapped by poor leadership. I think a lot of us, at one point or another in our careers, have worked with potato leadership that can be short-sighted or political. It isn’t a comment on the engineers.
mronetwo 1mo ago
We have a business relationship so no we shouldn’t have any sympathy. They sell a service and they’re failing to provide it.
- amelius 1mo ago
  Why? If we have sympathy for Apple then we can have sympathy for Microsoft ...
  - mirashii 1mo ago
    You're the only mention of Apple in this thread, and I don't see what they have to do with it.
  - pimeys 1mo ago
    Wait, I don't sympathize Apple at all... Or any other American corporation.
  - itemize123 1mo ago
    One, logically, your statement doesn't follow. They're two different companies that have primarily in different market segment. Two, it's not true in the first place - did we side with apple and its cut of the app store revenue for instance?
JDups 1mo ago
Constant building with AI is something that they (Microsoft) promote and are heavily invested in.
- chris_money202 1mo ago
  And then get penalized for in the market when they say we need to build more DCs and spend huge capex... which is what GH needs right now
dpz 1mo ago
Maybe for a free account. But we pay enterprise license and GitHub is a big dependency in our software flow. If this continues to be a problem as an enterprise product they need to do something. Otherwise theyre are going to to start losing business
theideaofcoffee 1mo ago
Give me a break. Sympathy? For microsoft? That might have flown when github was like seven people, but they have nearly unlimited resources to make it better. They're just choosing not to. Let's talk contracts and money before we pull the sympathy card. I used to be on-call in a high-traffic environment where single customers pushed more bits than entire nations. I chose the role. I didn't want people's sympathy, if anything, I wanted them to complain to management. If it gets too bad they can quit. Maybe that would be for the best, just wear the thing down until it outright fails and no one wants to touch it. One less bullshit service sucking all of the oxygen out.
niltecedu 1mo ago
idk man, we pay stupid amounts of money to microsoft, we are an enterprise customer, expecting better availablity compared to my laptop isnt really a high bar.
jraph 1mo ago
They did that to themselves, and doubly so. They decided they needed to capture the whole open source ecosystem by turning open source work into social networking... on a proprietary platform (because open source is great, especially when it's others' software). That was before they joined Microsoft. And then Microsoft pushed AI everywhere, including on GitHub itself with copilot. I would have liked if they had left the open source projects alone and didn't create that FOMO for not using them. I have no sympathy.
- kstrauser 1mo ago
  > And then Microsoft pushed AI everywhere, including on GitHub itself with copilot. Ding ding ding, we have a winner. I like AI. I work for an AI company. Still, Microsoft aggressively pushed GitHub users toward Copilot. They don't get to do that and complain about increased volume from AI-generated changes. No Copilot + reasonable operation: the way things were Copilot + reasonable operation: Well done! No Copilot + being overwhelmed by AI commits: Sympathy. Copilot + being overwhelmed by AI commits: "Where did that petard come from that's hoisting us?"
bamboozled 1mo ago
No, we don’t.
VCFundedGenYer 1mo ago
"Leave the big trillion dollar corporation alone" is not the right idea.
mcmcmc 1mo ago
It’s commendable that they let people give them training data for free?
sandeepkd 1mo ago
I think somewhere down the line you have an answer to problem itself. Its for a while that companies are not into the business of sympathies, we should probably not get into that side for the topic. 1. Github has enterprise users who paid for the service, their day job requires Github to be available and working 2. Github has generous free tier which is the one which is exploring a lot more with the AI generated code. It is a complexity in itself but the traffic should have been separated, the free users should not be allowed to bring down Github for enterprise customers (Just to clarify, I am free user myself). And if they do not have capacity it would have been perfectly fine to push back or throttle new users/repositories.
- IcyWindows 1mo ago
  Do we have a breakdown on how much traffic is free users vs paid users?
  - reassess_blind 1mo ago
    I’d be surprised if paid traffic was more than 1%.
- tyre 1mo ago
  I agree that they should reserve capacity for enterprises and paying users. But I think they’re in a tough spot. GitHub has historically been a huge supporter, proponent, and provider for open source projects. Engineers are difficult customers, to say the least, and the community would likely freak tf out of the segmented traffic. The logical, pragmatic, and justifiable answer doesn’t always align with your market.
  - sandeepkd 1mo ago
    Not sure if I would agree to that completely. Free tier is always ripe for abuse so in the beginning for a long time, Github has this requirement that you can only have a public repository if you are a non-paying user. Somewhere down the road when machine learning picked up (everyone was on the band wagon of hoarding the data, datalakes, hadoop, flink were at the top) they relaxed that condition allowing one to have a private repository on the free tier. They are in a tough spot cause they want to continuously ingest all the data to train their LLMs. Any fork or decentralization at large scale of git is going to impact the training pipeline.
0xbadcafebee 1mo ago
Our company pays for GitHub. We're paying for a broken product that stops our work. I don't have sympathy for the companies whom I pay for a product and give me broken shit in return. This is entirely preventable and their own fault. A restaurant makes pizzas. They suddenly get 100x more popular. They can't make 100x more pizzas. But they are still taking orders from 100x more people. Not only are they not getting enough pizzas delivered that they took orders for, but in their rush to make and deliver more pizzas, they set the kitchen on fire, which makes an even longer wait for pizzas. When the pizza you ordered doesn't get delivered, do you have sympathy for the restaurant? Or do you tell them to stop taking orders they can't fill and try not to set the kitchen on fire? Now consider the pizza restaurant has 21 billion dollars in cash, is taking your money, and not giving you pizza.
bluerooibos 1mo ago
> have a little more sympathy for the billion dollar company No. Their leadership went all in on AI and in the last blog post essentially admitted some missing test coverage for a critical path. Time to learn lessons and fix your vibe coded shitslop and stop using "user graph go up" as some kind of excuse.
alwa 1mo ago
For that matter, I admire their relative transparency about their incidents. I can think of other big players who will cheerfully show green statuses across the board while everyone can see that their pants are down...
Aeolun 1mo ago
Yeah, all those 1100 commits per second. Like, is this the standard for 'scale of traffic' that we're impressed by these days? And it's not like this snuck up on them. The graph shows a very consistent trend that gave them months and months to respond.
rgbrenner 1mo ago
Microsoft made $134B in net profit last year, and is currently valued at nearly $4T. They have the resources to solve this if they wanted to.
- chris_money202 1mo ago
  Both Microsoft and GH issue is capacity which is being solved with huge capex in DC investment, you can't create a new DC overnight and its even harder in the current climate where everyone assumes the DC is for AI and not just normal workload compute like what GH needs...
minaguib 1mo ago
Paying enterprise clients who can not push/pull code and collaborate on PRs (table-stakes) disagree with having a little more sympathy.
loloquwowndueo 1mo ago
I’d have sympathy for the pre-Microsoft GitHub. For the thing they are now, backed by a top-5 company with trillions of dollars? Cry me a river.
aduwah 1mo ago
Depends who you are aiming sympathy at. C-levels or Engineering?
madrox 1mo ago
I have a lot of sympathy for the people working at GitHub trying to keep it going, but I do not have sympathy for GitHub. That's what they're paid for. I have things to get done, and it has no sympathy for me.
rpearl 1mo ago
If you do the math, their numbers are something like 1000 commits per second. That's not pushes per second either, it's just the total number of commits summed up over a week, not all of which trigger much of anything to happen. Yes, that is scale. And yes, that's not actual requests per second. But it's the sort of scale that big (and even mid-sized) tech has known how to deal with for decades. Microsoft doesn't have an excuse.
- vrosas 1mo ago
  Their excuse is called Azure and if you’ve ever had to deal with it you’d know it’s an amazing(ly awful) excuse to have.
  - DANmode 1mo ago
    Free GitHub, let the engineers choose their own infra. Bit of a Clownstrike this has become!
ChrisMarshallNY 1mo ago
I strongly suspect that quite a few HN members are busy creating agents that are furiously stuffing GH. As I said before (and it was not a popular comment), it's easy to be an "armchair quarterback," with these services, as I think the brittleness was already baked in, and just waiting for the right time to crack. The only true way to have a robust platform, is to design something that will scale, from the start, and many startups don't do that, because they are feverishly trying to get out an MVP; even if it is a mess of bubblegum and baling wire. They always say "We'll get it done right, once we get funding," but that never happens. But my sympathy is limited by the fact that MS paid a lot of money for this, quite a while ago, and that is one company that knows all about issues of scale. They should have seen this coming.
BigTTYGothGF 1mo ago
> I think we need to have a little more sympathy for GitHub We certainly do not.
xyst 1mo ago
"sympathy" for one of the worst corporations in history is some insane corp bootlicking logic. This is a multi billion dollar corporation. History of robbing and stealing from others of their labor or IP. History of enshittifying once great services.
anonymousab 1mo ago
> they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. That is an active choice they are making, over and instead of any reliability for the rest of their users. It is a problem they are embracing, and actively encouraging, for themselves.
bmitc 1mo ago
GitHub literally went all in on doing almost nothing but AI-related feature additions for Copilot. So they built their own prison. I don't have sympathy for them. They have disrupted my work so much in the past month that it's ridiculous, and my company likely pays them millions of dollars. And you can see from this site's plotting that there is a trend towards more frequent and more critical outages. It has been extremely bad the past month.
fragmede 1mo ago
The question is, do we think Google would have the same problems running the service? (I'm biased from having worked there tho.)
- chanux 1mo ago
  I realized they killed Google Cloud Source Repositories and introduced some new product in true Google fashion. However, YouTube did well where Google Videos did not. Who knows what it would be like if big G bought GH.
evilfred 1mo ago
lol they are Microsoft, and the AI stuff has only multiplied usage like 2x, they haven't had sudden exponential growth
chanux 1mo ago
They probably have bad PR because of huge AI push, Sizeable layoffs etc. It's probably not fair to the people who are left but alas, this is apparently how the world works.
stevejb 1mo ago
I don't know enough about Github or Microsoft to say anything intelligent, but I agree with your sentiment in general. And, one thing is for sure - I have used Github for years, pushing many thousands of commits on personal, private projects. I've burned through as many free action minutes as I can, and I use my own action runners as well. I've done this all for free, so on balance I am grateful for them.
flohofwoe 1mo ago
> they're not limiting access to the site or (intentionally) throttling newcomers They can simply throttle access for everyone to a push/merge frequency which was normal for human-maintained projects. They should look at how MMOs battle cheating/botting and apply similar countermeasures (it's actually quite absurd that MMOs which are actively attacked every minute have a better uptimes than Github, and on Github the problems are even self-inflicted, not caused by external attacks). But of course throttling bots to human-level push/merge frequency wouldn't be compatible with the Github leadership's 'AI bullishness'.
adithyassekhar 1mo ago
I’ve noticed there is 5000 api call limit on gh cli. Haven’t looked at whether it was already there or when does it reset etc. good for them finally if they weren’t already doing that.
crote 1mo ago
Absolutely not. This is 100% self-inflicted. What we are seeing here is a trillion-dollar company screwing over its existing well-established customer base in a desperate attempt to get some crumbs of the current hype cycle. GitHub isn't "dealing with" scaling issues, they are intentionally causing it by shoveling giant piles of coal onto the AI fire. GitHub's issues would be over tomorrow if they were to put reasonable rate limits on their services. Stop the bots and GitHub is no longer permanently overloaded. It is an intentional business decision not to do this, and instead welcome bots with open arms - to the detriment of their existing paying customers. I do have some sympathy for the low-level GitHub staff who have to deal with this - especially those who aren't being compensated with insane Bay Area salaries.
noir_lord 1mo ago
> You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. How about no? They are owned by a company that is neck deep in foisting AI into everything, AI that a lot of us don't want in everything, they are reaping what they seem determined to sow. So it goes.
overfeed 1mo ago
> ...they're dealing with now that everyone is constantly building and pushing with AI. Also known as the AI chickens coming home to roost. Microsoft was the first to use code uploaded to GitHub to train AI models.
whateveracct 1mo ago
isn't github scaling pretty embarrassingly parallel tho? doesn't "ready to scale" at any "good" tech company mean "we can handle 10x without architectural changes?" that's the mentality i was taught at faang
anon7000 1mo ago
I appreciate it’s tough for the SREs, but GH as a business assumes they must just fall over and keep scaling. What if they figured out what their platform can actually handle, provided the contractual level of service to their paying customers first, and put some god damn limits on usage. Like they should just stop signups if they can’t handle the capacity yet. It’s a business decision, not a technical decision, to decide to screw over customers instead of adding limits based on your actual capacity.
KronisLV 1mo ago
> You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. Does nobody use GitLab or Bitbucket or whatever? Why aren't those full of high profile outages as much? > I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. I'd kind of expect that for all non-paid members will get rate limited in some ways, sooner or later. Same as what happened with DockerHub, they're probably just trying to hold out due to the reputational hit such a change would carry.
olivewong 1mo ago
also this site and the news's measure of github's single 9 availability (1 service at all down even copilot) is a bit harsh https://mrshu.github.io/github-statuses/ https://mrshu.github.io/github-statuses/

herpdyderp 1mo ago

> I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. I don't think this is commendable at all. I give GitHub a lot of money and I'm tired of it being wasted with downtime.

jjice 1mo ago
It's a shame that the GitHub org that we use at my job that we pay a lot of money for gets affected the same way my personal nonsense does. I don't know the architecture or any of that, but I feel like there could be (and it's not like they would've really known this until the last year or two with the massive spike) separate infrastructure for paid users/orgs vs free the same way they make the distinction with enterprise. I get the massive load changes that they are under over the last two years, but why does a bunch of vibe coded slop take down the same resources that my company pays for every single month and has for years? I imagine properly splitting that out would be an absolute headache and not worthwhile for them vs stabilizing the rest of the service, but damn it sucks when I get blocked at work because GH is down.
- tempest_ 1mo ago
  Our GitlabCE instance has been sitting in the racks for nearly two years with almost 100% uptime running on 10 year old xeons that have long since paid for themselves. Of course it is not free of all management but for our use case it is working. There are hiccups with the CI runners from time time but nothing major and we have another machine in another rack that serves as a backup which can be brought up ~< 20 minutes. I know companies have long since tossed their expertise for hosting their own stuff in favour of SaaS but at some point its hard to beat the up time of a single machine.
  - Uvix 1mo ago
    We have the expertise. We don’t have the interest in fighting our internal “security” teams to keep running VMs.
    - snovv_crash 1mo ago
      Potato potato. Expertise isn't just "Do we have a sysadmin?", it's also "Does the org enable them to get their job done?".
      - a012 1mo ago
        You are missing the compliance. Enterprise is mostly compliance, making job done is just second
- vel0city 1mo ago
  You can pay for GitHub Enterprise Cloud which is essentially their private GitHub deployment stack that they'll host for you. It's theoretically isolated, supposed to be completely separate from these outages. It's a lot more expensive and has a good bit more limitations to the regular SaaS product.
  - herpdyderp 1mo ago
    As a subscriber to GitHub Enterprise Cloud I assure you that it is not isolated from outages, not in any meaningful way that I have ever noticed.

padjo 1mo ago

I don't typically have sympathy for businesses that fail to deliver a service as advertised. I can have sympathy for the humans caught in the crossfire but only managing one nine of availability on a commercial service is not acceptable.

gogobio 1mo ago

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

__MatrixMan__ 1mo ago
Sympathy negated by their 2026 Pwnie award: "Lamest Vendor" https://this.weekinsecurity.com/microsoft-wins-lamest-vendor-at-pwnie-awards-2026-for-threatening-security-researchers-with-legal-action/ https://this.weekinsecurity.com/microsoft-wins-lamest-vendor...
javier2 1mo ago
and it has been so damn slow for a number of years now, it drives me insane.
zvmaz 1mo ago
> Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime. I gather that you have intimate and deep knowledge on the teams and the problems they try to solve there.
- californical 1mo ago
  If I own a restaurant and buy bread from a supplier — BreadHub. And 99% of the bread that I get is good but 1% of the loaves, they forgot to add flour. Consistently, for years, they always have loaves missing a key ingredient that I still end up paying for. I can be pretty sure that BreadHub have a pretty major internal issue, and should probably be questioning their competence, regardless of their “scale”, and without any knowledge of the “problems they’re solving”
  - tyre 1mo ago
    I’m not saying users don’t have a right to be pissed or aren’t justified in looking at other options. I’m saying that GH is operating at a huge scale with (probably) lots of technical debt and a forced migration to new infrastructure. I would be (and am) highly critical of leadership. I’m not going to make strong assertions about ICs without knowing their context. I’ve worked at a company with a sterling reputation for engineering excellence where brilliant ICs were kneecapped by poor leadership. I think a lot of us, at one point or another in our careers, have worked with potato leadership that can be short-sighted or political. It isn’t a comment on the engineers.

mronetwo 1mo ago

We have a business relationship so no we shouldn’t have any sympathy. They sell a service and they’re failing to provide it.

amelius 1mo ago
Why? If we have sympathy for Apple then we can have sympathy for Microsoft ...
- mirashii 1mo ago
  You're the only mention of Apple in this thread, and I don't see what they have to do with it.
- pimeys 1mo ago
  Wait, I don't sympathize Apple at all... Or any other American corporation.
- itemize123 1mo ago
  One, logically, your statement doesn't follow. They're two different companies that have primarily in different market segment. Two, it's not true in the first place - did we side with apple and its cut of the app store revenue for instance?

JDups 1mo ago

Constant building with AI is something that they (Microsoft) promote and are heavily invested in.

chris_money202 1mo ago
And then get penalized for in the market when they say we need to build more DCs and spend huge capex... which is what GH needs right now

dpz 1mo ago

Maybe for a free account. But we pay enterprise license and GitHub is a big dependency in our software flow. If this continues to be a problem as an enterprise product they need to do something. Otherwise theyre are going to to start losing business

theideaofcoffee 1mo ago

Give me a break. Sympathy? For microsoft? That might have flown when github was like seven people, but they have nearly unlimited resources to make it better. They're just choosing not to. Let's talk contracts and money before we pull the sympathy card. I used to be on-call in a high-traffic environment where single customers pushed more bits than entire nations. I chose the role. I didn't want people's sympathy, if anything, I wanted them to complain to management. If it gets too bad they can quit. Maybe that would be for the best, just wear the thing down until it outright fails and no one wants to touch it. One less bullshit service sucking all of the oxygen out.

niltecedu 1mo ago

idk man, we pay stupid amounts of money to microsoft, we are an enterprise customer, expecting better availablity compared to my laptop isnt really a high bar.

jraph 1mo ago

They did that to themselves, and doubly so. They decided they needed to capture the whole open source ecosystem by turning open source work into social networking... on a proprietary platform (because open source is great, especially when it's others' software). That was before they joined Microsoft. And then Microsoft pushed AI everywhere, including on GitHub itself with copilot. I would have liked if they had left the open source projects alone and didn't create that FOMO for not using them. I have no sympathy.

kstrauser 1mo ago
> And then Microsoft pushed AI everywhere, including on GitHub itself with copilot. Ding ding ding, we have a winner. I like AI. I work for an AI company. Still, Microsoft aggressively pushed GitHub users toward Copilot. They don't get to do that and complain about increased volume from AI-generated changes. No Copilot + reasonable operation: the way things were Copilot + reasonable operation: Well done! No Copilot + being overwhelmed by AI commits: Sympathy. Copilot + being overwhelmed by AI commits: "Where did that petard come from that's hoisting us?"

bamboozled 1mo ago

No, we don’t.

VCFundedGenYer 1mo ago

"Leave the big trillion dollar corporation alone" is not the right idea.

mcmcmc 1mo ago

It’s commendable that they let people give them training data for free?

sandeepkd 1mo ago

I think somewhere down the line you have an answer to problem itself. Its for a while that companies are not into the business of sympathies, we should probably not get into that side for the topic. 1. Github has enterprise users who paid for the service, their day job requires Github to be available and working 2. Github has generous free tier which is the one which is exploring a lot more with the AI generated code. It is a complexity in itself but the traffic should have been separated, the free users should not be allowed to bring down Github for enterprise customers (Just to clarify, I am free user myself). And if they do not have capacity it would have been perfectly fine to push back or throttle new users/repositories.

IcyWindows 1mo ago
Do we have a breakdown on how much traffic is free users vs paid users?
- reassess_blind 1mo ago
  I’d be surprised if paid traffic was more than 1%.
tyre 1mo ago
I agree that they should reserve capacity for enterprises and paying users. But I think they’re in a tough spot. GitHub has historically been a huge supporter, proponent, and provider for open source projects. Engineers are difficult customers, to say the least, and the community would likely freak tf out of the segmented traffic. The logical, pragmatic, and justifiable answer doesn’t always align with your market.
- sandeepkd 1mo ago
  Not sure if I would agree to that completely. Free tier is always ripe for abuse so in the beginning for a long time, Github has this requirement that you can only have a public repository if you are a non-paying user. Somewhere down the road when machine learning picked up (everyone was on the band wagon of hoarding the data, datalakes, hadoop, flink were at the top) they relaxed that condition allowing one to have a private repository on the free tier. They are in a tough spot cause they want to continuously ingest all the data to train their LLMs. Any fork or decentralization at large scale of git is going to impact the training pipeline.

0xbadcafebee 1mo ago

Our company pays for GitHub. We're paying for a broken product that stops our work. I don't have sympathy for the companies whom I pay for a product and give me broken shit in return. This is entirely preventable and their own fault. A restaurant makes pizzas. They suddenly get 100x more popular. They can't make 100x more pizzas. But they are still taking orders from 100x more people. Not only are they not getting enough pizzas delivered that they took orders for, but in their rush to make and deliver more pizzas, they set the kitchen on fire, which makes an even longer wait for pizzas. When the pizza you ordered doesn't get delivered, do you have sympathy for the restaurant? Or do you tell them to stop taking orders they can't fill and try not to set the kitchen on fire? Now consider the pizza restaurant has 21 billion dollars in cash, is taking your money, and not giving you pizza.

bluerooibos 1mo ago

> have a little more sympathy for the billion dollar company No. Their leadership went all in on AI and in the last blog post essentially admitted some missing test coverage for a critical path. Time to learn lessons and fix your vibe coded shitslop and stop using "user graph go up" as some kind of excuse.

alwa 1mo ago

For that matter, I admire their relative transparency about their incidents. I can think of other big players who will cheerfully show green statuses across the board while everyone can see that their pants are down...

Aeolun 1mo ago

Yeah, all those 1100 commits per second. Like, is this the standard for 'scale of traffic' that we're impressed by these days? And it's not like this snuck up on them. The graph shows a very consistent trend that gave them months and months to respond.

rgbrenner 1mo ago

Microsoft made $134B in net profit last year, and is currently valued at nearly $4T. They have the resources to solve this if they wanted to.

chris_money202 1mo ago
Both Microsoft and GH issue is capacity which is being solved with huge capex in DC investment, you can't create a new DC overnight and its even harder in the current climate where everyone assumes the DC is for AI and not just normal workload compute like what GH needs...

minaguib 1mo ago

Paying enterprise clients who can not push/pull code and collaborate on PRs (table-stakes) disagree with having a little more sympathy.

loloquwowndueo 1mo ago

I’d have sympathy for the pre-Microsoft GitHub. For the thing they are now, backed by a top-5 company with trillions of dollars? Cry me a river.

aduwah 1mo ago

Depends who you are aiming sympathy at. C-levels or Engineering?

madrox 1mo ago

I have a lot of sympathy for the people working at GitHub trying to keep it going, but I do not have sympathy for GitHub. That's what they're paid for. I have things to get done, and it has no sympathy for me.

rpearl 1mo ago

If you do the math, their numbers are something like 1000 commits per second. That's not pushes per second either, it's just the total number of commits summed up over a week, not all of which trigger much of anything to happen. Yes, that is scale. And yes, that's not actual requests per second. But it's the sort of scale that big (and even mid-sized) tech has known how to deal with for decades. Microsoft doesn't have an excuse.

vrosas 1mo ago
Their excuse is called Azure and if you’ve ever had to deal with it you’d know it’s an amazing(ly awful) excuse to have.
- DANmode 1mo ago
  Free GitHub, let the engineers choose their own infra. Bit of a Clownstrike this has become!

ChrisMarshallNY 1mo ago

I strongly suspect that quite a few HN members are busy creating agents that are furiously stuffing GH. As I said before (and it was not a popular comment), it's easy to be an "armchair quarterback," with these services, as I think the brittleness was already baked in, and just waiting for the right time to crack. The only true way to have a robust platform, is to design something that will scale, from the start, and many startups don't do that, because they are feverishly trying to get out an MVP; even if it is a mess of bubblegum and baling wire. They always say "We'll get it done right, once we get funding," but that never happens. But my sympathy is limited by the fact that MS paid a lot of money for this, quite a while ago, and that is one company that knows all about issues of scale. They should have seen this coming.

BigTTYGothGF 1mo ago

> I think we need to have a little more sympathy for GitHub We certainly do not.

xyst 1mo ago

"sympathy" for one of the worst corporations in history is some insane corp bootlicking logic. This is a multi billion dollar corporation. History of robbing and stealing from others of their labor or IP. History of enshittifying once great services.

anonymousab 1mo ago

> they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. That is an active choice they are making, over and instead of any reliability for the rest of their users. It is a problem they are embracing, and actively encouraging, for themselves.

bmitc 1mo ago

GitHub literally went all in on doing almost nothing but AI-related feature additions for Copilot. So they built their own prison. I don't have sympathy for them. They have disrupted my work so much in the past month that it's ridiculous, and my company likely pays them millions of dollars. And you can see from this site's plotting that there is a trend towards more frequent and more critical outages. It has been extremely bad the past month.

fragmede 1mo ago

The question is, do we think Google would have the same problems running the service? (I'm biased from having worked there tho.)

chanux 1mo ago
I realized they killed Google Cloud Source Repositories and introduced some new product in true Google fashion. However, YouTube did well where Google Videos did not. Who knows what it would be like if big G bought GH.

evilfred 1mo ago

lol they are Microsoft, and the AI stuff has only multiplied usage like 2x, they haven't had sudden exponential growth

chanux 1mo ago

They probably have bad PR because of huge AI push, Sizeable layoffs etc. It's probably not fair to the people who are left but alas, this is apparently how the world works.

stevejb 1mo ago

I don't know enough about Github or Microsoft to say anything intelligent, but I agree with your sentiment in general. And, one thing is for sure - I have used Github for years, pushing many thousands of commits on personal, private projects. I've burned through as many free action minutes as I can, and I use my own action runners as well. I've done this all for free, so on balance I am grateful for them.

flohofwoe 1mo ago

> they're not limiting access to the site or (intentionally) throttling newcomers They can simply throttle access for everyone to a push/merge frequency which was normal for human-maintained projects. They should look at how MMOs battle cheating/botting and apply similar countermeasures (it's actually quite absurd that MMOs which are actively attacked every minute have a better uptimes than Github, and on Github the problems are even self-inflicted, not caused by external attacks). But of course throttling bots to human-level push/merge frequency wouldn't be compatible with the Github leadership's 'AI bullishness'.

adithyassekhar 1mo ago

I’ve noticed there is 5000 api call limit on gh cli. Haven’t looked at whether it was already there or when does it reset etc. good for them finally if they weren’t already doing that.

crote 1mo ago

Absolutely not. This is 100% self-inflicted. What we are seeing here is a trillion-dollar company screwing over its existing well-established customer base in a desperate attempt to get some crumbs of the current hype cycle. GitHub isn't "dealing with" scaling issues, they are intentionally causing it by shoveling giant piles of coal onto the AI fire. GitHub's issues would be over tomorrow if they were to put reasonable rate limits on their services. Stop the bots and GitHub is no longer permanently overloaded. It is an intentional business decision not to do this, and instead welcome bots with open arms - to the detriment of their existing paying customers. I do have some sympathy for the low-level GitHub staff who have to deal with this - especially those who aren't being compensated with insane Bay Area salaries.

noir_lord 1mo ago

> You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. How about no? They are owned by a company that is neck deep in foisting AI into everything, AI that a lot of us don't want in everything, they are reaping what they seem determined to sow. So it goes.

overfeed 1mo ago

> ...they're dealing with now that everyone is constantly building and pushing with AI. Also known as the AI chickens coming home to roost. Microsoft was the first to use code uploaded to GitHub to train AI models.

whateveracct 1mo ago

isn't github scaling pretty embarrassingly parallel tho? doesn't "ready to scale" at any "good" tech company mean "we can handle 10x without architectural changes?" that's the mentality i was taught at faang

anon7000 1mo ago

I appreciate it’s tough for the SREs, but GH as a business assumes they must just fall over and keep scaling. What if they figured out what their platform can actually handle, provided the contractual level of service to their paying customers first, and put some god damn limits on usage. Like they should just stop signups if they can’t handle the capacity yet. It’s a business decision, not a technical decision, to decide to screw over customers instead of adding limits based on your actual capacity.

KronisLV 1mo ago

> You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. Does nobody use GitLab or Bitbucket or whatever? Why aren't those full of high profile outages as much? > I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. I'd kind of expect that for all non-paid members will get rate limited in some ways, sooner or later. Same as what happened with DockerHub, they're probably just trying to hold out due to the reputational hit such a change would carry.

olivewong 1mo ago

also this site and the news's measure of github's single 9 availability (1 service at all down even copilot) is a bit harsh https://mrshu.github.io/github-statuses/ https://mrshu.github.io/github-statuses/