4 ms·

Github is struggling because AI-boosted coding increased the number of commits 14x in the past year, and the pace is still accelerating. The site is struggling

• TZubiri 2mo ago
  Are these disproportionately in the free tier? Or is the increased storage load similar across the paid tiers? Might be worth considering a /GB/MO fee. Let's face it, VCS is a (glorified) storage service.
  • cute_boi 2mo ago
    I think Github should start charging who goes beyond fair usages. I just checked no 1 vibe coder https://github.com/steipete https://github.com/steipete and he has like 279,440. Compared this to Taylor who works on huge project like Laravel and is only around 8k commits.
    • ProofHouse 2mo ago
      The scam that anthropic pulled, is they remove free tier (bad idea for Github honestly), but blame it on AI vibe code abusers. Instead of for example finding the 1-5% of absolutely aggregous users like Steipete and restricting them.
      • logicallee 2mo ago
        it's a bit of a stretch to call removing a free tier a scam. as you can see a lot of people in this thread are saying Github should start charging.
• pluc 2mo ago
  Nobody at Microsoft could've predicted AI was going to be so popular!
  • blipvert 2mo ago
    Like the way that they shipped ‘95 without a browser?
    • aleph_minus_one 2mo ago
      > Like the way that they shipped ‘95 without a browser? Around the time when Windows 95 was released, Microsoft did predict that the internet will become popular also among private users. That is why - they included Winsock (Windows' TCP/UDP/IP stack) as native part of Windows 95 (in opposite to Windows 3.x where you had to install a third-party implementation of the Winsock API such as Trumpet Winsock), though it was typically not installed by default, - support to use MSN (the dial-up online service and internet servicen provider [1]) was pre-installed with Windows 95. What was not clear at that time was that the world wide web would become the killer application for the internet for consumer users (at that time, the more likely killer application of the internet for consumers was seen in proprietary content providers such as AOL, Compuserve or MSN), so Microsoft provided no web browser with the original version of Windows 95. But even at that time, it was conceivable, though not clear that the world wide web could become important. So, already at the release time of Windows 95, Microsoft provided Internet Explorer 1.0 as part of the Plus! add-on pack for Windows 95. With Windows 95 OEM Service Release 1, the Internet Explorer 2.0 got part of Windows 95, and Windows 98 introduced the (optional) Active Desktop [2], in was centered around the Internet Explorer. So, in my opinion Microsoft did everything "correct" here: they provided Internet Explorer from beginning on as an optional separate component (Plus! add-on pack), and once it was obvious that the world wide web would become very important, they reacted fast and included the Internet Explorer in the next service release of Windows 95 (Windows 95 OEM Service Release 1). -- [1] https://en.wikipedia.org/wiki/MSN_Dial-Up_Internet_Access https://en.wikipedia.org/wiki/MSN_Dial-Up_Internet_Access [2] https://en.wikipedia.org/wiki/Active_Desktop https://en.wikipedia.org/wiki/Active_Desktop
• cute_boi 2mo ago
  It is just an excuse. If they can't support paid member or don't have infra they should start charging these slop projects who are committing relentlessly.
• bhouston 2mo ago
  > Github is struggling because AI-boosted coding increased the number of commits 14x in the past year, and the pace is still accelerating. Sure, but how is OpenAI, and Claude handling the increased usage? They are handling it a lot better than Github. Just because you have fast growth does not mean you can not handle it. Instead you have to realize that you are entering into a high growth phase and design around that. It has been clear that we are in a high growth phase for around a year now and Github still hasn't adapted to that properly. At some point you have to admit that Github ready for the moment.
  • traviswingo 2mo ago
    GitHub is on Azure. Azure is notorious for being terrible. There was a post here recently from an ex-Azure engineer explaining all of this in a multi-part series. Edit: Found the thread: https://news.ycombinator.com/item?id=47616242 https://news.ycombinator.com/item?id=47616242
    • Tiberium 2mo ago
      Isn't a lot of OpenAI infra also on Azure? They seem to be faring better than GitHub.
      • bhouston 2mo ago
        I bet OpenAI has failovers to other data centers. I think they have a variety of data center provides all over the place.
        • prewett 2mo ago
          There's already a noun for this: "resources"
    • moralestapia 2mo ago
      Q. how can someone publish an article like that, with their name, etc... Don't they sign NDAs? Do they not care? Are they not enforceable? Very curious to know.
• arjie 2mo ago
  The obvious answer is to end free unlimited private repos. Source is no longer a specific thing. GitHub is just a free S3 or Azure provider.
  • SoftTalker 2mo ago
    I think this would cut out a lot of the mindless commit activity. Charge $1/month per repo. If your project isn't worth that, run your own git or use another free forge.
    • odo1242 2mo ago
      I think GitHub leans on this free repos thing a lot for getting new users though. Like I could see people just moving to GitLab
      • hakunin 2mo ago
        …and encountering the same issues most likely.
        • odo1242 2mo ago
          Fair, but the point is GitHub’s model is: Free Users use GitHub => when a company needs source control => all their coders use GitHub => company buys expensive enterprise contract. If there’s less of an advantage to using free GitHub then that funnel works less effectively.
      • neko_ranger 2mo ago
        Free private repos on github is relatively new, only introduced after a rise in popularity of gitlab (which has free private repos). Seems like github is too popular for itself.
        • klausa 2mo ago
          I know you said "relatively", but it's been seven and a half years: https://github.blog/changelog/2019-01-08-pricing-changes/ https://github.blog/changelog/2019-01-08-pricing-changes/
• fourseventy 2mo ago
  Github is owned by Microsoft, its not like its some boostrapped startup who has a closet of servers that are melting.
  • platevoltage 2mo ago
    And on top of that, Microsoft is one of the companies that created this problem in the first place.
• bluerooibos 2mo ago
  > Github is struggling because AI-boosted coding increased the number of commits 14x in the past year, OK so then use the additional revenue to add more resources and employees? What the f*k are they doing over there?
  • thevinter 2mo ago
    What revenue do AI-boosted commits provide? Am I missing something?
    • bluerooibos 2mo ago
      You're missing the fact that Microsoft reported over $331 billion in total revenue and an operating income exceeding $155 billion - this is a fixable problem. If they're really that cash-strapped at GitHub, then are they planning to just continue allowing this to happen and drown, or fix the issues? This has been ongoing for over a year.
      • rurban 2mo ago
        So why did all the Azure and Github engineers leave then? There's only management and outsourced cheap labor there, and management has no idea. Cheap labor even less
  • pavel_lishin 2mo ago
    Is there additional revenue from a bunch of people pushing their vibe-coded stuff up to Github?
  • flexagoon 2mo ago
    > OK so then use the additional revenue to add more resources and employees? What revenue? I'd assume the people pushing thousands of AI-generated commits are not the people paying for GitHub Enterprise.
    • bluerooibos 2mo ago
• 0xbadcafebee 2mo ago
  If that's the superficial cause, then the root cause is a lack of limits. Any production system in the world will go down from too much load if you don't limit the load. You have to do performance testing to determine what the system's peak utilization is, and put limits in place to prevent exceeding that limit. There are multiple kinds of limits you need. First you need the raw i/o limits on things like networking, load balancer requests per second, etc so you don't slow to a crawl on high load; you can throttle max i/o so demanding users don't steal performance for others, or even throttle network traffic to the client IP and internal services. Then you need additional limits at app layer, like number of requests you can process that make DB calls (as most DBs have a connection and request limit), or limit API requests to X/minute. Then you need more business logic limits, like max number of concurrent logged-in users, max number of repos a user can have, max number of requests for free users, priority for paid requests, etc. When you hit a limit, you have a couple choices. You can shed load (dropping calls immediately, which results in 503 errors for users, or even network disconnects), but it makes a very poor user experience, and can cause larger issues/conflicts in a distributed system. You can instead keep a max number of active sessions, and adjust that number over time, to ensure that (on average) your users have a stable experience; this requires implementing a "waiting room" for new sessions when old ones expire or log out; still not great for users, but it prevents everyone from getting random errors, which means the majority of users have a good experience. There are way more kinds of limits to implement. Each business has to analyze its entire system to identify performance bottlenecks and implement limits to ensure the majority of users continue to have good service. Without those limits, you get a shit experience for everyone, your brand is tarnished, and customers run for the hills. Finally, the performance testing helps you plan how much infrastructure you need to serve a given number of users. If you plan properly, you know X infra allows for Y number of users. Once you exceed Y users, you need to start throttling or stopping new user signup (and requests per minute) until you can increase capacity. Continuing to allow signups and additional usage, without the capacity to support it, results in what we see today. But competent engineers know all this. If they had competent engineers, and it gets this bad anyway, the problem is management. Microsoft likely already made back what they acquired GitHub for. They probably figured they could keep riding the gravy train until the train went off the rails, and either eventually sell GitHub or write it off. Such is business.
• adzuci 2mo ago
  I think that data is a bit dated and misleading, but I would love to see if AI slop is a contributing factor here once the dust settles and how Github will address it. Looking at the return to mean in https://areyougoingexponential.rhys.dev/kdaigle https://areyougoingexponential.rhys.dev/kdaigle makes me think the COO spike could be a Karpathy wiki or an agent too.
• evanelias 2mo ago
  My sense is that a lot of this activity isn't even legitimate use. As one example: my company has a GitHub app, and last night some bot added my app to 1700 repos. Then the bot immediately started rapid-firing commits which each affected hundreds of files, triggering a deluge of GitHub webhooks to my servers. The repo names all matched other GitHub apps, so my sense is this bot had added 1700 GitHub apps to 1700 repos, with that number rapidly increasing at the upper bound of whatever GitHub's rate limit is. My systems caught it quickly and auto-booted the bot, but the whole situation is ridiculous.
  • digitalsushi 2mo ago
    there's this attack on youtube where people who dont want their real names known will autoblock comments with their real names; bots will spam comments with every word and note the ones that were not visible. kinda reminds me of it. not my most valuable contribution to forensics, admittedly.
    • chrisjj 2mo ago
      This works on only people dumb enough to block only their own name, right?
  • phoenixy1 2mo ago
    The GitHub bot situation is so frustrating! A good ~50% of community traffic on our repos is spam and bots and to report them I have to fill out a lengthy form and MAYBE GitHub will ban them six weeks later. And I also can't tell whether our repos are being used (and thus worth investing in): GitHib only shows two weeks of traffic data and what they do have is completely useless because they can't filter out bot traffic.
  • conartist6 2mo ago
    Yeah I think that's really important. The average value of each commit, action, or hook has become significantly lower, and that necessarily means that they can't spend on quality. If you set the quality bar to cater to the top 10% of value-creating users (us) the amount you'd pay for quality at times when quality matters none would truly and completely sink you
• sharts 2mo ago
  I would take what C-level execs say with a grain of salt. Remember these folks hire people with plenty of system design experience. They likely also have the money for hardware resources. If load was an issue these things would have been solved already. Or they are purposely causing millions of dollars of productivity loss daily by not solving this. Something more serious is at play. And I want to hear from the actual underlings.
  • pan69 2mo ago
    > I would take what C-level execs say with a grain of salt. 100% agree. > Remember these folks hire people with plenty of system design experience. Not my overall experience 30 years in the industry. > They likely also have the money for hardware resources. You'll always be asked to do more with less, at least, that's my own experience.
    • joshspankit 2mo ago
      Also my experience
      • jasper1111 2mo ago
        Also my experience
        • rawland 2mo ago
          +1
          • haddr 2mo ago
            Also my experience
  • srmatto 2mo ago
    Infra is a cost center for a business unless your a PaaS and even then maybe still. Cost centers are usually ferreted out for savings, not investment.
• vb-8448 2mo ago
  I'm curios what is the bottleneck then? Because github(ms) has enough resources to hire competent people and provision needed hw to solve scaling the issues.
  • c0rruptbytes 2mo ago
    need competent people to hire competent people first - it seems like their only directive is to raise $MSFT not make good tech (talking about leadership, not my lovely msft engineers reading this)
  • moralestapia 2mo ago
    Yeah, aren't they super geniuses with infra that scales with a button, etc.?
    • aleph_minus_one 2mo ago
      > infra that scales with a button That is how Microsoft markets Azure. :-)
  • lallysingh 2mo ago
    Probably contention on databases.
  • tsss 2mo ago
    Looking at the design of Github Actions, I don't see any competency.
  • dspillett 2mo ago
    Ignoring the possibility of plain ol' management incompetence, when things are changing as fast as they are the problem becomes one of lead time. Hiring competent people does not happen at the click of a finger, nor in the current climate does getting new hardware resources. Hardware probably isn't going to do a lot with demand rising 10x or more at peak times, if they had scaled and tested systems with a view to supporting an order of magnitude more requests than expected then all that contingency resource is already gone. Adding 10% more hardware is going to be very expensive and will likely not have that much effect anyway without design changes to make better use of that hardware, scaling by orders of magnitude often reveals bottlenecks that were not apparent at the previous scales, or were apparent but dismissed (I imagine somewhere in a comment or design doc someone having written “we will need to rethink this if the system is to be asked to process 10ˣ⁺¹ things/period rather than 10ˣ”). And once you've hired those people, even if they get up to speed pretty much instantly, a system change isn't going to be designed/made/rolled-out in short order so patchwork fixes are still needed in the meantime, and any roll-out probably needs to be gradual which adds further complication. Scaling isn't easy. Scaling fast is a royal PITA.
• maccard 2mo ago
  GitHub’s reliability issues started earlier than the boom in agentic coding they talk about. I have comments on here from 2025 talking about massive outages with actions and issues.
  • fooey 2mo ago
    When you're the uncontested market leader, you don't budget operations to be perfectly reliable, you budget to not be too unreliable. All those 9's are expensive after all. The AI boom caught them napping though, and they're slipping well into the too unreliable category
    • aleph_minus_one 2mo ago
      > The AI boom caught them napping though Microsoft, who invested eyewatering sums into AI, even near to betting the whole company on an AI boom, is surprised about the AI boom?
      • minraws 2mo ago
        Microsoft bet everything on possible AGI, no one thought the result would be a bonafide commit printing machine.
        • samudrijan 2mo ago
          PRRRR
        • jasonvorhe 2mo ago
          That's not what Nadella said 1-2 years ago. Everyone seemed pretty aware of LLMs soon being able to handle a lot of steps of software development. I bet it's mostly due to Azure being brittle and the AI boom is just a nice cover story (not saying it's not challenging to handle all that new load) to save face and sell more Azure services.
          • 2mo ago
• meerita 2mo ago
  I can't believe they've been asleep all these months, not foreseeing that AI would generate more commits and Git usage and, consequently, not preparing themselves to scale.
• jjice 2mo ago
  I want to believe that it's justification, but looking at the historical uptime [0] (early data is actually overly positive on this graph since it had to scrape historical data instead of realtime), I can't be convinced of this anymore. They do have more load - that's for sure. I just don't believe it's an excuse for their god awful uptime. [0] https://damrnelson.github.io/github-historical-uptime/ https://damrnelson.github.io/github-historical-uptime/
• layer8 2mo ago
  Turns out that the cloud doesn’t scale that easily by itself after all?
  • aleph_minus_one 2mo ago
    > Turns out that the cloud doesn’t scale that easily by itself after all? The cloud does scale that easily, the software doesn't. :-)
• lluisantoni 2mo ago
  That is a good point. And this issue looks analogous to what is happening in the research publication world (arxiv, peer-review, etc). In particular, I wonder whether code pushed is of lesser quality, CI pipelines break more often too, and this saturates github servers.
• LarsDu88 2mo ago
  The fools thought we needed data centers for agents, but we actually needed data centers for Github!
• winrid 2mo ago
  That doesn't answer the question at all. I suspect most components are already horizontally scalable, if not, which ones? That would be an interesting discussion for HN.
• rpearl 2mo ago
  That is 455 commits per second. I know that's not qps over the system but it's not that huge of an input number either. Not for big tech like msft.
• zldar 2mo ago
  COOs lie. There is no chance that MSFT would openly admit that its mandated vibe coding activities at GitHub have failed. What speaks against the volume activity is that GitHub works perfectly after an outage. People don't just stop AI coding after an outage. If anything, the volume should be higher for people to catch up. It is more like rebooting a vibe coded WIN-95 because of a vibe coded integer overflow (joke).
• ksec 2mo ago
  This is the part Github is doing really bad at, it's PR. Had these DATA and message been tossed around more people wouldn't be so frustrated and would be more understanding.
• smcleod 2mo ago
  GitHub's reliability has been going downhill longer than AI has been becoming popular. I feel like the load is becoming the scapegoat.
• bcx 2mo ago
  14X at github's scale is unlikely an easy problem to solve. Additionally, there is a strategic reason to be the repository for all code, free, or not which is likely at conflict with reliability goals. (e.g. they could get rid of their free tier which would improve reliability at the cost of allowing a new competitor to be the repository for all new AI driven code) The long term strategic benefit of being the home to all code (or most of it) is enormous. They are likely balancing that strategic goal with what is seen as short-term internally as reliability.
• ls-a 2mo ago
  [dead]
• bmitc 2mo ago
  Maybe they should have planned on that when they went all in on Copilot stuff several years ago? Even the GitHub organization is under the larger Cloud/AI organization at Microsoft. Further, it is clear that Microsoft and GitHub have been strongly encouraging, if not requiring, AI tool use in their own, internal development. I have seen several automated PRs within GitHub repos that were 100% done by AI. Big surprise where that's leading.