5 ms·
I had a lot of goodwill for GitHub but I think today is the tipping point. Looking at a unicorn page, I feel this lingering hope that it's transient (like it u
by khvn26 2mo ago
I had a lot of goodwill for GitHub but I think today is the tipping point.
Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again.
The hope is dead.
- stingraycharles 2mo agoI’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?
- benrutter 2mo agoI'm really curious how other codeforges are faring. I only use codeberg and github - codeberg seems to have less outages, especially for a much smaller organisation, but they've recently removed support for LLM projects which could be linked. Anyone here have data on whether the likes of gitlab have been impacted in the same was as github?
- sznio 2mo agomy personal forgejo instance has had 100% availability for the last two years.
- spockz 2mo agoSo it has run on a server that hasn’t had updates in 2 years? And it itself hasn’t been updated in 2 years?
- cyberax 2mo agoYou can run it in an HA configuration with multiple servers and live updates.
- spockz 2mo agoYes. And the same for the database and then you get it into “professional” territories with more moving parts that can also fail and take 3 hours to figure it out and recover.
- cyberax 2mo agoYup. It's kinda binary: either almost no downtime for years with a crazy number of nines, or a total mess for several days while people figure out the issues. That being said, a Git forge for a small/mid-sized company is not something that requires rocket surgery. Forgejo with an SQLite database and a daily backup to S3 would be more than enough for the majority of users.
- forgejojoj 2mo agoThe key is that maintenance is done on your own terms, and thus it's available when you need it.
- koalp 2mo agoI love codeberg but it also have outages and is sometimes quite slow. But I’m more lenient with codeberg because they are a non-profit and as you say, they are banning vibe-coded project because of the cost to infrastructure. In the meantime, GitHub is encouraging vibe-coding, multiplication of the source code, and genAI discussions.
- Maxion 2mo agoThey've done the typical sillicon valley sillyness of having 10 000 engineers over-engineer a product so that there's way too many features built on top of what should be a simple tool and are now paying the price. Literally every larger US company makes this mistake.
- nicoburns 2mo agoI saw an HN comment a few weeks back claiming their traffic has increased ~10x.
- ayewo 2mo agoYup. The original source was Kyle Daigle, GH COO. - https://x.com/kdaigle/status/2040164759836778878 https://x.com/kdaigle/status/2040164759836778878 - https://xcancel.com/kdaigle/status/2040164759836778878 https://xcancel.com/kdaigle/status/2040164759836778878
- rvz 2mo agoWell, it has been 8 years since the acquisition and they have had access to Azure infrastructure to keep up with the scale and yet, it is still struggling due to free compute thanks to Actions. They need to start getting rid of the free tiers for GitHub Actions and actually charge appropriately or increase prices for it.
- kedihacker 2mo agoIt should be embarrassingly paralel I don't get how they can't figure out
- bamboozled 2mo agoIsn't all the LLMs and AI supposed to be "solving these problems" because all the engineers are now 10x ? Not being sarcastic but I mean, this is what we were told and sold...by the likes of...Microsoft.
- worble 2mo agoI have little sympathy that they're buckling over a problem they helped create.
- bamboozled 2mo agoExactly.
- kokanee 2mo agoYeah let's not forget that in those brief few days when Sam Altman was fired, Satya Nadella swooped in and hired him. Despite the failure of that move, Microsoft has invested $13 billion directly into OpenAI, and has spent $100 billion on the partnership. No sympathy from me watching their software crumble as a result.
- account42 2mo agoThey have also been pushing people very hard to use the slop generators, including through GitHub.
- slowin 2mo agoServices like Facebook have orders of magnitude more activity than GitHub and manage much better uptime.
- dannyw 2mo agoServices like Facebook aggressively block automated activity, including AI agents. GitHub sells this as a service.
- slowin 2mo agoBots or not, Facebook has orders of magnitude more traffic.
- dannyw 2mo agoYes, but their shapes are very different. Most users interact with a small, geographically constrained bubble. It is (comparably) an easier problem for the usage patterns of Facebook. The app is also designed so when you open Insta, only a few images are shown, infinite scroll is there for a reason. The search page suggestions? Probably from a cache and then re-ranked quickly.
- aurareturn 2mo agoHTTP traffic is different from run this CPU intensive pipeline service for 10 minutes and now increase the volume by 100x in a few months. I used to manage a popular app/website that had tens of billions of real user page views per month. It wasn't easy but it also wasn't hard. It was surprisingly easy with a single large Postgres DB, a lot of caching, optimized tables and queries.
- sharts 2mo agoEven Claude could have caught and addressed most of these scalability issues long ago. This can’t be due to load or capacity. This is something else.
- dannyw 2mo agoI’d love to see the org chart sizes How many hires went into infra and platform? How many went to Gen AI org units? From what I see, and that is a constant wave of very meh features, it seems like the latter.
- deleted 2mo ago[deleted]
- BowBun 2mo agoHow many weeks/months can they use this excuse? They are literally are at the forefront of this emerging industry and are capturing untold value. To let their product suffer and potentially lose market share because of it is extremely foolish
- kypro 2mo agoScaling can take time but they've had at least a year to prepare. It's not like any demand increase they've seen was overnight... At most normal companies you monitor your systems and address potential bottlenecks before they reach a tipping point. And generally you want enough headroom that a sudden 2-3x increase in demand wouldn't take out the service. Either Github's technical leadership/talent is completely out of their depth (seems unlikely), or they're not hiring enough engineers to work on these capacity issues.
- unknownfuture 2mo ago> Either Github's technical leadership/talent is completely out of their depth (seems unlikely), or they're not hiring enough engineers to work on these capacity issues. Regarding their leadership, I'd argue the latter implies the former. So yeah I'd say they're out of their depth.
- dannyw 2mo agoMore likely, as seen in a lot of companies, perhaps infra teams got whittled down (or frozen HC, less than BAU, etc) with resources reallocated to AI org units.
- lkjdsklf 2mo agoHardware resources are also allocated to AI uses so you can't scale out that way anymore.
- peezd 2mo agoI got the impression from a previous thread that the continual issues were largely driven by the required move to Azure, and underlying Azure stability
- mrighele 2mo ago> scaling problems due to the influx of LLM assisted coding You mean people pushing LLM assisted coding, or github's own coders pushing LLM assisted code to production ? :-). I am more inclined to think that it is the second, but if the reason is the first, what is particular hard to scale ? It's not like everybody is vide-coding into a single repository, so I would expect that most of the load would naturally be scalable by increasing the resources. If for some reason that is not possible, I would expect a corporation as big as Github to keep the paying customers working by gracefully denying service to the free tier, not to collapse like an house of cards.
- krs_ 2mo ago> You mean people pushing LLM assisted coding, or github's own coders pushing LLM assisted code to production ? :-). Why not both?
- Groxx 2mo agoAll three: their own quite aggressive pushing of LLM coding as a whole.
- m4rtink 2mo agoA perfect slop storm - slop bringing down a slopified service.
- theappsecguy 2mo agoThey've had an insane influx. It seems that they should more aggressively limit free tier activity or isolate paid users, this has become unbearable...
- dannyw 2mo agoOr maybe prioritise humans / human-shaped activity? Agents can tolerate a 503+Retry-After is a lot less frustrating to an agent than a human.
- cautiouscat 2mo agoThat’s the reason they gave, and I believe them. But I also think there has to be more to it. If your paying users are seeing degradation caused by (likely) free tier users (for years by the way), restrict the free users.
- lijok 2mo agoIf they did, they would’ve paused new feature development a long time ago and redirected that resource towards scaling the platform. And yet they’ve been releasing tonedeaf features more than ever.
- StilesCrisis 2mo agoFeature work is frankly much easier than scaling and bullet-proofing.
- catigula 2mo agoEnterprise software isn't about giving "breaks"; it's about does the product work for you or not.
- nxc18 2mo agoThe quality and performance of GitHub enterprise has degraded horribly this year. Their problems are self-inflicted and influx is just an excuse. Despite minimal new features, and drastically increasing hardware resources to keep up, self-hosted GitHub enterprise went from things being instant to most requests taking many seconds. And activity on issues in one repository tends to mysteriously re-open issues with the same number in other repositories. I think they are vibe coding and facing the consequences.
- ALLTaken 2mo agoPlease let this not be true :( I mean nothing against vibe coding for hobby and assisting you when you get stuck, but at something important like Github they should have hooks to stop low-quality commits and repeated CI-breaking or issue causing should automatically restrict accounts at such companies I think. Do you think they have such measures in place? I remember when my indention wasn't spotless our university rejected my commits or if my function complexity was too high.. Tears in my eyes.. I think it was called Gerrit Code Review.
- ToucanLoucan 2mo agoIt’s become clear to me that any company acquired by Microsoft should be avoided. All software dev is hard. Software dev within a huge organization is damned hard. And Microsoft is SO HUGE and has so much infighting and finger pointing going on that they’re now, as far as I’m concerned, incapable (let alone unwilling) to ship good software. No faith left. None. They turn everything into a quagmire.
- DrewADesign 2mo agoCome on now let’s be fair. They couldn’t possibly fix all of GitHub‘s bugs and deployment problems while maintaining their breakneck pace of mission critical feature deployments. You know, probably something sexy like replacing Win11’s maximize, minimize and close buttons with a chat prompt where you can just tell copilot what you want the window to do.
- pera 2mo agoSave that for your local mom-and-pop store: Microsoft is a multi-billion dollar corporation with enough resources to, at the very least, provide a reliable service for enterprise clients. The "AI is using a lot of resources" excuse was maybe acceptable last year but not in Q3 2026.
- unknownfuture 2mo agoDoubly so given this is hardly a surprise. We've been on this trajectory for at least a couple of years now. They don't get to shrug, point at 10x volume, and act like they've been blindsided.
- mhitza 2mo agoThey had large increase in volume, and trying to move to Azure at the same time. I don't envy them for either work they need to do. But also don't feel pithy because it's Microslop, at the end of the day.
- pdimitar 2mo agoAll being said, I have most sympathy towards engineers of all people. They don't get to call the shots, they do as they are told. 99% of the time by an extremely out-of-touch managers. Managers and executives though? Those I do blame. Surely at this point it should be blindingly obvious their current strategy is not working.
- account42 2mo agoThose engineers also chose to work in US big tech for the ridiculously high salaries that come with it. Having to deal with the problems resulting from that scale is just part of the deal.
- inquirerGeneral 2mo ago[dead]
- 2mo ago
- iAMkenough 2mo agoConsidering their leadership ushered that in themselves, that’s just poor planning.
- Scubabear68 2mo agoThe thing is Github’s core is git’s core plus extras, and thet core is fundamentally text wrangling. Youtube and Insta and Facebook and many other social media platforms handle very resource intensive media for billions of people plus bots. Increased influx from AI is not plausible when viewed from that lens.
- edgyquant 2mo agoIt is though. Maybe you think they should be doing a better job but 12-24xing scale when you were already handling billions of commits is definitely understandable to me. It just does feel like they should have some way of ensuring customers, paying ones, don’t have to suffer so much.
- spockz 2mo agoYes. It seems that at the very least repositories owned by paying customers should be processed on priority threads. And perhaps throttle commits/pushes to repos where there are a lot of pushes from the same person all the time. This can be effectively throttled at the push level.
- user43928 2mo agoIf I hadn't already switched to GitLab long ago, I'd certainly do so when my pushes to GitHub are throttled.
- ajasmin 2mo agoLots of paying customers also depend on the open source part of GitHub to run their business.
- verve_rat 2mo agoThat sounds like an opportunity for GitHub to clip the ticket for providing a way for those businesses to support those projects more directly.
- 2mo ago
- mooreds 2mo agoFound this reddit thread from a month ago that talks about the scale: Commit volume went from 1 billion total in all of 2025 to 275 million per week by early 2026. https://www.reddit.com/r/softwaredevelopment/comments/1uv5s73/github_went_from_1b_commits_in_2025_to_275mweek/ https://www.reddit.com/r/softwaredevelopment/comments/1uv5s7...
- ChoGGi 2mo agoDo you know how many thousands/millions of dollars that would cost to serve!? Microsoft has to spend more money on acquiring games studios for billions of dollars then shut them down.
- mooreds 2mo agoNo idea about the cost. I do know that very few systems are designed to handle a 14x increase in traffic painlessly (275M*52 = 14.3B), especially at scale.
- rfgplk 2mo agoTracks with some of my internal metrics. There are certain projects I'm tracking (solo dev or a small team of a few people) doing 500k LoC changes _per week_. There's one project that caught my attention in particular because they had something like 800k commits and around 7 million lines of code... and they created the repo in _July_. Also, yes I checked, the repo gradually grew over time they didn't just splat it in the first commit.
- ljm 2mo agoGithub has had a pretty spotty track record for uptime for a very, very long time. It didn't get any better when MS bought it out. It has really just coasted on GH's network effect because everybody uses GH for git by default. It doesn't really have much direct competition and anything that does offer something like GH just doesn't have the same mindshare. In that way, GH became more like MS.
- jimbob45 2mo ago[flagged]
- esseph 2mo ago> HN is heavily anti-US This is a very distorted lens. Many on the outside might say the US is very pro-self and anti-everyone-else right now! America first, right? Well, take a random country. We'll make a made-up one called Tristonia. Should Tristonia do things in Tristonia's best interests? Absolutely! If Tristonia had some kind of military or economic threat from another country, should they take steps to protect their country, resources, and people from the threat? Absolutely! The same thing the US would do. Most of these people aren't necessarily anti-US, they're against being beholden to another country, especially one that has been incredibly... inconsistent, as of late.
- mnewme 2mo agoThe vast majority of HN users are from the US.
- shimman 2mo agoI'm a US citizen and I recommend people avoid most US corporations all the time. Helping US corporations does not help the blight of everyday Americans, all you do is make like a few thousand billionaires more wealthy. I also find the excuses for these trillion dollar companies pathetic. These are entirely human constructed systems, if they can't understand what they made and how to improve it then maybe we need to question the way they write software in general if it's this unsustainable. I hardly doubt some other team with better leadership qualities couldn't improve things to be better than they are now.
- ALLTaken 2mo agoNo, that's almost exactly right, except that scaling problem is inverse! MSFT is giving away most of their Azure Datacenters resources to AI, to make money and doesn't care if Github suffers along the way and drains Github's available resources to bare minimum. Leading to outages like this, but you're 100% spot on right too. All those Claude/OpenClaw instances produce so much Code, people setup more and more repositories.
- m4rtink 2mo agoAre they really making money from AI or just dumping resources on hoping they will be making money from it ?
- hansmayer 2mo ago[dead]
- PrayagS 2mo agoThat excuse was valid a few months ago or back in 2025. Not anymore.
- bjackman 2mo agoI deployed Forgejo on my homelab yesterday. Now GitHub is just a free backup service. I guess they're quite happy with that TBF, it doesn't seem like they're that interested in being a git forge anyway.
- fmbb 2mo agoThey should replace the unicorn image with a horse or at least a zebra during ongoing incidents.
- chrisjj 2mo agoDonkey, surely.
- brahma-dev 2mo agoAtleast you're getting the unicorn. I am getting 404 on valid links. For a moment I thought the repo is gone. https://imgur.com/b0Xj3Z6 https://imgur.com/b0Xj3Z6
- SeriousM 2mo agoThat unicorn was already served in the RubyOnRails days!
- nine_k 2mo agoBecause it comes from https://en.wikipedia.org/wiki/Unicorn_(web_server) https://en.wikipedia.org/wiki/Unicorn_(web_server)
- thanh76 2mo agoI think the same. But the GitHub CLI is the biggest block for me. It is a killer feature. No other tool is close. To self-host the forge is easy. To replace gh in my scripts and agent workflows is not.
- regexorcist 2mo agoAny good LLM should be able to come up with the equivalent git and curl commands fairly easily.
- jbs789 2mo agoThe Unicorn page rubbed me the wrong way. Thankfully I was able to revert changes to my prod instance using another approach. But I count on GH for reliability.