4 ms·
To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amoun
by ieie3366 3mo ago
To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code
Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside.
If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health
- kilroy123 3mo agoAs a _paying_ customer, I disagree.
- mococa 3mo ago> increased something like 50x due to LLM This should not be an excuse since they probably can use AI to fix it /partially sarcasm
- eddieroger 3mo agoFor the amount of money enterprises are paying to GitHub, there is a reasonable (and contractual) expectation of uptime. I don't think my boss would find it to fun if I bailed work to hit the gym just because GitHub was down.
- deleted 3mo ago[deleted]
- iamcoder18 3mo agoYes, but GitHub enterprise has different uptime (https://us.githubstatus.com/posts/dashboard https://us.githubstatus.com/posts/dashboard) and actually does have an SLA. Edit: it seems GHE is also affected by the outage and isn't much better.
- twistedpair 3mo agoSince upgrading to Cloud Enterprise... why didn't my GH uptime get any better? Guess on-prem GHE server is the only way to do that.
- cyansmoker 3mo agoAs an enterprise user I can tell you that: - we are getting no benefit beyond getting excuses replies to our emails - right or wrong, but we depend heavily on GitHub actions so this has a very real impact on us
- tedivm 3mo agoThis, like all of their dashboards, is complete bullshit. As a GitHub enterprise user i am absolutely affected by the outage today, and have been affected by all of the GitHub Cloud outages. There is no separate enterprise platform unless you buy their self hosted server product.
- eddieroger 3mo agoYeah, I know, and it's down.
- AgentK20 3mo agowhat if I do have an emergency hotfix for a production outage though ;-;
- askonomm 3mo agoThat's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.
- AlexandrB 3mo agoBoomer opinion but trusting third parties to be critical infrastructure, especially with no SLA in sight, will always end in tears. "The cloud" is very convenient, but its providers will never care about your infrastructure or your customers as much as you will.
- BryanBeshore 3mo agoTruly a boomer opinion. Respect!
- Maxion 3mo agoOlder millennial here and I agree. If you rely on Github you should at least be have your processes such that you can work around it.
- whynotmaybe 3mo agoThat's the reason why my build agent is self-hosted
- raffael_de 3mo ago+1 (as a millenial) ... especially given that setting up a git server for non-OSS company code isn't too much of a challenge really. also, no need to self-denigrate this reasonable opinion in preemptive obedience.
- otterley 3mo agoGitHub is way more than just a git repo host. It manages code reviews, merge (pull) requests, and has an entire CI/CD workflow engine in it. Replicating all that is a challenge that most orgs are not up to.
- raincole 3mo agoOr, you know, it's perfectly reasonable and natural to feel angry when a service you paid for gets worse over time. > Unless you have a emergency hotfix (you don’t) Oh, we do. Given the sheer number of users, it's almost guaranteed someone is on fire ever time GitHub is down. Statistics is a very charming branch of reality.
- leishman 3mo ago> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Which means they have a pricing problem. Rate limit non-paying users.
- anoplus 3mo agoRight. It is not an excuse and they were notorious for outages before LLM era. Also, LLM can serve Github engineers as well, so we are all playing in the same field.
- cluckindan 3mo agoThey already do
- rzzzt 3mo agoI can't check the commit history of some OSS projects without hitting a rate limit if I'm not signed in. My request rate is one request per (arbitrary time interval) at that point.
- xnorswap 3mo agoJust take away free Actions and it would surely solve a lot of their scale issues. It's absurd that I have dozens of repos, many with GH Actions that run CI, test and then package and push to prod/package managers, and haven't paid GH anything.
- chrisjj 3mo agoNo. It means they have a capacity problem.
- s_dev 3mo agoWhat if we're not angry and just perplexed? Do we calm down further or can we still ask questions?
- ethagnawl 3mo agoNo questions. Only bench press.
- Topfi 3mo agoNo one ever has to commit anything time sensitive.
- sausagefeet 3mo agoActually, I do have an important fix for a deal we are trying to close that does need to go out. And I am paying GitHub to host this. Once or twice, I can see, but GitHub's SLA is getting worse than just hosting it myself, and that's the entire reason I pay GitHub.
- world2vec 3mo agoGeez, sorry that the paid service my company paid for is down and I can't do my work.
- jjice 3mo ago> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code I sympathized with them when they said this a handful of months ago, but then I saw this [0] page that shows how it's been shot for years prior (which tracks with my memory). I feel for the GH engineers that have to deal with this, especially the SREs. I also don't hate the downtime right now, as I'll make a cup of coffee and do something else. I will say though, I did have a hotfix a week or so ago during the Actions outage, which really was a pain. You're right that getting angry and ranting isn't the right reaction here, but I do no give them the LLM load excuse. I don't give them an out for having awful uptime during work hours for a product we pay quite a bit for as an org. [0] https://damrnelson.github.io/github-historical-uptime/ https://damrnelson.github.io/github-historical-uptime/
- Topfi 3mo agoGithub being under the CoreAI division probably also doesn't help the engineers prioritize addressing infrastructure issues and makes using LLM load as an excuse feel self-inflicted. Akin to feeling sorry when a pyromaniacs house burns down...
- debarshri 3mo agoIt looks like right after Microsoft acquisition this started going off. This is the outcome of violating single responsibility principle in business.
- twistedpair 3mo agoThey _are_ the LLM coding agent vendor, and are _owned_ by MS, the ~majority~ biggest shareholder of OpenAI. How can you NOT consider that 20x+ scaling in your capacity roadmap projections, where you are trying to get everyone to use these agents as part of your core OKRs? Or, maybe your #1 IT priority was moving everything to Azure instead ;)
- rafram 3mo ago> the majority owner of OpenAI Largest shareholder (27%), not majority.
- cesarb 3mo ago> go hit the gym or walk outside. My mind immediately went to this classic xkcd: https://xkcd.com/303/ https://xkcd.com/303/
- javier2 3mo agoWhile I agree in principle, Github with their CI is very critical to many parties.
- sarjann 3mo agoThis has to be rage bait, this is a critical piece of infrastructure for many people. Stuff going down can lead to deployments failing and as you mentioned in many cases emergency hotfix's. The idea that we should be fine with this unreliability is just amazing. It's not a mental health issue to have problems when important infra fails.
- john_strinlai 3mo ago>The idea that we should be fine with this unreliability is just amazing. the comment doesnt say you should be "fine" with the unreliability. they are saying people shouldnt get so emotionally worked up over it. which, while i wouldn't phrase it in the way the parent did, i agree with the direction of their point.
- sarjann 3mo agoExcept he tries justifying it. "Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code" "Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside." with assumptions like above. If it was, "hey they f*ed up but there's no point having an overly emotional reaction" that would be fine. But he seems to be justifying this. Especially with Github's record up to now of unreliability I think it's completely fair to be annoyed.
- zahlman 3mo agoI don't even see a whole lot of emotional worked-up-ness, though. Just continuing disappointment.
- dimgl 3mo agoWhat if you, you know, have a job?
- phtrivier 3mo agoI understand the need to level-head the discussion and calming everyone. However, your "(you don't)" comment is not going to calm down all the people, who, you know, DO have a hotfix to push now, and DO have an angry customer that could not care less for which part of our infrastructure is breaking _their_ workflow. The only things would calm everyone down is guarantee that Microsoft would be paying for _our_ SLA breach compensation. But they don't. And I don't think anyone is paying their GH bill with a prorata of the number of time the platform was actually available. (I'm also aware that the wording of the contract probably clearly says that you should not use GitHub for anything critical, that Microsoft is only a small startup in their garage, that you can't credibly expect 90% uptime anyway, and that it's all the fault of LLM slop ! Bad LLM slop ! Also, please buy our LLMs to generate more slop, please.)
- fn-mote 3mo agoIt's past time to move your critical infrastructure off Github then, no?
- phtrivier 3mo agoIt's not exactly done in a pinch, but yeah, I guess someone must be working on disrupting CIs by the audacious value proposition of "we'll work when you need to".
- maccard 3mo agoThat effectively means “stop using GitHub” though. Which is definitely worth considering but also kind of insane to think as the best option.
- bhouston 3mo ago> To everyone who is angry: calm down. Github’s servers are constantly on fire Is this rage bait? Isn't Google, Amazon, and all the other services in the world similarly impacted by LLM's? Isn't Claude, OpenAI, etc.? Why can they handle the load but not Github?
- denysvitali 3mo agoStop blaming LLMs: https://damrnelson.github.io/github-historical-uptime/ https://damrnelson.github.io/github-historical-uptime/
- c2403 3mo agoIn this case, perhaps they shouldn't allow AI slop or promote usage of AI agents...
- madeforhnyo 3mo ago> due to LLM sloppers pushing large amounts of trash code Which they encouraged by pushing Copilot down everyone's throat
- slowin 3mo agoWell, they’re a critical piece of infrastructure with terrible stability. I’m in the process of migrating us off GitHub now. I’m not a Meta fan but it’s interesting that they manage to keep their systems up with an order of magnitude more traffic. GitHub’s uptime is inexcusable.
- AustinDev 3mo agoAlternatively, don't use GH for critical infrastructure they are no longer reliable, simple as.
- TZubiri 3mo agois the infrastructure of the free tier shared with that of paid users? That might be the issue. Otherwise if usage of paid users scales, then it would be fine.
- john_strinlai 3mo ago>is the infrastructure of the free tier shared with that of paid users? github enterprise is fully operational at the moment
- nope1000 3mo agoAt least enterprise EU is partially affected https://eu.githubstatus.com/posts/dashboard https://eu.githubstatus.com/posts/dashboard We cannot pull any actions images (or how it is called) for example.
- neo_doom 3mo agoAgreed. This impatience culture is so toxic. Been down less than 30 mins.
- anonymars 3mo agoThat's cool, I mean if you'd gotten here earlier you could have said "been down less than 5 minutes" So with the "elapsed time" out of the way (now over an hour, incidentally), how about "how long will it be down?"
- neo_doom 3mo agoEnded up being down for about 4 hours. Hope you were able to get out and touch grass.
- anonymars 3mo agoFor some people losing half a day of work is kind of important. Hope you were able to get out and find some empathy https://news.ycombinator.com/item?id=49331481 https://news.ycombinator.com/item?id=49331481
- cb33 3mo ago2 hours later - still down. Is this "impatience culture" or incompetence culture at this point?
- anonymousab 3mo agoAllowing and accepting the LLM load is a willful choice they made and are making at the expense of their users, including their paying enterprise users. They could easily tighten things up in that regard, and make a choice that is right for their main users at the expense of The MS corpo mandate/mission. It is a choice to do otherwise.
- pamcake 3mo agoAllowing and accepting? More like encouraging and instigating.
- constant_flux 3mo agoGH shouldn't have promoted LLM usage the way they have if they didn't have the infrastructure to support it. Regardless of how you feel about the code quality (which you have zero evidence of), if GH has made commitments to supporting broad LLM usage, they need to back that up with the proper hardware and without whatever fragile SDLC processes they have.
- pamcake 3mo ago> GH shouldn't have promoted LLM usage the way they have No qualifiers necessary. But try arguing with a lawnmower...
- stetrain 3mo ago> as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Github and their parent company are active participants in pushing LLM-driven coding.
- DharmaPolice 3mo agoThis might be naïve but wouldn't the appropriate response be to reduce access/rate limit free/new accounts in order for service to be maintained for everyone else?
- nerdix 3mo agoWouldn't be surprised if we lost free private repos because of the slopocalypse. Though most don't seem to be shy about sharing their slop with the world so I'm not sure if that will actually increase stability.
- deleted 3mo ago[deleted]
- sharts 3mo agoIt has nothing to do with load.
- otterley 3mo agoThe CTO said otherwise: https://github.blog/news-insights/company-news/addressing-githubs-recent-availability-issues-2/ https://github.blog/news-insights/company-news/addressing-gi...
- karissa 3mo agohttps://damrnelson.github.io/github-historical-uptime/ https://damrnelson.github.io/github-historical-uptime/
- otterley 3mo agoFirst, that source is unreliable, particularly for before the acquisition. (We don’t know whether the data is accurate, it’s based on GitHub’s own self reporting.) Secondly, the scale is WAY off and makes it look far worse than it is. It makes it look like 99.5% availability is practically zero availability. Finally, GitHub’s availability is not a binary all-or-nothing proposition. They report incidents on a granular level, for example, webhook firing can be impaired while Git hosting may be working fine.
- maccard 3mo ago0.5% downtime is an hour a week. If that happens for 10 minutes a day at 11pm PT it doesn’t matter, if it happens for half a working day once a month, it’s a disaster.
- OtomotO 3mo agoYou know, I am not employed. I have little time in my day to spend with my daughter. But I also have responsibilities. And one client has decided to bet on github. It's the only client I've ever worked with who has their code on github. EVERY other client has hosted their own gitforge or used bitbucket. So now I am not angry because some critical piece of US-american infrastructure is down all the time. I am angry because instead of spending time with my daughter I have to work on this later, because there are due dates and "Well, fucking GitHub was down" ain't gonna cut it.
- malfist 3mo agoWhy do you get to tell me how I should feel?
- semiquaver 3mo agoWhy would you take what they say at face value? I believe that this is basically a lie covering up deeper rot.
- catigula 3mo agoFYI this is an argument to not use Github, not an argument to use it.
- _joel 3mo agoIt's every other day now. We pay for a service, we expect service.
- bradhe 3mo ago> due to LLM sloppers pushing large amounts of trash code hey bud your bias is showing
- robofanatic 3mo ago> constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code I find it hard to believe that github APIs don't have rate limits to handle traffic spikes.
- thejosh 3mo agoit's been trash for years, their status page just doesn't reflect that.
- deleted 3mo ago[deleted]
- armSixtyFour 3mo agoHard disagree, people are paying GitHub a lot of money. If they are facing an influx of commits from bots maybe they need to start metering commits from those accounts and charge them money. This is a self imposed problem.
- bmitc 3mo agoIf we all go to the gym every time GitHub is down, we're all gonna get ripped. GitHub was down all day just last week. They get paid millions of dollars by my company. They need to get it together.
- bluerooibos 3mo ago> calm down If people and companies are paying for critical infra they expect it to be available - it's not MY problem that their servers are on fire - last time I checked, having too much demand was a good problem to have and they've had over a year to fix it and come up with a plan. I'm so glad I migrated to GitLab months ago.