9 ms·
GitHub: Git operation failures
- shooker435 1y agoThe internet is having one heck of a day! we focus on ecommerce technology and I can't help but think our customers will be getting nervous pre-BFCM.
- broosted 1y agocan't do git pull or push 503 and 500 errors
- meirp 1y ago[dead]
- bhouston 1y agoI cannot push/pull to any repos. Scared me for a second, but of course I then checked here.
- shooker435 1y agohttps://www.githubstatus.com/incidents/5q7nmlxz30sk https://www.githubstatus.com/incidents/5q7nmlxz30sk it's up now (the incident, not the outage)
- dogman123 1y agohell yea brother
- MattGaiser 1y agohttps://www.githubstatus.com/incidents/5q7nmlxz30sk https://www.githubstatus.com/incidents/5q7nmlxz30sk
- fidotron 1y agoIt used to be having GitHub in the critical path for deployment wasn't so bad, but these days you'd have to be utterly irresponsible to work that way. They need to get a grip on this.
- MattGaiser 1y agoEh, the lesson from us-east-1 outage is that you should cling to the big ones instead. You get the convenience + nobody gets mad at you over the failure.
- bhouston 1y agoEverything will have periods of unreliability. The only solution is to be multi-everything (multi-provider for most things), but the costs for that are quite high and hard to see the value in that.
- dylan604 1y agoyes, but if you are going to provide assurances like SLAs, you need to be aware of your own allow for them. if you're customers require working with known problem areas, you should add a clause exempting those areas when they are the cause.
- thinkindie 1y agoI really can't believe this. I had issues with CircleCI too earlier, soon after the incident with Cloudflare resolved.
- arbol 1y agoI'm also getting this. Cannot pull or push but can authenticate with SSH myrepo git:(fix/context-types-settings) gp ERROR: user:1234567:user fatal: Could not read from remote repository. myrepo git:(fix/context-types-settings) ssh -o ProxyCommand=none git@github.com PTY allocation request failed on channel 0 Hi user! You've successfully authenticated, but GitHub does not provide shell access. Connection to github.com closed.
- smashah 1y agosame
- keepamovin 1y ago[flagged]
- JLCarveth 1y agoThe last outage was a whole 5 days ago https://news.ycombinator.com/item?id=45915731 https://news.ycombinator.com/item?id=45915731
- jmclnx 1y agoDidn't I hear github is moving to Microsoft Azure ? I wonder if these outages are related to the move. Remember hotmail :)
- bhouston 1y agoHuh? What were they on before? The acquisition by MSFT is 7 years ago, they maintained their own infrastructure for that long?
- JLCarveth 1y agoThe Github CEO did step down a few months ago, they never named a successor. Could have something to do with the recent issues. https://news.ycombinator.com/item?id=44865560 https://news.ycombinator.com/item?id=44865560
- silverwind 1y agoYes they are/were on their own hardware. The outages will only get worse with this move.
- kennysmoothx 1y agoWhat a day...
- clbrmbr 1y agoSame issue here for me. Downdetector [1] agrees, and github status page was just updated now.
- cjonas 1y agoI didn't really want to work today anyways. First cloudflare, now this... Seems like a sign to get some fresh air
- dlahoda 1y agowe depend too much on usa centralized tech. we need more soverenity and decentralization.
- worldsavior 1y agoHow is this related to them being located in the USA?
- CivBase 1y agoThe sad part is both the web and git were developed as decentralized technologies, both of which we foolishly centralized later. The underlying tech is still decentralized, but what good does that do when we've made everything that uses it dependent on a few centralized services?
- lorenzleutgeb 1y agoPlease check out radicle.dev, helping hands always welcome!
- letrix 1y ago> Repositories are replicated across peers in a decentralized manner You lost me there
- hungariantoast 1y ago"Replicated across peers in a decentralized manner" could just as easily be written about regular Git. Radicle just seems to add a peer-to-peer protocol on top that makes it less annoying to distribute a repository. So I don't get why the project has "lost you", but I also suspect you're the kind of person any project could readily afford to lose as a user.
- smashah 1y agoSpooky day today on the internet. Huge CF outage, Gemini 3 launches now I can't push anything to my repos.
- sgreene570 1y agogithub has had a few of these as of late, starting to get old
- baq 1y agoRemember talking about the exact same thing with very similar wording sometime pre-COVID
- cluckindan 1y agoMSFT intentionally degrading operations to get everyone to move onto Azure… oh, wait, they just moved GitHub there, carry on my wayward son!
- blasphemers 1y agoGitHub hasn't been moved onto azure yet, they just announced it's their goal to move over in 2026
- kennysmoothx 1y agoFYI in an emergency you can edit files directly on Github without the need to use git. Edit: ugh... if you rely on GH Actions for workflows though actions/checkout@v4 is also currently experiencing the git issues, so no dice if you depend on that.
- ruuda 1y agoFYI in an emergency you can `git push` to and `git pull` from any SSH-capable host without the need to use GitHub.
- cluckindan 1y agoFYI in an emergency you can SSH to your server and edit files and the DB directly. Where is your god now, proponents of immutable filesystems?!
- BadBadJellyBean 1y agoI love when people do that because they always say "I will push the fix to git later". They never do and when we deploy a version from git things break. Good times. I started packing things into docker containers because of that. Makes it a bit more of a hassle to change things in production.
- noir_lord 1y agoDepends on the org, the big ones I've worked for regular Devs even seniors don't have anything like the level of access to be able to pull a stunt like that. At the largest place I did have prod creds for everything because sometimes they are necessary and I had the seniority (sometimes you do need them in a "oh crap" scenario). They where all setup on a second account in my work Mac which had a danger will Robinson wallpaper because I know myself, far far too easy to mentally fat finger when you have two sets of creds.
- 1718627440 1y agoIt's easier, when the deployed instance on the server is also a Git repo, so you can commit the changes there and pull.
- zackify 1y agothis is actually the 5-6th time this month. actions have been degraded constantly now push and pull breaks back to back
- deleted 1y ago[deleted]
- etchalon 1y agoWhat is today and who do I blame for it
- baq 1y agoComputers are great at solving problems that wouldn’t have existed without computers
- stronglikedan 1y agoComputers and alcohol.
- deleted 1y ago[deleted]
- saydus 1y agoCherry on top will be another aws outage
- linsomniac 1y agoFunny you should say that, I'm here looking because our monitoring server is seeing 80-90% packet loss on our wireguard from our data center to EC2 Oregon...
- linsomniac 1y agoFYI: Not AWS. Been doing some more investigation, it looks like it's either at our data center, or something on the path to AWS, because if I fail over to our secondary firewall it takes a slightly different path both internally and externally, but the packet loss goes away.
- SimoncelloCT 1y agoSame issue, and I need to complete my work :(
- imdsm 1y agoCloudflare, GitHub...
- personjerry 1y agoLooks like Gemini 3 figured out the best way to save costs on its compute time was to shut down github!
- ssawchenko 1y agoSame. ERROR: no healthy upstream fatal: Could not read from remote repository.
- whynotmaybe 1y agoWe gonna need xkcd "compiling" but with "cloudflare||github||chatgpt||spotify down". https://xkcd.com/303/ https://xkcd.com/303/
- sre2025 1y agoWhy are there outages everywhere all the time now? AWS, Azure, GitHub, Cloudflare, etc. Is this the result of "vibe coding"? Because before "vibe coding", I don't remember having this many outages around the clock. Just saying.
- elicash 1y agoI think it has more to do with layoffs. "Why do we need so many people to keep things running!?! We never have downtime!!"
- Refreeze5224 1y agoWhich the true reason for AI, reducing payroll costs.
- themafia 1y agoThe reason I detest those who push AI as a technological solution. I think AI as a field is interesting but highly immature, but it's been over hyped to the point of absurdity, and now it is having real negative pressure on wages. That pressure has carry over effects and I agree that we're starting to observe those.
- brovonov 1y agoHas to be a mix of both.
- noosphr 1y agoThey fired a ton of employees with no rhyme or reason to cut costs, this was the predictable outcome. It will get worse if it ever gets better. The funny thing is that the over hiring during the pandemic also had the predictable result of mass lay-offs. Whoever manages HR should be the ones fired after two back to back disasters like this.
- baq 1y agoAnd yet we keep paying the company
- _pdp_ 1y agoIs it just me or it seems that there is an increased frequency of these types of incidents as of late.
- RGamma 1y agoICE keeps finding immigrants in the server cabinets.
- 0dayman 1y agoGithub is down a lot...
- laurentiurad 1y agoA lot of failures lately during the aI ReVoLuTiOn.
- SOLAR_FIELDS 1y agoGitHub has a long history of garbage reliability that long predates AI
- matkv 1y agoJust as I was wondering why my `git push` wasn't working all of a sudden :D
- chrsstrm 1y agoI thought I was going crazy when I couldn't push changes but now it seems it's time to just call it for the day. Back at it tomorrow.
- peciulevicius 1y agosame, i've started pulling my hair out, was about to nuke my setup and set it up all from scratch
- keepamovin 1y agolol same. Hilarious when this shit goes down that we all rely on like running water. I'm assuming GitHub was hacked by the NSA because someone uploaded "the UFO files" or sth.
- Mossly 1y agoSeeing auth succeed but push fail was an exercise in hair pulling.
- curioussquirrel 1y agoSame, even started adding new ssh keys to no avail... (I was getting some nondescript user error first, then unhealthy upstream)
- chrsstrm 1y agoWould love to see a global counter for the number of times ‘ssh -T git@github.com’ was invoked.
- mepage 1y agoSeeing "ERROR: no healthy upstream" in push/pull operations
- _jab 1y agoGitHub is pretty easily the most unreliable service I've used in the past five years. Is GitLab better in this regard? At this point my trust in GitHub is essentially zero - they don't deserve my money any longer.
- noosphr 1y agoYou can make it as reliable as you want by hosting it on prem.
- deleted 1y ago[deleted]
- themafia 1y agoFlashbacks to me pushing hard for GitLab self hosting a few months ago. The rest of the team did not feel the lift was worth it. I utterly hate being at the mercy of a third party with an after thought of a "status page" to stare at.
- cindyllm 1y ago[dead]
- jakub_g 1y ago> as reliable as you want We self-host GitLab but the team owning it is having hard time scaling it. From my understanding talking to them, the design of gitaly makes it very hard to scale it beyond certain repo size and # of pushes per day (for reference: our repos are GBs in size, ~1M commits, hundreds of merges per day)
- deleted 1y ago[deleted]
- loloquwowndueo 1y agoForgejo, my dudes.
- mepage 1y agoSeeing "ERROR: no healthy upstream" in push/pull.
- dadof4 1y agoSame for me, fatal: unable to access 'https://github.com/repository_example.git/ https://github.com/repository_example.git/': The requested URL returned error: 500
- projproj 1y agoObviously just speculation, but maybe don't let AI write your code... Microsoft CEO says up to 30% of the company’s code was written by AI https://techcrunch.com/2025/04/29/microsoft-ceo-says-up-to-30-of-the-companys-code-was-written-by-ai/ https://techcrunch.com/2025/04/29/microsoft-ceo-says-up-to-3...
- dollylambda 1y agoMaybe AI is the tech support too
- Aloisius 1y agoSweet. 30% of Microsoft's code isn't protected by copyright. Time to leak that.
- tauchunfall 1y agoIt's degraded availability of Git operations. The enterprise cloud in EU, US, and Australia has no issues. If you look at the incident history disruptions happen often in the public cloud for years already. Before AI wrote code for them.
- TimTheTinker 1y agoThe enterprise cloud runs on older stable versions of GitHub's backend/frontend code.
- smsm42 1y agoThat sounds very bad, but I guess it depends also on which code it is. And whether Nadella actually knows what he's talking about, too.
- angrydev 1y agoWhat a ridiculous comment, as if these outages didn't happen before LLMs became more commonplace.
- malfist 1y ago
- case0x 1y agoI wish I could say something smart such as “People/Organisations should host their own git servers“, but as someone who had the misfortune of doing that in the past I rather have a non-functional GitHub.
- mkreis 1y agoI'm curious to learn from your mistakes, can you please elaborate what went wrong?
- Mossly 1y agoI've found Gitea to be pretty rock solid, at least for a small team.
- gelbphoenix 1y agoWould even recommend Forgejo (the same project Codeberg also uses as the base for their service)
- lol768 1y ago> We are seeing failures for some git http operations and are investigating It's not just HTTPS, I can't push via SSH either. I'm not convinced it's just "some" operations either; every single one I've tried fails.
- olivia-banks 1y agoA friend of mine was able to get through a few minutes ago, apparently. Everyone else I know is still fatal'ing.
- deathanatos 1y agoI'm convinced the people who write status pages are incapable of escaping the phrasing "Some users may be experiencing problems". Too much attempting to save face by PR types, instead of just being transparent with information (… which is what would actually save face…) And that's if you get a status page update at all.
- silverwind 1y agoIt's not only http, also ssh.
- deleted 1y ago[deleted]
- alexskr 1y agoMandatory break time has officially been declared. Please step away from your keyboard, hydrate, and pretend you were productive today.
- brovonov 1y agoGood thing I already moved away from gh to a selfhosted Forgejo instance.
- SteveNuts 1y agoI have a serious question, not trying to start a flame war. A. Are these major issues with cloud/SaaS tools becoming more common, or is it just that they get a lot more coverage now? It seems like we see major issues across AWS, GCP, Azure, Github, etc. at least monthly now and I don't remember that being the case in the past. B. If it's becoming more common, what are the reasons? I can think of a few, but I don't know the answer, so if anyone in-the-know has insight I'd appreciate it. Operations budget cuts/layoffs? Replacing critical components/workflows with AI? Just overall growing pains, where a service has outgrown what it was engineered for? Thanks
- kkarpkkarp 1y ago> If it's becoming more common, what are the reasons? Someone answered this morning, while Cloudflare outage, it's AI vibe coding and I tend to think there is something true in this. At some point there might be some tiny grain of AI engaged which starts the avalanche ending like this.
- junon 1y agoBeen on GitHub for a long time. It feels like they're more often. It used to be yearly if at all that GitHub was noticably impacted. Now it's monthly, and recently, seemingly weekly.
- cmrdporcupine 1y agoIn the early days of GitHub (like before 2010) outages were extremely common.
- junon 1y agoNot from my recollection. Not like this. BitBucket on the other hand had a several day outage at one point. That one I do recall.
- sampullman 1y agoI remember periods of time when GitHub was down every few weeks, my impression is that it's become more stable over the years.
- 85392_school 1y agoSeeing "404: Not Found" for all raw files
- keepamovin 1y agoIt's weird to think all of our data lives on physical servers (not "in the cloud") that are falliable and made and maintained by falliable humans, and could fail at any moment. So long to all the data! Good ol' byzantine backups.
- mandus 1y agoGood thing git was designed as a decentralized revision control system, so you don’t really need GitHub. It’s just a nice convenience
- ElijahLynn 1y agoYou just lose the "hub" of connecting others and providing a way to collaborate with others with rich discussions.
- parliament32 1y agoAll of those sound achievable by email, which, coincidently, is also decentralized.
- Aurornis 1y agoSome of my open source work is done on mailing lists through e-mail It's more work and slower. I'm convinced half of the reason they keep it that way is because the barrier to entry is higher and it scares contributors away.
- octoberfranklin 1y agoWell it does prevent brigading.
- deleted 1y ago[deleted]
- awesome_dude 1y agoWait, email is decentralised? You mean, assuming everyone in the conversation is using different email providers. (ie. Not the company wide one, and not gmail... I think that covers 90% of all email accounts in the company...)
- drykjdryj 1y agoEmail at a company is very not decentralized. Most use Microsoft 365, also hosted in azure, i.e. the same cloud as github is trying to host its stuff in.
- mysteria 1y agoCloudflare this morning, and now this. A bunch of work isn't getting done today. Maybe this will push more places towards self-hosting?
- mendyberger 1y agoAfter restarting my computer, reinstalling git, almost ready to reinstall my os, I find out it's not even my fault
- OptionOfT 1y agoIt is insane how many failures we've been getting lately, especially related to actions. * jobs not being picked up * jobs not being able to be cancelled * jobs running but showing up as failed * jobs showing up as failed but not running * jobs showing containers as pushed successfully to GitHub's registry, but then we get errors while pulling them * ID token failures (E_FAIL) and timeouts. I don't know if this is related to GitHub moving to Azure, or because they're allowing more AI generated code to pass through without proper reviews, or something else, but as a paying customer I am not happy.
- veighnsche 1y agoProbably because AI-generated reviews has made qa way worse.
- manbitesdog 1y agoSame! The current self-hosted runner gets hung every so often
- lherron 1y agoGemini 3 = Skynet ?
- spapas82 1y agoHaving a self hosted gitea server is a godsend in times like this!
- broosted 1y agocan't do git pull or git push 503 and 500 errors
- consumer451 1y agoWe live in a house of cards. I hope that eventually people in power realize this. However, their incentive structures do not seem to be a forcing function for that eventuality. I have been thinking about this a lot lately. What would be a tweak that might improve this situation?
- mistercheph 1y agoUsing p2p or self hosted, and accepting the temporary tradeoffs of no network effects.
- sznio 1y agoNot exactly for this situation, but I've been thinking about distributed caching of web content. Even if a website is down, someone somewhere most likely has it cached. Why can't I read it from their cache? If I'm trying to reach a static image file, why do I have to get it from the source? I guess I want torrent DHT for the web.
- consumer451 1y agoThat is genuinely interesting. But, let's put all "this nerd talk" into terms that someone in the average C-suite could understand. How can C-suite stock RSU/comp/etc be tweaked to make them give a crap about this, or security? --- Decades ago, I was a teenager and I realized that going to fancy hotel bars was really interesting. I looked old enough, and I was dressed well. This was in Seattle. I once overheard a low-level cellular company exec/engineer complain about how he had to climb a tower, and check the radiation levels (yes non-ionizing). But this was a low level exec, who had to take responsibility. He joked about how while checking a building on cap hill, he waved his wand above his head, and when he heard the beeps... he noped tf out. He said that it sucked that he had to do that, and sign-off. That is actually cool, and real engineering/responsibility at the executive level. Can we please get more of that type of thing?
- rewgs 1y agoI think that that kind of domain knowledge and getting your hands dirty is more necessary when you're actually having to solve real problems that real people pay real money for -- money that isn't able to be borrowed for free. It's no coincidence that the clueless MBA who takes pride in knowing nothing about the business they're apart of proliferated during economic "spring time" -- low interest rates, genuine technological breakthroughs to capitalize on, early mover advantage, etc. When everyone is swimming in money, it's easier to get a slice without adequately proving why you deserve it. Now we're in "winter." Interest rates are high, innovation is debatably slowing, and the previous early movers are having to prove their staying power. All that to say: the bright side, I hope, of this pretty shitty time is that hopefully we don't _need_ to "put all this nerd talk into terms that someone in the average C-suite could understand," because hopefully the kinds of executives who are simultaneously building and running _tech companies_ and who are allergic to "nerd talk" will very simply fail to compete. That's the free market (myth as it may often be in practice) at work -- those who are totally uninterested in the subject matter of their own companies aren't rewarded for their ignorance.
- stuffn 1y agoCentralized internet continues to show it's wonderful benefits. At least microsoft decided we all deserve a couple hour break from work.
- netsharc 1y agoI remember a colleague setting up a CI/CD system (on an aaS obviously) depending on Docker, npm, and who knows what else... I thought "I wonder what % of time all those systems are actually up at the same time"
- chazeon 1y agoSeem images on GitHub web also not showing
- theoldgreybeard 1y agocan the internet work for 5 minutes, please?
- Argonaut998 1y agoMercury is in retrograde
- WesolyKubeczek 1y agoSo that’s how the Azure migration is going.
- pyenvmanger 1y agoGit pull and push not working
- pyenvmanger 1y agoGit push and pull not working. Getting a 500 response.
- bstsb 1y agoside effect that isn't immediately obvious: all raw.githubusercontent.com content responds with a "404: Not Found" response. this has broken a few pipeline jobs for me, seems like they're underplaying this incident
- pm90 1y agoyeah something major is borked and they're unwilling to admit it. The status page initially claimed "https git operations are affected" when it was clear that ssh were too (its updated to reflect that now).
- treeroots 1y agowhat else is out there like github?
- parliament32 1y agoGitLab is probably the next-largest competitor. Unlike GitHub it's actually open source, so you can use their managed offering or self-host.
- kragen 1y agoGitlab, Forgejo, Gitea, Gogs, ... or you can just push to your own VPS over SSH, with or without an HTTP server. We had a good discussion of this last option here three weeks ago: https://news.ycombinator.com/item?id=45710721 https://news.ycombinator.com/item?id=45710721
- robowo 1y agoGive https://codeberg.org/ https://codeberg.org/ a go
- deleted 1y ago[deleted]
- olivia-banks 1y agoThis is incredibly annoying. I've been trying to fix a deployment action on GitHub for a the past bit, so my entire workflow for today has been push, wait, check... push, wait, check... et cetera.
- luca616 1y agoYou should really check out (pun intended) `act` https://github.com/nektos/act https://github.com/nektos/act
- olivia-banks 1y agoI’ve tried! Most I’ve ever gotten was an inefficient way to fill some disk space and an ‘act’ that didn’t work :-)
- kevinlajoye 1y agoPain My guess is that it has to do with the Cloudflare outage this morning.
- lenerdenator 1y agoIt would be nice if this was actually broken down bit-by-bit after it happened, if only for paying customers of these cloud services. These companies are supposed to have the top people on site reliability. That these things keep happening and no one really knows why makes me doubt them. Alternatively, The takeaway for today: clearly, Man was not meant to have networked, distributed computing resources. We thought we could gather our knowledge and become omniscient, to be as the Almighty in our faculties. The folly. The hubris. The arrogance.
- swedishuser 1y agoAlmost one hour down now. What differs this from recent AWS and Cloudflare issues is that this appears to be a global issue?
- angrydev 1y agoTon of people in the comments here wanting to blame AI for these outages. Either you are very new to the industry or have forgotten how frequently they happen. Github in particular was a repeat offender before the MS acquisition. us-east-1 went down many times before LLMs came about. Why act like this is a new thing?
- Aeroi 1y agojust realized my world stops, when github does.
- mrguyorama 1y agoI'm going to awkwardly bring up that we have avoided all github downtime and bugs and issues by simply not using github. Our git server is hosted by Atlassian. I think we've had one outage in several years? Our self hosted Jenkins setup is similarly robust, we've had a handful of hours of "Can't build" in again, several years. We are not a company made up of rockstars. We are not especially competent at infrastructure. None of the dev teams have ever had to care about our infrastructure (occasionally we read a wiki or ask someone a question). You don't have to live in this broken world. It's pretty easy not to. We had self hosted Mercurial and jenkins before we were bought by the megacorp, and the megacorp's version was even better and more reliable. Self host. Stop pretending that ignoring complexity is somehow better.
- hyperb1iss 1y agoI wonder how much of this stuff has been caused by AI agents running on the infra? Claude Code is amazing for devops, until it kubectl deletes your ArgoCD root app
- whinvik 1y agoHaha I don't know if its a good test or not but I could not figure out why git pull was failing and Claude just went crazy trying so many random things. Gemini 3 Pro after 3 random things announced Github was the issue.
- randall 1y agocan’t go down is better than won’t go down. the problem isn’t with centralized internet services, the problem is a fundamental flaw with http and our centralized client server model. the solution doesn’t exist. i’ll build it in a few years if nobody else does.
- usui 1y agoIt's working again now.
- arbol 1y agoIts back
- ashishb 1y agoI have said this before, and I will say this again: GitHub stars[1] are the real lock-in for GitHub. That's why all open-core startups are always requesting you to "star them on GitHub". The VCs look at stars before deciding which open-core startup to invest in. The 4 or 5 9s of reliability simply do not matter as much. 1 - https://news.ycombinator.com/item?id=36151140 https://news.ycombinator.com/item?id=36151140
- theideaofcoffee 1y agoMan, I sound like a broken record, but... Love that for them. How many more outages until people start to see that farming out every aspect of their operations maybe, might, could have a big effect on their overall business? What's the breaking point? Then again, the skills to run this stuff properly are getting more and more rare so we'll probably see more and more big incidents popping up more frequently like this as time goes on.
- aeldidi 1y agoI'm becoming concerned with the rate at which major software systems seem to be failing as of late. For context, last year I only logged four outages that actually disrupted my work; this quarter alone I'm already on my fourth, all within the past few weeks. This is, of course, just an anecdote and not evidence of any wider trend (not to mention that I might not have even logged everything last year), but it was enough to nudge me into writing this today (helped by the fact that I suddenly had some downtime). Keep in mind, this isn't necessarily specific to this outage, just something that's been on my mind enough to warrant writing about it. It feels like resiliency is becoming a bit of a lost art in networked software. I've spent a good chunk of this year chasing down intermittent failures at work, and I really underestimated how much work goes into shrinking the "blast radius", so to speak, of any bug or outage. Even though we mostly run a monolith, we still depend on a bunch of external pieces like daemons, databases, Redis, S3, monitoring, and third-party integrations, and we generally assume that these things are present and working in most places, which wasn't always the case. My response was to better document the failure conditions, and once I did, realize that there was many more than we initially thought. Since then we've done things like: move some things to a VPS instead of cloud services, automate deployment more than we already had, greatly improve the test suite and docs to include these newly considered failure conditions, and generally cut down on moving parts. It was a ton of effort, but the payoff has finally shown up: our records show fewer surprises which means fewer distractions and a much calmer system overall. Without that unglamorous work, things would've only grown more fragile as complexity crept in. And I worry that, more broadly, we're slowly un-learning how to build systems that stay up even when the inevitable bug or failure shows up. For completeness, here are the outages that prompted this: the AWS us-east-1 outage in October (took down the Lightspeed R series API), the Azure Front Door outage (prevented Playwright from downloading browsers for tests), today’s Cloudflare outage (took down Lightspeed’s website, which some of our clients rely on), and the Github outage affecting basically everyone who uses it as their git host.
- HardwareLust 1y agoIt's money, of course. No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray.
- agnishom 1y ago(epistemic status: frustrated, irrational) This is what happens when they decide that all the budget should be spent on AI stuff rather than solid infra and devops
- captainkrtek 1y agoReflecting on the last decade, with my career spanning big tech and startups, I've seen a common arch: Small and scrappy startup -> taking on bigger customers for greater profits / ARR -> re-architecting for "enterprise" customers and resiliency / scale -> more idealism in engineering -> profit chasing -> product bloat -> good engineers leave -> replaced by other engineers -> failures expand. This may be an acceptable lifecycle for individual companies as they each follow the destiny of chasing profits ultimately. Now picture it though for all the companies we've architected on top of (AWS, CloudFlare, GCP, etc.) Even within these larger organizations, they are comprised of multiple little businesses (eg: EC2 is its own business effectively - people wise, money wise) Having worked at a $big_cloud_provider for 7 yrs, I saw this internally on a service level. What started as a foundational service, grew in scale, complexity, and architected for resiliency, slowly eroded its engineering culture to chase profits. Fundamental services becoming skeletons of their former selves, all while holding up the internet. There isn't a singular cause here, and I can't say I know what's best, but it's concerning as the internet becomes more centralized into a handful of players. tldr: how much of one's architecture and resiliency is built on the trust of "well (AWS|GCP|CloudFlare) is too big to fail" or "they must be doing things really well"? The various providers are not all that different from other tech companies on the inside. Politics, pressure, profit seeking.
- Esophagus4 1y agoWell said. I definitely agree (you’re absolutely right!) that the product will get worse through that re-architecting for enterprise transition. But the small product also would not be able to handle any real amount of growth as it was, because it was a mess of tech debt and security issues and manual one-off processes and fragile spaghetti code that only Jeff knows because he wrote it in a weekend, and now he’s gone. So by definition, if a service is large enough to serve a zillion people, it is probably big and bloated and complex. I’m not disagreeing with you, I liked your comment and I’m just rambling. I have worked with several startups and was surprised at how poorly their tech scaled (and how riddled with security issues they were) as we got into it. Nothing will shine a flashlight on all the stress cracks of a system like large-scale growth on the web.
- JonChesterfield 1y agoWhat's the local workaround for this? Git is distributed, it should be possible to put something between our servers and github which pulls from github when it's running and otherwise serves whatever it used to have. A cache of some sort. I've found the five year old https://github.com/jonasmalacofilho/git-cache-http-server https://github.com/jonasmalacofilho/git-cache-http-server which is the same sort of idea. I've run a git instance on a local machine which I pull from, where a cron job fetches from upstream into it, which solved the problem of cloning llvm over a slow connection, so it's doable on a per-repo basis. I'd like to replace it globally though because CI looks like "pull from loads of different git repos" and setting it up once per-repo seems dreadful. Once per github/gitlab would be a big step forward.
- grepfru_it 1y agoThere was a comment on another GitHub thread that I replied to. I got a response saying it’s absurd how unreliable Gh is when people depend on it for CI/CD. And I think this is the problem. At GitHub the developers think it’s only a problem because their ci/cd is failing. Oh no, we broke GitHub actions, the actions runners team is going to be mad at us! Instead of, oh no, we broke GitHub actions, half the world is down! That larger view held only by a small sliver of employees is likely why reliability is not a concern. That leads to the every team for themselves mentality. “It’s not our problem, and we won’t make it our problem so we don’t get dinged at review time” (ok that is Microsoft attitude leaking) Then there’s their entrenched status. Real talk, no one is leaving GitHub. So customers will suck it up and live with it while angry employees grumble on an online forum. I saw this same attitude in major companies like Verio and Verisign in the early 2000s. “Yeah we’re down but who else are you going to go to? Have a 20% discount since you complained. We will only be 1% less profitable this quarter due to it” The kang and kodos argument personified. These views are my own and not related to my employer or anyone associated with me.
- h4kunamata 1y agoWith Microsoft (Behind GitHub) going full AI mode, expected things to get worse. I worked for one of the largest company in my country, they had "catch-up" with GitHub and it is not longer about GitHub as you folks are used to but AI aka CoPilot. We are seeing major techs such as but not limited to Google, AWS and Azure going under after making public that their code is 30% AI generated (Google). Even Xbox(Microsoft) and its gaming studio got destroyed (COD BO7) for heavily dependency on AI. Don't you find it coincidence all of these system outage worldwide happening right after they proudly shared heavily dependency on AI?? Companies aren't using AI/ML to improve processes but to replace people, full stop. The AI stock market is having a massive meltdown as we speak with indications that the AI bubble went live. If you as a company wanna keep your productivity at 99.99% from now on: * GitLab: Self-hosted GitLab/runners * Datacenter: AWS/GCP/Azure is no longer a safe option or cheaper, we have data center companies such as Equinix which have a massive backup plan in place. I have visited one, they are prepared for a nuclear war and I am not even being dramatic. If I was starting a new company in 2025, I would go back to datacenter over AWS/GCP/Azure * Self-host everything you can, and no, it does not require 5 days in the office to manage all of that.
- amazingman 1y agoI didn't see a case made for self-hosting as the better option, instead I see that proposition being assumed true. Why would it be better for my company to roll its own CI/CD?
- h4kunamata 1y agoI worked at a bank that self-hosted GitLab/runners. As the AI bubble goes sideways, you don't know how your company data is being held, CoPilot uses GitHub to train its AI for instance. Yes, the big company I work for had a clause to forbids GitHub from using the company's repo from AI training. How many companies can afford having a dedicated GitHub team to speak to?? How many companies read the contracts or have any saying?? Not many really. Yeah sure, cloud is easier, you just pay the bills, but at what cost??
- rurban 1y agoSo was this caused by the Cloudflare duplicate features file, causing 5xx errors internally? They said they shipped a fix, but no details yet
- Phelinofist 1y agoRandom unrelated reminder that GitHub still doesn't support IPv6 yet.