6 ms·
GitHub pull requests were down
- SwiftyBug 1y agoRight in the middle of a huge rebase. Great!
- AdventureMouse 1y agoIt’s never a good time for GitHub to be down!
- 1718627440 1y agoHow does an outage of a remote repo affects your local rebase.
- deleted 1y ago[deleted]
- pelagicAustral 1y agoGood thing I always commit directly to the main branch.
- mattwad 1y agothis is broken too!
- mmastrac 1y agoGood thing we're using a shared Samba drive and editing files directly without locks!
- SparkyMcUnicorn 1y agoProject_v2_final3 is looking good, but remember to grab the new actionscript files out of Project_v2_final4 as well.
- hnuser123456 1y agoAlso the foo_bar method from v1 worked better so pull that back in
- rwmj 1y agoYou joke, but when I was doing my start-up we made good money on the side from monitoring websites to detect when the designers had pushed regressions to the live site. We would keep track of change requests that were filed and resolved, then scripts would monitor the sites to see if any earlier changes had been backed out. (Getting the designers to use version control was considered to be in the "too hard" bucket. This was back in the mid 2000s.)
- ZiiS 1y agoStill better than CVS then /s
- taude 1y agoSo glad we never bothered to migrate from Visual Source Safe
- vehemenz 1y agoMake sure everyone has caching disabled, for maximum effect
- tetha 1y agoWe have post-its with file names on a wall in the office. You take one down if you edit the file, and put it back up when you're done. Easy. Though I wish I was entirely kidding. ~12 years ago or so we did that if one of two parallel development teams had to modify a message of the network protocol to avoid incompatibilities and merge problems. Mind you, these were SVN merges. I can't even verbalize my feelings about SVN merges but by a mixture of laughing and groaning in pain, like if you stubbed your toe in a painful, but entirely funny way.
- spicybright 1y agoKeeping it when tech can't keep up is genuinely a good hack for any kind of engineering. Physical lock out tag out on industrial machines for instance. Passing paper notes/wooden blocks in air traffic control towers to see who's responsible for what even if computers go down.
- xorcist 1y agoWhat is this eternal meme about merges in svn being harder than in other tools? Git used literally the same merge algorithm, even if that has changed a bit since then, and merge conflicts are not something a tool can't just magically make disappear. If you want concurrent edits (the c in cvs), conflicts come in the same package. Various algortihms can supply their own dose of magic, but they're more similar than different (minus a few special cases such as rerere in git).
- tetha 1y agoMy interpretation within that company: You know this new idea of "If it's painful, do it more"? People in that company didn't do that in the SVN days or earlier, because merges were painful. Thus, merges filled a sprint if they had to be done. This made sense if you came from CSV or nothing, tbh. Git in turn made branches easier, causing merges to be more prevalent and developers overall learned to merge more, merge more often.
- xorcist 1y ago
- davey48016 1y agoThat's a single point of failure. If you email code changes around and use an email client that copies everything offline, then the history of your code base is distributed across all of your developers' laptops.
- escapecharacter 1y agoThey must mean their local main branch.
- RonanSoleste 1y agoNo the remote one. No need for a local branch.
- jaredsohn 1y agoGood thing we just SSH into production and make the changes live.
- gloxkiqcza 1y agoVibe coding nonetheless #gofastandbreakthings
- redserk 1y agoSubtle Elixir/Erlang advocacy here.
- rwmj 1y agoYou're using Ansible?
- hdgvhicv 1y agoDo it manual you can screw up one server at a time Run ansible and you can screw it all up
- phendrenad2 1y agorsync is all you need ;)
- jaredsohn 1y agonah - ftp or run vi from ssh
- deleted 1y ago[deleted]
- antihero 1y agoGuess it's time to embrace AI.
- dorian-graph 1y agoContext: https://www.finalroundai.com/blog/github-ceo-thomas-dohmke-warns-developers-embrace-ai-or-quit https://www.finalroundai.com/blog/github-ceo-thomas-dohmke-w...
- sleepydog 1y ago... or get out.
- njovin 1y agoProps to Github for having an accurate status page. AWS and Google should take note.
- ietktnz 1y agoStatus page says "Incident with Pull Requests". Pull requests status is listed as "Normal". Status text says issue with degraded performance for Webhooks and Issues, does not mention Pull Requests. I would give that a 5/10 accuracy at best!
- samgranieri 1y agothey've updated the page since then. Take a look
- ericyan 1y agoThe status page has been updated. PR and webhook statused red and now listed as "Incident". (Disclosure: GitHub employee)
- sleepybrett 1y agoAs someone who is partially responsible for supporting github at a very large organization, no it isn't. At least not until the incident is at least 30m old if ever.
- hardworker02 1y agoUh.. pub?
- crinkly 1y agoAlready in it. Was a premonition.
- MattGaiser 1y agoGitHub gives everyone an extra long lunch.
- trashburger 1y agoEarly EOD for me!
- deleted 1y ago[deleted]
- arccy 1y agoI guess they let copilot review their code
- hiccuphippo 1y agoMaybe they are trying vibeops now.
- jennyholzer 1y agoAt Microsoft vibeops is an age old tradition.
- shakna 1y agoWell, the CEO did say to embrace AI or get out of code, 2 days ago... And MS previously said AI is not-optional for their devs...
- DeepYogurt 1y agoThey do actually
- esafak 1y agoAfter writing it :)
- crinkly 1y agoNo excuse. git-send-email out and stop slacking :)
- thimabi 1y agoI’ll be waiting expectantly for the post mortem of this. How ironic would it be if this issue was caused by a pull request itself?
- KoboldAdvocate 1y agoNot a good look when they're heavily pushing AI agents.
- tzury 1y agohttps://www.githubstatus.com https://www.githubstatus.com (+WebHooks) (+Issues)
- Imustaskforhelp 1y agoThis is why I recommend decentralized protocols like radical or I guess I hope that tangled.sh could fix this stuff too. I am not sure about tangled.sh, I might ask them in their discord about this now y'know.
- lionkor 1y agoGit is a decentralized protocol, it's just incomplete IMO
- 2OEH8eoCRo0 1y agoGit has a protocol called email.
- immibis 1y agoIndeed push/pull wasn't even the original design intention of Git. It was a tool for one person to manage trees of emailed patch files.
- pmontra 1y agoThere is git format-patch to create a diff and git send-email [2] to mail it to another developer and git-am [3] to apply the patches from a mailbox. The Linux kernel developers have been using that workflow for a lot of time. Maybe still now. [1] https://git-scm.com/docs/git-format-patch https://git-scm.com/docs/git-format-patch [2] https://git-scm.com/docs/git-send-email https://git-scm.com/docs/git-send-email [3] https://git-scm.com/docs/git-am https://git-scm.com/docs/git-am
- lucasoshiro 1y ago> The Linux kernel developers have been using that workflow for a lot of time Yes, they do. Git itself is also developed that way. Btw, you don't need to use format + send-email, send-email calls format-patch under the hood
- rightbyte 1y ago
- ItsABytecode 1y agoI didn’t think the code I just merged was that bad
- noreplydev 1y agohttps://www.githubstatus.com/ https://www.githubstatus.com/ "git operations: degraded", my git operations are degraded by default
- rileymichael 1y agodid all of the devs leave? https://www.businessinsider.com/github-ceo-developers-embrace-ai-or-get-out-2025-8 https://www.businessinsider.com/github-ceo-developers-embrac...
- maerF0x0 1y agocheaper than layoffs.
- xyse53 1y agoDevs leaving can often be a stability boost :)
- doubled112 1y agoRight up until it isn't.
- stevefolta 1y agoBut if that's what they want, they may be driving out the exact wrong subset of their devs.
- ryandrake 1y ago> Instead of selling products based on helpful features and letting users decide, executives often deploy scare tactics that essentially warn people they will become obsolete if they don't get on the AI bandwagon. For instance, Julia Liuson, another executive at Microsoft, which owns GitHub, recently warned employees that "using AI is no longer optional." So many clowns. It's like everyone's reading from the same script/playbook. Nothing says "this tool is useful" quite like forcing people to use it.
- bogzz 1y agoIt definitely feels like the imbecility of the corporate class has reached new levels.
- 1y ago
- axelpacheco 1y agoGive your best estimate on how much dollar value of creation is wasted every hour GitHub PRs are down
- StarlaAtNight 1y agoOne MILLION dollars puts pinky to corner of mouth
- bob1029 1y agoI estimate that on some days an outage like this could ultimately save some businesses money. There's a lot of cowboy development going on out there. Why not take this opportunity to talk to your customers for a bit? Make sure you're still building the right things.
- henryfjordan 1y agoAt a startup, sure. At any decently-sized org, the developers are not allowed to talk to customers on their own accord.
- bob1029 1y agoI've never worked somewhere I couldn't email a customer as long as the team was CC'd. This is a bit of a circular problem because if you don't get exposed to the customer in some capacity you'll never get good at working with them. If the business is afraid to let you email the customer, you might need to work on your communications skills and go through some intentional demonstration efforts. For example, "Good morning <boss>, here's a draft of what I think we should send <CTO's name @ customer> regarding their feedback on the last build.". That's literally all it takes to get into the game. Don't ask for permission to write the draft because then your managers will think it's gonna be this big ordeal and they'll definitely say no.
- henryfjordan 1y agoAt a B2B, I might agree with you. At a B2C, I would not email a customer directly without sign-off. We have marketing teams, research teams, comms, customer support, etc. I would be stepping on so many toes, and risking brand reputation, if I were to interact with our customers.
- darth_avocado 1y agoDupe: https://github.com/github/site-policy/pull/582 https://github.com/github/site-policy/pull/582
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- greenie_beans 1y agodon't wanna be spreading fake news, but i wonder if this is related to a cloudflare issue? i've been unable to login to cloudflare for the past ~30 minutes. and: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/
- jdthedisciple 1y agoThis is strange: I was just having issues with Pull Requests on BitBucket too. Coincidence, actually?
- aduwah 1y agoIt's all a central svn in AWS
- elmwoodfire3 1y ago[dead]
- politelemon 1y agoDoes GitHub development happen on GitHub? And if the fix for broken pull requests requires a pull request would they have a way to review it...
- naikrovek 1y agoif GitHub Enterprise Server is anything to go by, they build (almost) everything for containers, and the entire site is hosted by containers managed by Nomad. So there are probably lots of older images around that they can fall back on if the latest image of any container causes problems. How they would deploy the older container, I don't know. A lot of this is guesswork, I don't work for them or anything. And I know that GHES in the way that my employer manages it is very unlike the way that GitHub host github.com, so everything i've assumed could be wrong.
- deleted 1y ago[deleted]
- gimenete 1y agoI worked there for 3 years and yes GitHub development happens on github.com. Of course there’s ways to deploy and rollback changes while the site is down but that’s very unusual. The typical flow happens in github.com and uses the regular primitives everybody uses: prs, ci checks, etc. The pipeline for deploying the monolith doesn’t happen in GitHub Actions though but in a service based in jenkins. Fun fact: playbooks for incidents used to be hosted in GitHub too but we moved them after an incident that made impossible to access them while it lasted.
- coryrc 1y ago> that made impossible to access them Couldn't they just be checked out by cron on any number of local machines hosting Apache?
- gimenete 1y agoI don't remember clearly where we moved them. It was probably to something owned by Google (because GitHub uses Google Workspaces) or Microsoft (for obvious reasons).
- lrvick 1y agoDoes not impact me, because my team and I self-host Forgejo for all our work. People seem to forget Git was meant to be decentralized.
- redrove 1y agoYes, but you may work with other people, other organizations, or at least depend on open source code that's hosted on GitHub. I agree with the sentiment though.
- lrvick 1y agoDo work in and rely on self hosted forks so you are not blocked, and upstream when upstream code submissions become possible again.
- 8n4vidtmkvmk 1y agoI found this hilariously confusing when I first heard about DVCSs. I'm like ok... So they're "distributed".. how do I share the code with other people? Oh..I push to a central repository? So it's almost exactly like SVN? Cool cool.
- maerF0x0 1y agoweird... this is redirecting me to `Privacy Statement Updates September 2022 #582` https://github.com/github/site-policy/pull/582 https://github.com/github/site-policy/pull/582
- crazysim 1y agoIt was probably just an example.
- csh602 1y agoSeems total downtime was from 15:51 to 16:14 UTC
- mintaka5 1y ago[dead]
- sidcool 1y agoIt's up now.
- eats_indigo 1y agoGiven Github's critical role in software engineering delivery, their SLA commitments are really quite poor, perhaps unacceptable.
- graemep 1y agoIt is critical for those who choose to use it. If you deliberately decide to use a system that introduces a single point of failure into a decentralised system, you have to live with the consequences. From their point of view, unless they start losing paying users over this, they have no incentive to improve. I assume customers are happy with the SLA, otherwise why use Github?
- yunwal 1y agoNetwork effects are quite strong
- harrison_clarke 1y agoluckily, git itself works pretty well when there's an outage sucks for people that use issues/PRs for coordination and had a planning meeting scheduled, though
- clysm 1y agoWhy is this linking to a merged PR, or a PR at all, and not a status page?
- organsnyder 1y agoIt must be back up!
- mintaka5 1y ago[dead]
- bdcravens 1y agoHN sure has changed. A few years ago there would be at least a dozen comments about installing Gitlab, including one major subthread started by someone from Gitlab.
- factorialboy 1y agoNot just HN, Gitlab has perhaps changed as well.
- lrvick 1y agoWe recommend Codeberg/Forgejo now since it is better in every way, and Gitlab went corpo.
- NewJazz 1y agoGitlab was always for profit. And forgejo doesn't have feature parity at all with gitlab. Neither does github, for that matter. Just take a look at how to push container images from a cicd pipeline in gitlab vs. Forgejo.
- fmbb 1y agoWhat’s the difference? Pushing images is a oneliner.
- NewJazz 1y agoIn gitlab, yes (well, two lines, login then push). In forgejo, there is no cicd token that gives you scoped access to the built in container registry. You must create a long lived token and add it as a secret to the repo you want to push from. See here: https://mteixeira.wordpress.com/2025/02/03/my-self-hosted-forgejo-runner-setup/#creating-a-registry-push-secret https://mteixeira.wordpress.com/2025/02/03/my-self-hosted-fo...
- deleted 1y ago[deleted]
- 1y ago
- john01dav 1y agoI use gitea on a server in my basement because I don't trust these hosted solutions to not use my code for LLM training or who knows what else.
- zaphar 1y agoMe too. I have it mirroring stuff from github too for occasions just like this.
- toastercat 1y agoI've got this feeling that the endless feature creep of Github has begun to cause rot of core essential features. Up until only recently, the PR review tab performed so poorly it was practically useless for large PRs.
- dewey 1y agoGitHub in essence is still pretty much the same, there's products that have feature creep but I wouldn't say GitHub does that. I can't say that I'm having issues with the performance either. I work with large PRs too (Especially if there's vendored dependencies) but I never ran into a show stopping performance issue that would make it "useless".
- davidspiess 1y agoI noticed this recently too when using Firefox.
- inetknght 1y ago> there's products that have feature creep but I wouldn't say GitHub does that. I remember GitHub from years ago. I still find myself looking for things that were there years ago but have since moved. Also, GitHub search is (still) comically useless. I just clone and use grep instead.
- toastercat 1y ago> GitHub in essence is still pretty much the same, there's products that have feature creep but I wouldn't say GitHub does that. I think we're using two different products. Off the top of my head, I can think of Github Projects (the Trello-like feature), Github Marketplace, Github Discussions, the complete revamp of the file-viewer/editor, and all the new AI/LLM-based stuff baked into yet another feature known as Codespaces. > I can't say that I'm having issues with the performance either. I work with large PRs too Good for you. I suffered for maybe 4 years from this, and so have many others: https://github.com/orgs/community/discussions/39341 https://github.com/orgs/community/discussions/39341
- ath3nd 1y agoWait, wasn't GitHub a company ran by the guy who just two days said that devs should either embrace AI or leave the field? https://www.developer-tech.com/news/embrace-ai-or-leave-career-say-developers/ https://www.developer-tech.com/news/embrace-ai-or-leave-care... Maybe his developers embraced AI a bit too much? Or maybe they left the field?
- phendrenad2 1y agoIt used to take a whole team of developers to take down production, now, one programmer with a fleet of agents can do it in 1/10th the time!
- alberth 1y agoEmail-based workflow, does have a few benefits like mitigation from this issue.
- sitzkrieg 1y agohow many more years of this before people realize its actually not good at all?
- drumdance 1y agoAt first I thought this mean that the absolute count of pull requests was trending down and this could be a new BLS data point.
- udev4096 1y agohttps://radicle.xyz https://radicle.xyz is the future!
- dang 1y ago(Presumably?) related ongoing thread: Why is GitHub UI getting slower? - https://news.ycombinator.com/item?id=44799861 https://news.ycombinator.com/item?id=44799861 - Aug 2025 (76 comments)
- hnthrow90348765 1y agoI miss the days where downtime would be like half a day or more and you could use it as an excuse to go home or do something else. Weirdly people were less angry about it back then than we seem to be today.
- vouaobrasil 1y agoThat's because people can't handle speed. With a natural delay, they could cool down or at least become more detached. Society needs natural points where people are forced to detach from what they do. That's one reason why AI and high-speed communications are so dangerous: they accelerate what we do too quickly to remain balanced. (And I am speaking in general here, of course there will be a minority who can handle it.)
- tyre 1y agoIt was like a snow day! So fun.
- dakoosha 1y ago[dead]