8 ms·
Disruption with Some GitHub Services – Resolved
- deleted 2mo ago[deleted]
- CerebralCoding 2mo agoMust be a day ending in Y
- brian626 2mo agohttps://www.dayswithoutgithubincident.com/ https://www.dayswithoutgithubincident.com/
- 98codes 2mo agoCertificate expired over 100 days ago
- xigoi 2mo agoLooks like we need “Days without ‘Days without GitHub incident’ incident”
- isaacdl 2mo agoWhere do you see that? Common Name (CN) www.dayswithoutgithubincident.com Organization (O) <Not Part Of Certificate> Common Name (CN) YR1 Organization (O) Let's Encrypt Issued On Monday, August 10, 2026 at 10:01:51 AM Expires On Sunday, November 8, 2026 at 9:01:50 AM
- SingularCrane 2mo agoalso seeing an expired cert: Common Name R12 Validity Not Before Tue, 10 Feb 2026 17:28:52 GMT Not After Mon, 11 May 2026 17:28:51 GMT
- nosioptar 2mo agoAt this point, maybe it'd be more appropriate for people to post about github to HN when githubs actually working.
- acedTrex 2mo agoOh thank god my pink unicorn site is back online, its had great uptime lately so thats nice.
- ad_fontes 2mo ago> Update - We've identified an issue with a database primary and are failing over to a replica immediately Seems like a weird thing to post on a status page. Shouldn't this have happened automatically and therefore precluded the need to inform users of it?
- thecosmicfrog 2mo ago> Update - primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues
- zackify 2mo agoCan't even run self hosted github actions lol
- time0ut 2mo agoNotice odd behavior on GitHub. Get gaslit by a green status page. Notice more odd behavior on GitHub. Think it must be me this time. See unusual action queuing. Ah, an incident on the status page. Go for a walk and check HN on my phone. The AI SDLC.
- serial_dev 2mo agoThe status page is the last one to get the update. Reddit, HN, X, company chat are all always reporting it sooner.
- nr378 2mo agoGitHub needs to completely bifurcate their enterprise/paid services from their free services at the infra level.
- weli 2mo agoThat's what I don't understand. They could mitigate their name so much if they just split free/paid/enterprise. It's already shown that enterprise is much more estable and is largely unaffected from service disruptions. Why don't they go one more layer? For sure it's worth the extra complexity.
- gaigalas 2mo agoDepending on the cause of the current issues, that move would likely cause more harm to paid services than good. Their last postmortem made clear that their challenges are operational. Scale puts pressure on operation, but it's not what blocks them from keeping up. Doubling the operation doubles the operational challenges.
- lbriner 2mo agoThere is no such thing as "just split" there is 20+ years of legacy decisions and even if the split is relatively clean it is still probably 1 years work for 200 people for maybe a marginal improvement. The real money is going to go towards, "make this all more reliable".
- sdetheridge 2mo agoAccording to their status pages (e.g. https://eu.githubstatus.com/ https://eu.githubstatus.com/, https://us.githubstatus.com/ https://us.githubstatus.com/), their Enterprise Cloud uptime for Actions is significantly higher.
- everfrustrated 2mo agoThat is a different and later product with a confusingly similar name.
- 2mo ago
- frenchie4111 2mo agoI noticed earlier this week that my URL bar now pre-fills githubstatus.com instead of github.com when I type "gith"
- vinnymac 2mo agoFor me this started happening when I type "stat" too. Which surprised me because of the many hundreds of services I rely on regularly that also have status pages.
- b112 2mo agoGood lord, hundreds?! Uptime reliability is nkt additive, it's multiplicative, sometimes even logarithmic. Downtime of 99% and 99% is 98.1%. I hope almost all those services are non-prod related.
- ncr100 2mo agoA browser extension to indicate githubstatus being yellow/red: https://chromewebstore.google.com/detail/is-github-down/lcfoclnaebbfglncdbjiepfbclfhpphf?pli=1 https://chromewebstore.google.com/detail/is-github-down/lcfo... A VSCode extension: https://marketplace.visualstudio.com/items?itemName=RuslanRystsov.github-status https://marketplace.visualstudio.com/items?itemName=RuslanRy... Firefox: https://github.com/matagus/github-status-checker https://github.com/matagus/github-status-checker Caveat: https://news.ycombinator.com/item?id=49450924 https://news.ycombinator.com/item?id=49450924 (disclaimer: none of these are recommended by me to use - merely sharing to build on for this discussion)
- embedding-shape 2mo agoAnother useful disclaimer: all of these lets the developers push updates to your computer as they wish by default in most setups, these days you might want to decrease that kind of attack surface and just check the status in the official website when "git push" suddenly stop working. Especially when extensions and stuff sometimes changes hand and it's kind of hard to keep track of it, except for when something bad happens and gets news attention. I'm also not sure why you'd share links to software you don't even recommend yourself? Isn't it better to just not share that then? I think others could use search engine/LLM too if they need whatever, but most generally expect things shared in the comments here to actually at least have been looked at by the person sharing them.
- xray42 2mo agoSo a normal Wednesday
- everfrustrated 2mo ago> We've identified an issue with a database primary and are failing over to a replica immediately This is why it's hard to take GitHub seriously. How can a single database cause an outage for everyone? This is amateur stuff. Have they no sharding or partitioning internally? Paying customers should not be impacted in the same way as free ones are.
- ferguess_k 2mo agoI wonder what is this database, and why it is hard to fall-over automatically.
- inigyou 2mo agoRDBMS replication and failover is way more difficult and manual than anyone would like. You can't just set up two postgres, tell them they're clustered and have it basically work; at a minimum you have to design the client to somehow know which one is currently the master, or use some sort of proxy (which becomes its own SPOF). RDBMS integrity basically requires that one master server is responsible for the whole data set and other servers may replicate from it. And it usually doesn't wait for a quorum of replicas, just for one, because the design is to recover from a hardware failure, not a network partition, although that could be fixed at the cost of increased latency.
- ferguess_k 2mo agoThanks! I didn't get the chance to manage RDBMs but that's good to know.
- winrid 2mo agoMongo will auto failover if you have 3+ replicas. Primary knowledge is abstracted into the driver or query router level depending on setup.
- inigyou 2mo agoWe all know that /dev/null is web scale. This can be implemented in front of any RDBMS as a separate layer. Traditionally they weren't designed for quorum-sensing since it hadn't been invented yet. I'd be surprised if something like pgbouncer couldn't do it.
- qkwrv 2mo agoWe can't keep living like this.
- pocksuppet 2mo agoObviously we can because we are choosing to. Because servers are scary.
- pajamasam 2mo agoApparently we can because a lot (most?) of us are still using GitHub even after all their outages recently.
- nubinetwork 2mo agoExcept nobody moves to a privately hosted "gitweb"...
- xbryanx 2mo agoI spent a bunch of time during the outage last week setting up forgejo and some custom action runners. At the time, I was worried I was wasting time and getting distracted from my real work...alas, I guess not. Gonna finish up that work and complete the move today.
- NewJazz 2mo agoGood for you. These kinds of migrations never feel productive at the time, but the right tools can make your life and work so much better.
- inigyou 2mo agoI should make a business selling git hosting. Apparently it's really easy because it doesn't have to actually work.
- brazukadev 2mo agoSure but first you gotta be Microsoft.
- pydry 2mo agoMicrosoft and Oracle have the market cornered on selling stuff to corporations who are relaxed about whether it works.
- everfrustrated 2mo ago>Update - primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues And now they're blaming their upstream vendor! Embarrassing stuff to be writing on a public page.
- AdrienPoupa 2mo agoI read that as an upstream service they own, but I agree the wording a bit weird.
- heaney-555 2mo agoWhy is that a problem if it _is_ an upstream vendor problem? (assuming it is)
- ecshafer 2mo agoVitess is a distributed mysql database. Github could very well be managing it entirely on their own. I have only seen people managing their own vitess, its entirely open source afaik.
- fosterfriends 2mo agohttps://cursor.com/origin https://cursor.com/origin
- deleted 2mo ago[deleted]
- tomw1808 2mo agoThings can go wrong, but really, its been a lot and we're normalizing that to an unhealthy degree... I wonder if it was down that much, if users would get credits the way we pay when we use the services - its kind of ridiculous for a critical service to be down that much and all we do is "ah okay, its just github". Like, as if that was normal to be down that much...
- zaik 2mo agoI think the authors of SMTP had a healthy attitude towards server uptimes: Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4-5 days. https://datatracker.ietf.org/doc/html/rfc5321#section-4.5.4.1 https://datatracker.ietf.org/doc/html/rfc5321#section-4.5.4....
- pocksuppet 2mo agoAt the time, most email was either local to the host (big mainframe in the basement) or transferred once a day through scheduled dial-up connections during off-peak phone hours
- creshal 2mo agoWhich is a good fit to a distributed, offline first version control system, if you use it like that.
- donmcronald 2mo agoI saw an application once about 25 years ago that was built on SMTP for communication. I think it was an inter library loan system, but can't remember for sure. I was a pretty smart way of not having to worry much about redundancy in the communications.
- thinkingtoilet 2mo agoI don't think it's been "normalized". Github uptime is literally a joke in the tech community. They have first mover advantage and a behemoth behind them so they're not going away, but everyone knows how shit it's uptime is. It takes time for organizations to move away from services like this but I would bet anything that many are starting to try to move away, as well as new companies knowing they shouldn't use the service.
- igleria 2mo agomust feel bad that at this point every dev checks github status before going to work like it was the weather app.
- iso1631 2mo agoMust be a weekday
- kelvinjps10 2mo agoI have switched off from github to my own server besides two websites that depend on gitbub integration to deploy to clpudfare pages
- shepardrtc 2mo agoFun read about Azure and having 173 agents running a node: https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion https://isolveproblems.substack.com/p/how-microsoft-vaporize... Probably just a coincidence that Github started to have issues after beginning their move to Azure at the end of last year.
- Syntaf 2mo agoOne of my favorite articles posted here for this year, it's a great read and really gives you an idea of just how dysfunctional azure is.
- caust1c 2mo agoAzure's going to suffocate github. I'm curious to see what's next. Will self-hosting the code repository come back in vogue or will another social-coding platform take off?
- jscott817 2mo agoMy small org has definitely had internal discussions around self-hosting gitlab. We'll see what happens.
- vinnymac 2mo agoIf you're actively considering self hosting, I recommend giving forgejo a try. The experience is much more similar to what GitHub offers, actions API is nearly identical to name one similarity of which there are many. Moving essentially becomes one prompt and a coffee later for a small team.
- hosh 2mo agoIf people are happy with Actions API, that sounds great. I’m not happy with the Actions API. I think Gitlab’s cicd design is much better, and I’m not fighting it all the time when I use it.
- Elfener 2mo agoAh so that's why I got a random "github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks"
- stalfosknight 2mo agoSo what’s stopping you (or your org) from leaving GitHub?
- esafak 2mo agoI don't like any open source solution, Codeberg and Gitlab are more of the same and Cursor isn't ready yet.
- lucky_cloud 2mo agoI started the campaign to move us from Bitbucket to Github before the Microsoft acquisition. If I knew MS would be involved, I'd have campaigned for something else. I would never recommend any Microsoft product. It took ages for us to get permission and licenses. We only recently finished moving the last repos over and shut down Bitbucket. If AWS offered a product like GH, I could probably unilaterally start moving to that. Any other alternative would take 3-4 years of meetings, budgets, lawyers, and other such nonsense before we could declare we'd left Github.
- maccard 2mo agoAWS has codecommit.
- sevenseacat 2mo agoWas wondering why all my Actions just stopped running
- broyojo 2mo ago[dead]
- nickwanninger 2mo agoSame time next week?
- theanonymousone 2mo agoThe joke was a good one the first, second, or third time. It's not even funny anymore...
- esafak 2mo agoCursor had better not miss this opportunity.
- mportela 2mo agoGitHub Actions upkeep is down to one 9. I miss the days big tech aimed for four or five 9s of reliability :(
- rvz 2mo agoAnother outage, this time with GitHub Actions. Last time that happened was 5 days ago [0] and another outage happened on the postmortem announcement as well. [1] While GitHub is imploding itself, maybe you should think about self-hosting. [0] https://news.ycombinator.com/item?id=49379172 https://news.ycombinator.com/item?id=49379172 [1] https://news.ycombinator.com/item?id=49379225 https://news.ycombinator.com/item?id=49379225
- guhcampos 2mo agoHonestly? If the problem is scalability, just rate limit git commands on free accounts already. Nobody realistically need to push multiple times per minute, and that alone is bound to trickle down to anything that triggers on commits and pushes.
- brazukadev 2mo agoSure. The problem is just an easy fix that nobody there thought about yet.
- guhcampos 2mo agoMy point is: they blame it on scale. If that's the story they want us to believe, then they have to explain WHY such a simple fix isn't possible. There may be non technical reasons for it, even: legal, or PR, but if they want me to believe the "scalability excuse" they need to convince me they're trying.
- materielle 2mo agoI don’t know, but typically when orgs don’t make logical decisions, it’s because internal politics are incentivizing something else. I suspect that rate-limits would fly in the face of a lot of narratives like “Azure is ready for hyperscale”. And getting eyes on Github is probably a part of their Copilot product strategy.
- pydry 2mo agorealistically the problem is vibe coding.
- roncesvalles 2mo agoThe problem is when you stuff your codebase with so much unreviewed vibecoded slop that engineers lose their grasp on how it all works.
- 2mo ago
- thatwasunusual 2mo agoI have been very happy with github for _years_. Lately, not so much. Now I want to try a self-hosting alternative again. I tried gitlab years ago (pre-pandemic, IIRC), and it wasn't for me (they had _serious_ security issues as well, which of course didn't affect my self-hosting thing, but still...). So - where do we stand in August 2026? *EDIT:* No, I don't want anything related to Felon Musk.
- stephenway 2mo ago[dead]
- lossolo 2mo agoHow can you do so badly with Git when its architecture is basically so friendly to partitioning and horizontal scaling?
- Traubenfuchs 2mo agoThis is unacceptable for paying customers. Either 1) infrastructure for paying and non paying customers must be separated, 2) excessive load must be ended by not offering the free tier anymore or 3) they must fix their broken monolith but they are seemingly incapable of doing so.
- watermelon0 2mo agoIt very well may be that the paying customers are actually the problem. Those can actually afford to throw enough money at AI providers to significantly increase the number of commits & other actions.
- jp_sc 2mo agoMust be Wednesday
- firatsarlar 2mo agoScaling is a dream or theory we think we solved. Distance between ... to practice. So ... should be reconsidered again.
- rglover 2mo agoThis level of scale is extreme but it can be handled with proper time investment and careful engineering. But that doesn't fit into the "spend gobs of money as fast as possible so you can be a hypothetical feudal lord in a sci-fi dystopia novel you read when you were twelve" line of thinking.
- firatsarlar 2mo agoExactly.
- mrshu 2mo agoThe historical uptime has been getting better but it is on a downwards trajectory in August: https://mrshu.github.io/github-statuses/?view=all https://mrshu.github.io/github-statuses/?view=all
- 1matin 2mo agoCan't wait for Anthropic's downtime tomorrow
- JsonDemWitOster 2mo agoUnbelievable! Just this morning I was thinking to myself, hey Github's due for a downtime soon or they are finally getting their act together.
- firemelt 2mo agoat this rate this is a FEATURE!
- rglover 2mo agoGood. Now everyone is getting to see what over-investment in complexity nets you. This bad of reliability screams "tangled spaghetti shit show" behind the scenes. It's a shame that this is happening and hopefully Microsoft do the right thing and spin it back off into a private entity with its own leadership and resources. Also, self-serving: this is why the whole "big company means better" thing is bs. We need to aggressively go back to a culture that rewards and supports independent teams who actually give a damn about what they're building and responsible for. Otherwise, this is just a preview of everything everywhere in a few years.
- SoMomentary 2mo agoAt this point it feels like it would be more honest for this to be titled "Disruption with Some GitHub Service".
- Artoooooor 2mo agoMicrosoft implemented a disruptive technology that caused...well... disruption.