10 ms·
GitHub Git Operations Are Down
- j45 2y agoI guess it's a good time as any to setup a backup or check out alternatives that could mirror or standalone. Gitlab, Gitea, any others worth checking out?
- dboreham 2y agoGitea works for us.
- codetrotter 2y agoI use a fork of Gitea called Forgejo. It works nicely as well. https://forgejo.org/ https://forgejo.org/
- acidburnNSA 2y agoWhat drove you to the fork?
- codetrotter 2y agoFirst I was using Gogs. Someone forked Gogs and made Gitea, because Gogs was under control of a single person and some other people found that frustrating. https://blog.gitea.com/welcome-to-gitea/ https://blog.gitea.com/welcome-to-gitea/ I was using Gitea for a long time, and then someone forked Gitea to create Forgejo. At this time, my installation of Gitea was already out of date a bit because I had previously been manually building and installing Gitea from source. Soon after Forgejo was created, it landed in FreeBSD ports and then it became available in the FreeBSD package manager. So at this point, and having read a bit about Forgejo and seeing that Forgejo was maintained by people with connection to Codeberg, I thought “hey I need to migrate my current Gitea setup anyway. Either to Gitea installed from FreeBSD packages, or to something else. I might as well try Forgejo.” And that’s how I ended up installing Forgejo and I’ve stuck with it since.
- acidburnNSA 2y agoAha, makes sense. Thanks for explaining. I see that Forgejo was created at least partially due to concerns that Gitea was trending towards freemium as well. Good to know.
- kivikakk 2y agoConsider Forgejo: https://forgejo.org/ https://forgejo.org/.
- vv_ 2y agoWhat's the point of having a mirror? It'll only lead to conflicts that'll be difficult to resolve once the main service you use is back up again. For example, I doubt you would be able to easily merge any pull-requests or use the same CI/CD code for the same services without hacky solutions.
- homebrewer 2y agoJFYI, the next version of gitea (which should come out in April) will have full GitHub mirroring functionality (which will let you set up a mirror and have it pull code/issues/wiki/PRs/etc. every few minutes). The current version can either migrate the full repo once, or mirror the code and nothing else.
- hunkins 2y agoGitHub git operations are down. Any git operation returns a 500 error response.
- localghost3000 2y agoCan't pull anything from my repos which is not great.
- hunkins 2y agoYeah, this is not good. I've experienced other outages on their other products (e.g. GitHub Actions), but not even being able to do a basic git pull is an entirely different level.
- localghost3000 2y agoI'm effectively dead in the water here. I guess I'll go touch some grass. Thank you to the SRE's who are looking at this.
- tonetegeatinst 2y agoLooking forward to a blogpost or writeup from github on this.
- normie3000 2y agoIs it possible to download a zip of a repo through the GUI?
- __d 2y agoSSH access appears to be rejecting my key too.
- __turbobrew__ 2y agoYep, 100% of git operations are failing. CI/CD and dev operations are at a halt.
- jxi 2y agoGitHub Actions are also failing with 500 (gateway timeout) errors when trying to fetch the repository.
- lukeh 2y agoYup, SSH rejecting my key, thought it was something at my end but no. Can't run any Yocto builds... I guess a good opportunity to take a walk or do some life admin.
- lukeh 2y agoBack up now for me.
- remram 2y agoI hope you didn't wait until now to push your day's work!
- l33t_d0nut 2y agoI did :(
- unwoven 2y agolmao i deleted and readded a new key thinking my key was fucked
- anonyrat 2y ago[dead]
- vinayan3 2y agoExperiencing this as well. I was a bit concerned something was wrong with my key but alas it is not my key but an outage. Developer "snow day".
- TheRizzler 2y agoGIT is down today, Code rests in snowy silence, Developers play.
- itissid 2y agoSpent like 30 mins tryig to figure out why ssh auth was not working. Compared sha256 signatures, doubted my reading abilities and pulled my hair.
- sc68cal 2y agosame here I was scratching my head trying to figure out if something had corrupted my system or what
- parpfish 2y agodamn, why couldnt it happen during east coast business hours?
- crest 2y agoIt's almost as if only Git is distributed, but people sold out to Github for convenience. Too bad Git lacks a distributed bug tracker and wiki system like Fossil. Guess Github has to fail a lot more for things to change.
- bastardoperator 2y agoIt's this type of negativity ruining the internet. Nothing thoughtful to say, nothing to add, and hoping for failure. I hope everything is okay over there...
- liontwist 2y agoThey are advocating for decentralized tools. That’s actionable and hoping to prevent failure.
- vv_ 2y agoThere is no reason to host 'decentralized' tools besides regulation. It's considerably cheaper to use GitHub (or other alternatives like GitLab) than hosting your own and hiring people to maintain and support the solution. Their issue tracking system is very convenient for small teams too.
- bastardoperator 2y agoExactly, hire a team of 3 and pay 500K in compensation, or spend 100K on a system that works and you get a support person to call in the event of an issue. The math is so simple.
- liontwist 2y agoExcept you’re not considering the cost of when you can’t deliver something on time for a customer because infrastructure you don’t control is down. You don’t outsource things that prevent you from doing your core competency.
- JSTrading 2y agoPanic for a minute SSH rejected a key!
- atulvi 2y agoI just reset my ssh key lol.
- JSTrading 2y agoThey are going to get DDoS's by people trying to do this haha.
- itsjustjordan 2y agoI was going insane doubting my SSH knowledge, stopped short of creating new keys thankfully!
- donohoe 2y agoSame - but created new keys :) ... which also didn't work, and then I went to check... oh well.
- atonse 2y agoI thought my account was hacked and ssh keys removed. Panicked a little bit and then double checked that my keys were still there. Then went to github status and calmed down.
- eru 2y agoI briefly entertained the notion that I was fired and my ssh keys revoked. But I still had access to slack and email etc, so I banished that thought.
- Layvier 2y agowell on my side I'm the proud owner of a new ed25519 key.. The status page didn't update quick enough
- smus 2y agoas am I!
- G1N 2y agoWas trying to install homebrew on a device and kept wtfing because the clone step kept failing… thought I was crazy
- NautilusWave 2y agoI had just upgraded to Windows 11 last week, and for some godforsaken reason earlier today, I could SSH via WSL but not from the host OS even though they were both using keys served from the Windows OpenSSH agent! I'm just going to blame this service outage and hope for the best tomorrow.
- rasengan 2y agoJust the info I was expecting to see on HN. Unfortunately (or Fortunately?), a better source for these kinds of things than anywhere else.
- cdme 2y agoIt's just a hub at the moment.
- hunkins 2y agoUpdate: looks like the problem has been identified. "We've identified a cause of degraded git operations, which may affect other GitHub services that rely upon git. We're working to remediate."
- arnath 2y agoIt's odd to me that there's nothing indicating an issue is ongoing on the github.com homepage
- clintonb 2y agoUnclear if related, but the Terraform Registry is also having issues serving data: https://status.hashicorp.com/incidents/jypswvyh0h3z https://status.hashicorp.com/incidents/jypswvyh0h3z. In a wonderful twist, we are relying on a couple modules served from GitHub!
- impure 2y agoI thought it was my internet. Oh well, time to finish that game of Civ 6. Edit: and it literally came back up the moment I posted this.
- tom1337 2y agoNot sure if I am just noticing GitHub's issues more often as I am using their tools pretty much every day but their availability is kinda not great. Be it Actions failing or something "not core business" (read: git operations) but I can't remember a month in the past where I was not annoyed by any outage on GitHub
- croemer 2y agoAgreed, I wonder what their downtime percentage is. If guess it's down on the order of one hour per month, so 1/1000. Update: They promise >99.9% on a quarterly basis for enterprise customers - https://github.com/github/docs/blob/main/content%2Fsite-policy%2Fsite-policy-deprecated%2Fgithub-enterprise-service-level-agreement.md https://github.com/github/docs/blob/main/content%2Fsite-poli...
- ratmeadow 2y ago99.9% uptime corresponds to about 2 hours downtime per quarter, if my maths is correct. If that is indeed the guarantee, based on the experience at my company, GitHub has failed its promise recently (or is getting damn close). I recall 2 decent outages in the past few weeks alone. It's making me begin to doubt if GitHub's reliability is appropriate for an enterprise service.
- croemer 2y agoI'm not sure whether the uptime guarantee is 99.9% independently or jointly for each service, i.e. if service A is down for 0.06% and service B another 0.06% but not at the same time, will this count as overall uptime >99.9% or <99.9%. Rereading the SLA, it looks like Github can have each service feature like issue, pull requests, git operations be down 0.1% and still not reimburse. In your head you might not account separately for each feature, but Github does.
- FridgeSeal 2y agoThe whole product feels like it’s getting g progressively jankier. The front-end is glacial nowadays and frequently has issues actually loading the page, actions frequently has some kind of panic attack and breaks or just grinds along a glacial speeds. The UX has gotten worse (why no merge-queue button?).
- 486sx33 2y agoDamn you azure! Or is Microsoft not even running their precious GitHub in azure
- bobnamob 2y agoWho said it was an infrastructure failure? There's innumerable causes of this kind of failure that aren't rooted cloud service provider shenanigans (Admittedly the duration of the outage does "feel" like an infra outage)
- qqqult 2y agolol I changed my ssh key 2 times until I figured out pubkey permission denied was an issue on their side
- james33 2y agoAll checkmarks just turned green and my push went through. Snow day over.
- Rexogamer 2y agothe incident has been marked as resolved; I am now able to push
- freewizard 2y agoIt was down for ~2 hours. The status website claims "degraded performance", but in reality we get git@github.com: Permission denied (publickey). Either GitHub didn't know how to communicate, or they were not sure about the real impact. This is bad.
- manfre 2y agoStatus pages are rarely honest. The company will lie to salvage their SLA. "Degraded performance" or "some customers are experiencing an elevated error rate" should be interpreted as "service unavailable / outage"
- croemer 2y agoI'd be curious by how much they downplay downtime. Wouldn't be too hard to put together an honest status page that pulls & pushes something new to main every 5 minutes, creates an issue, comments on it etc. Very basic high level checks.
- croemer 2y agoSomeone else lamenting the delayed status page updates almost 2 years ago: https://news.ycombinator.com/item?id=35887213 https://news.ycombinator.com/item?id=35887213
- enasterosophes 2y agoI'm not convinced it's deliberate dishonesty. Just a communication disconnect. Firstly, it can take time from the first yellow flags to the full realization that there really is an incident underway, secondly it needs someone to decide how to communicate that incident, and thirdly the engineers who are actually working on the incident need to be able to get on with it instead of being pestered for an update every 10 minutes.
- eru 2y agoThey were also down a few days ago, but HN flagged my submission as a dupe. :)
- burgerrito 2y agoThankfully I use Forgejo! No reason to stick with GitHub when Git itself is distributed by nature
- simondotau 2y agoTrue. Also there's no reason not to stick with GitHub when Git itself is distributed by nature. That's the beauty of distribution.
- ashishb 2y agoWhy does GitHub go down almost once a month?
- 1propionyl 2y agoAcquisition. ducks
- 0xbadcafebee 2y ago> all Git operations were unavailable due to a configuration change eats popcorn waiting for explanation of why they didn't catch it in non-prod
- pnw_throwaway 2y agoHave you never fixed a prod-specific bug before? They’re common enough, and it sounds like they patched this one within a reasonable amount of time.
- 0xbadcafebee 2y agoI expect more from billion-dollar companies. If you have that much money, you can pay somebody to ensure the changes are properly tested. (I've worked in billion dollar companies, and usually they are too cheap to do this, despite their massive wealth. They just leave it up to ICs and hope they are testing right, rather than making sure)
- noplacelikehome 2y agoI've just ended my Pro plan; enough is enough. I've been debating moving my repositories to a self-hosted forge for a while; it'll be an interesting opportunity to move more CI operations out of Actions and into Nix.
- jcims 2y agoTangential at best but I work in a regulated industry and we're starting to get some heavy scrutiny from external auditors and regulators on the way we plan to address extended outages and forced exits from third party vendors. This goes beyond basic DR/BCP and plans are reviewed with at least a high level scenario exercise. https://www.federalregister.gov/documents/2023/06/09/2023-12340/interagency-guidance-on-third-party-relationships-risk-management https://www.federalregister.gov/documents/2023/06/09/2023-12... On the surface a product like managed git repos would seem to be relatively straightforward to deal with, but these same regulated firms are also under tremendous scrutiny for access management, change management, SDLC, etc etc. They also have a huge software footprint which just increases the impact. Self-hosting is obviously the traditional answer, but that's not exactly a simple one either. Just an interesting problem.
- dangero 2y agoThat's a lot of overhead. Do you know how they are calculating the possible risk of these events? It feels like there are a million rabbit holes like this you could go down when modern infrastructure is so cloud connected. Is the risk of your git repos higher than a chip shortage causing you to lose access to the infrastructure you need? So many factors to consider. A chip shortage doesn't seem that unlikely with geopolitics. The list of scenarios you mitigate for seem like they could very easily be an arbitrary list of the of scenarios a single person came up with.
- jcims 2y agoIt's an astonishing amount of overhead that slows everything to a veritable crawl. It also creates downstream issues because you need to build in contingencies for your service consumers until you are able to fully abstract the dependency to a point where rehoming doesn't impact them. The risk calculations are very primitive at the moment, I'm guessing they will be refined over time as industry feedback starts to resonate.
- cheeze 2y agoAnd the ultimate problem that you don't need a solution for, until you _need_ it. Which often leads to "yeah we have backups" but without regular testing, can you trust them?
- the_svd_doctor 2y agoHa so that's why my git pull were so slow. I should have checked HN faster.
- parpfish 2y agoit'd be great if they set up their CLI tools such that if there's an error/timeout, they'll do a check of their status page so you don't get the standard 'error' but rather a 'dont worry, youre not doing anything wrong. its borked, our bad' message.
- xyst 2y agoYet another reason to self host your VCS. Only thing hosted to “the cloud” these days are my backups - split between S3 and GCS. ~$25/mo
- orf 2y agoThat’s expensive, I get it for free with GitHub. And a loooot more functionality.
- souenzzo 2y agowindows window operations are down
- Johndavid20224 2y ago[dead]