8 ms·
GitHub Incident
- openthc 3y agoSee also this thread: https://news.ycombinator.com/item?id=35967797 https://news.ycombinator.com/item?id=35967797 which just came up about GitHub stability sentiment.
- JoshGlazebrook 3y agogithub.com is 500'ing for me
- rvz 3y agoAgain? Last time this happened was 5 days ago. [0] It is not even the end of the month and the outages are increasing every month and it is now chronically unreliable. Seriously, we have given GitHub more than 3 years to improve and it clearly isn't working. That is plenty of time. At this point, you might as well self-host like the rest of the open source projects out there, since GitHub is falling apart every week and it seems to be more reliable to self-host than to sit on GitHub, go all in and tolerate these outages every calendar month. [0] https://news.ycombinator.com/item?id=35903116 https://news.ycombinator.com/item?id=35903116
- jamal-kumar 3y agoI mean one of the beauty parts of git has always been that you can and, under ideal best practices, should have more than one place to put your code with the distributed nature of it
- willsmith72 3y agoI dunno about "should", that feels like overkill. Expecting to get something like a 99.99% SLA shouldn't be unreasonable to expect out of a hosted solution
- jamal-kumar 3y agoA seperate git remote isn't incredibly unreasonable either - Even if it's just something on localhost in case of scenarios like this one Github being down for a bit shouldn't disrupt your work flow
- beardog 3y agoThe value add for Github is mostly the issue/PR interface and actions.
- Karellen 3y agoBut also, and more relevantly here, git is designed to be usable entirely offline. And if you do need to have some connection with remotes, git allows you to do so extremely sporadically if your network connectivity, or remote availability, is limited.
- anaganisk 3y agoBut currently that's not the beauty part of GitHub, it's the actions, discoverability and other features not part of Git.
- crote 3y agoYeah, they had incidents the 9th, the 10th, and the 11th. So this is the fourth incident this month.
- upbeat_general 3y agoYep, I'm unable to push.
- captaindiego 3y agoWhile new features like search are appreciated, I wish GitHub would focus on stability for a bit, things have gotten pretty so-so
- hn_throwaway_99 3y agoFully agree. I've been super impressed with the number of new features since the MS acquisition, but the number of outages is getting abysmal. At the very least I think GitHub would attempt to shard their infrastructure and repositories - right now it seems like when GitHub goes down it nearly always goes down for everyone simultaneously. This kind of GH outage is actually pretty rare in my experience in that basically every kind of write operation looks down.: basic git operations from the command line, actions, logins, etc.
- zelphirkalt 3y agoI think it is quite individual. I am not impressed by any of the newer features. The UI become more cumbersome to use. I sometimes need to search for things now, while I don't see any new advantages compared to before.
- toastal 3y agoThe new search? You can't even use search on source code without being first authenticated, which requires signing up for an account.
- nrabulinski 3y agoHaven’t been able to go to GitHub.com for over 15 minutes now, just returns a 500. Pretty ironic given their latest blogpost
- kossTKR 3y agoIs this because of Microsoft? If so, why? Are they bad at cloud?
- RobotToaster 3y agoMaybe they moved to windows servers? lol
- nwradio 3y agoLogin is currently broken. Has been for about 10m…
- aj397 3y agoTime to move to gitlab or something else.
- mulmen 3y agoMove to git and email. Like Linus intended.
- hoofhearted 3y agoCan't create new branches :( It seems like GitHub is down at least once a month now, but more like weekly on average.
- sethd 3y ago> Can't create new branches :( Sure you can: git switch -c BRANCH_NAME :)
- sdan 3y ago[flagged]
- alganet 3y agoNothing to worry about. These instabilities are probably caused by Open AI/Copilot improvements gradually becoming Skynet. Seriously now: these instabilities happened a lot on the past weeks. I wonder what's going on there.
- belter 3y agoLike in the movie Wargames it's a trick. They will say the humans can't run a site like GitHub so put it all in the hands of the AI. Then the AI will have full control of GitHub...
- mullingitover 3y agoAt this point it looks obvious that GitHub RIF’d a bus factor earlier this year.
- vxNsr 3y agoRIF?
- cratermoon 3y agoReduction In Force. Layoffs.
- kstrauser 3y agoReduction In Force, aka fired / laid off / made redundant.
- simultsop 3y agoEventually it will be the same everywhere but for sake of economy (stakeholders), staff and customers pay the price, always. By time they find the bottleneck they will be hiring as crazy then.
- smcin 3y agoWhy don't we just use plain language like "laid off" like we used to for decades?
- mullingitover 3y agoThe only thing engineers love more than a TLA is an ETLA.
- tedunangst 3y agoTechnically, a layoff could be temporary, and that was once upon a time the meaning of the term. RIF signals that it's permanent.
- smcin 3y ago
- ivanrenescorcia 3y agoPR service is back :=)
- nabnob 3y agoLooks like it's back up. Still, this seems like an absurd absurd amount of issues recently
- vb-8448 3y agoThey Just published this blog post https://github.blog/2023-05-16-addressing-githubs-recent-availability-issues/ https://github.blog/2023-05-16-addressing-githubs-recent-ava...
- tastysandwich 3y agoWorst timing! Just had an incident and had to deploy a hotfix. Pushes weren't working and I thought I was going crazy. Who would have thought software could be such an adrenaline rush. Move over Alex Honnold!
- exogen 3y agoI was just saying it's kind of ironic that pre-acquisition GitHub inspired confidence, and Microsoft-owned GitHub has been more "move fast and break things." I also don't just mean outages like this... it's clear that Microsoft GitHub has been cramming a bunch of new stuff into the UI and not really treating it with care. I notice very amateur UI bugs, misalignment, bad spacing, overlapping elements, etc. all the time now. It was clear that old GitHub passed through a professional designer's eye quite carefully (or maybe just a developer with extreme attention to detail).
- jahnu 3y agoI dunno. For me the two or three years pre MS were quite stagnant imo. Plenty of outages back then too.
- exogen 3y agoThe "move fast" part was definitely sincere, to Microsoft's credit!
- giancarlostoro 3y agoJust like Skype... I felt like Skype was top tier before Microsoft bought it out. Voice calls were high quality, no issues.
- imran-iq 3y agoSkype was also horribly insecure and leaked your IP which was a cause of a lot of folks in esports getting DDoS'd from it[0]. 0: https://blog.destiny.gg/protection-from-ddos-attacks/ https://blog.destiny.gg/protection-from-ddos-attacks/
- charcircuit 3y agoSkype resolvers definitely still worked post Microsoft acquisition.
- mulmen 3y agoSomeone should invent a decentralized version control system so we can avoid these kinds of widespread events.
- lionkor 3y agogit itself is not enough for teams to work together - you need to asynchronously communicate for the whole thing to come together. Thats what GitHub/Lab/ea bring in value.
- zamnos 3y agoI feel like "communication" isn't descriptive enough. It's right, but eg Slack (or Microsoft Teams, or maybe both) isn't the right shape of communication.
- TimTheTinker 3y agoSomeone should write a local git+ssh service (+ email client) that simulates an upstream by hosting git repos locally, emailing all pushed commits/branches/tags to other team members, and keeping the hosted repos up-to-date via the emailed updates from others. Bonus points for a local HTTP service that provides an optional GitHub-like experience in the browser, with comments/ branch reviews/etc. federated through emailed messages (and represented as commits on meta-repos). With that data model, the primary remaining challenge would be setting up repos on a new machine -- perhaps BitTorrent could help :)
- unboxingelf 3y agogood idea. let’s call it “git”
- whynotmaybe 3y agoNot quite sure, the name should imply that the Source is Safe and provide some Visual tools.
- dchnshA 3y agoBTW: There is work in progress to decentralize modern git based workflows by https://nlnet.nl/project/ForgeFed/ https://nlnet.nl/project/ForgeFed/ (paid for by the European Union), by decentralizing git not at the client level (which it obv. already has, but most people only use one origin), but also on the code forge level.
- labster 3y agoA moment of silence for all of the nines we lost today.
- Popcorn3865 3y agoSpoke to Gitlab rep today about our renewal price (prices going up by 50%+, 25% with "existing customer discount) and the rep used reliability as a key differentiator against GitHub. I quickly pointed about a major outage affecting Gitlabs shared runners which prevented us from deploying a hot fix (we worked around it, but was a ton of stress and extra work).
- rekwah 3y agoDon't leave us hanging. Did you get a discount?! ;)
- frellus 3y agoCan we all stop hating on github already? I assume most people don't even pay for it with their public and private non-work repos. Github has been a major supporter of OpenSource projects and while their uptime could use some help, everyone is struggling right now with RIFs and fewer resources. That's just my $0.02
- yjftsjthsd-h 3y ago> while their uptime could use some help, everyone is struggling right now with RIFs and fewer resources. So? That's entirely self inflicted.
- mulmen 3y agoWe don't owe GitHub anything. They put themselves in this situation. They burned capital to become the defacto source code hosting site then rug-pulled on reliability. They deserve more criticism, not less.
- junon 3y agoStarted as a joke that this was daily. Now it's no much of a joke.