7 ms·
GitHub is down
- cranberryturkey 3y agohomepage gives me a 500
- teruakohatu 3y agoWorking for me in NZ, logged in.
- cranberryturkey 3y agoits back
- jamil7 3y agoUp in Europe still.
- schleck8 3y agoNot for me, in Europe.
- _astromonkey_ 3y agoNot working for me in Europe
- mandalorianer 3y agoYea, in Germany it works.
- pgeorgi 3y agoThe failure has yet to propagate to Germany, given how slow internet is here.
- weberer 3y agoWhich country? There are multiple data centers across the continent.
- deleted 3y ago[deleted]
- _astromonkey_ 3y agoYeah, experiencing issues with Pull Requests and commits
- schleck8 3y ago"Degraded performance"
- zuck_vs_musk 3y agoThis happens fairly regularly. Do HNers think that Microsoft azure and/or Ruby has a role in it?
- furkansahin 3y agoWhy would Ruby play a role in it? Azure definitely does though. (ex azure employee here )
- usrme 3y agoGitHub itself is written in Ruby.
- chupasaurus 3y ago[0] is a my favorite demonstration of it. [0]: https://github.com/rails/rails/commit/b83965785db1eec019edf1fc272b1aa393e6dc57 https://github.com/rails/rails/commit/b83965785db1eec019edf1...
- enescakir 3y agoDon't underestimate Ruby's performance https://x.com/tobi/status/1728524453854756883?s=20 https://x.com/tobi/status/1728524453854756883?s=20
- furkansahin 3y agoWell, when we say GitHub, we are talking about thousands of services. I don’t think we can pinpoint the overall operational failures to programming language choice. In my experience, these kind of issues are mostly caused by poor organizational architecture + poor project management. In GitHub’s case, I’m also confident to say the choice of infrastructure plays a much bigger role than Ruby.
- zuck_vs_musk 3y ago> Why would Ruby play a role in it? Azure definitely does though. (ex azure employee here ) Oh yeah. If someone says C++ has poor memory safety than Rust then no one is triggered. But say something about Ruby (the language on a secular decline - https://www.dice.com/binaries/content/gallery/dice/insights/2023/github-languages.jpg https://www.dice.com/binaries/content/gallery/dice/insights/...) and the pitchforks are out.
- erazor42 3y agoClassic Github for the past few years we are used to it / resilient now.
- smokel 3y ago[flagged]
- logicchains 3y agoPresumably they intended to entertain. There are absolutely companies running much more complex services with more users than Github but better uptime, so it's absolutely fair to call Github out on their failures. Github has at least an order of magnitude fewer users than something like Google or Facebook. If your attitude is reflective of the attitude of anyone at Github, calling people entitled who dare complain about the site's downtime rather than taking responsibility and working to better it, it's no surprise that Github has so much downtime.
- cqqxo4zV46cp 3y agoYou’re being really mean. Do you work for GitHub? Or do you just have a habit of taking things too personally? “i bet you can’t do better!” is not how an adult responds to someone expressing their displeasure at the fact that the performance of a service that they / their org is almost certainly paying for is below their needs / expectations. Damn right I couldn’t do better. I’m also not the massive highly paid and talented workforce of GitHub. Next time you whinge about your mechanic not fixing your car or your plumber not fixing your toilet, I’ll be right there in your ear calling you a spoilt brat for not just doing it your damn self.
- miningape 3y ago100% we rely on others for goods and services, regardless of our ability to perform said service we are able to (and should be able to) judge it. Otherwise everyone would need to be an expert in everything. In other words, I don't need to be a banana farmer to say a banana is rotten.
- 3y ago
- pythux 3y agoThe status page says it's been resolved 8 minutes ago (Apr 05, 2024 - 08:48 UTC): https://www.githubstatus.com/incidents/bnkkbj90yhz6 https://www.githubstatus.com/incidents/bnkkbj90yhz6 But it definitely still happens now (500s on refresh on PRs and GitHub actions) Edit: still ongoing Edit 2: still ongoing at Apr 05, 2024 - 08:56 UTC (keeping updated for the record since their status page cannot be trusted apparently) Edit 3: I see they have switched to a different (ongoing) incident ID now: https://www.githubstatus.com/incidents/5ly0psff2s5d https://www.githubstatus.com/incidents/5ly0psff2s5d
- enescakir 3y agoThey marked this one as resolved. They keep providing updates on another active incident: https://www.githubstatus.com/incidents/5ly0psff2s5d https://www.githubstatus.com/incidents/5ly0psff2s5d
- manfre 3y agoAlready listed as being resolved. A 15 minute outage at 4:30 am EST.
- Macha 3y agoOr 9:30am GMT, so just as europeans are starting to work. Or middle of the working day in India.
- enescakir 3y agoApr 05, 2024 - 08:51 UTC: Update > We're seeing connection failures to some databases in two of three sites and are investigating.
- drcongo 3y agoThis should probably link to the main status page as it seems there's more than one incident ongoing.
- enescakir 3y agoIn the past two months, they've had over 10 incidents. Are they so hyped by AI features that they're neglecting their core services?
- cqqxo4zV46cp 3y agoPlease stop doing catsstrophising tech journalist cosplay.
- ta1243 3y agoNew features are sexy. Core stability is boring. AI is just the latest new feature.
- jq-r 3y agoTrue, and SRE teams get smaller and smaller and/or tasked to do other non-related things. So when users start leaving for some competitor, the same SRE team suddenly gets more heat, pressure, more belt tightening, more meetings "for transparency and coordination", and directly and indirectly more blame. Ask me how I know... I bet its a story as old as time though.
- pm90 3y agoThere is little incentive for them to invest in stability. Customers will eat the pain because they are so invested in the platform. They will never move away. Meanwhile all the Staff engineers trying to get promotions are building AI shit.
- pwmtr 3y agoSince last few months, GitHub has become very unreliable. For example, last month, they reported six official incidents in just one week. I've also noticed many shorter, non-official incidents (i.e. the ones where there is no active incident on their status page, but many API calls return a 500 error). I learned about this the hard way (we offer affordable GitHub Runners at Ubicloud, which gives us the opportunity to observe all these failures firsthand)
- masklinn 3y agoI’d say years though it might depend on both your physical location and te services you use. We’ve had to harden a bunch of automation to be more reliable in case of github going tits up, and while github is still currently the broker (as it’s where users get statuses / information from it made sense at the time to also have it handle broadcasts) we’re definitely planning on direct communication between services eventually.
- bushbaba 3y agoCurious if this is related to them adopting azure
- Am4TIfIsER0ppos 3y agoThank god (Linus) that git is a distributed "protocol" letting you pull and push to any location on your file system or anywhere you can access over ssh.
- ta1243 3y ago99% of git users have no concept of this. It's very much a client/server relationship with github.
- masklinn 3y agoGit itself is only a piece of the puzzle, you still need synchronisation points for collaboration e.g. bug reports, code reviews, discussions, CI, synchronisation, etc… Those are the services forges provide. Also git’s support for point to point distribution is bad. I miss `hg serve`.
- mr_mitm 3y agoThis comes up pretty much everytime Github is down. Github is not a replacement for git. Github is an issue tracker and CI platform just as much as it is a git hosting site, none of which git can provide by itself. It's complementary. The comment really doesn't make any sense. It's like being snarky about the distributiveness of git when the Linux kernel mailing list goes down.
- ta1243 3y agoMy on-prem gitlab has now survived 6 cloud-based github outages.
- Already__Taken 3y agohow long do you spend upgrading it?
- dijit 3y agoNot OP, but for mine I spend about 10 minutes every 2 weeks. Once: an upgrade went wrong and I had to roll-back, that made me spend 30 minutes instead of 10. Gitlab instance stats: Projects 256 Users 216 Groups 17 Disk usage: 877 GiB / 3.38 TiB
- _hao 3y agoIn London it's dead. I guess I'm taking the day off! Happy Friday everyone!
- Whiteshadow12 3y agoSorry everyone. This was me.
- notoverthere 3y agoWhat did you do?
- jjgreen 3y agoBounder
- blippitybleep 3y agoI saw a post on LinkedIn, from GitHub, boasting about how many changes they deploy in a day. I couldn’t believe it. GitHub actions have been broken so many times for me this past month, not able to deploy my code when I want to, so it felt like a slap in the face.
- nonrandomstring 3y agoMaybe what is happening in this thread should give everyone cause for concern and a clear heads-up that more serious problems are afoot. Very emotional reactions here, whether fuming at the unreliability of the service, or zealously defending and identifying with it, indicate people have far too much invested in this single point of service. People are saying - "That's me done for the day, have a good weekend!" What is the economic cost of this? On a global scale? The pattern is the same as we've seen recently with Office365 and other cloud services. People talk about setting up elaborate monitoring services, yet with a similar effort one could set up one's own development repositories and tools. It's not like this crowd is typically "helpless", or unaware of what is being done, yet people remain beholden; Github has diversified into a social network and AI tool, far beyond its initial remit as a code repository. People seem genuinely terrified at the prospect of it going. And rightfully so. In the wake of the xz backdoor attempt Microsoft will soon move to offer "developer protection" and a system of validation, testament and supply-chain protections. At that point Microsoft will have all of you absolutely in the palm of its hand and at its mercy. Now would be a good time to seriously invest in long term thinking and strategic protection of your development future, indeed of your fundamental ability and rights to develop software free from disruption, pressure and encumbrance.