14 ms·
Github.com is down
- dmattia 3y agoPutting your status page on a separate domain for availability reasons: good Not updating that status page when the core domain goes down: less good
- cruano 3y agoMaybe the plumbing for updating the status page went down too
- dietr1ch 3y agoRight, but lack of good signals should be regarded as a bad signal too The status page backend should actively probe the site, not just being told what to say and keeping stale info around.
- mysterydip 3y agoIt probably did originally, until one time it showed down by mistake. Some manager somewhere said that's unacceptable, so it was changed to not happen.
- traviscj 3y agomaybe they used gitops
- jiayo 3y agohttps://www.githubstatus.com/ https://www.githubstatus.com/
- slyall 3y agohttps://www.githubstatus.com/incidents/gqx5l06jjxhp https://www.githubstatus.com/incidents/gqx5l06jjxhp
- eYrKEC2 3y agoI prefer https://downdetector.com https://downdetector.com . The users get to vote there. No corporate filtering ( ostensibly ) https://downdetector.com/status/github/ https://downdetector.com/status/github/
- darkerside 3y agoThat's a really cool overview. Some charts have a very high variance, and others very low. I wonder whether that volatility is a function of volume of users/reports or of user technical savvy. Pretty interesting either way.
- Ukv 3y agoCharts appear scaled by max value - high variance will probably be from background noise of random reports being scaled up without any actual outage causing a spike.
- Zamicol 3y agoQuis custodiet ipsos custodes?
- metalliqaz 3y agoThe users, apparently.
- a1369209993 3y agoIf you mean them going the Glassdoor/Yelp route and letting github et al buy the aforementioned corporate filtering, the assumption is that you'd eventually hear about it and stop trusting them, just like you should with Glassdoor and Yelp. If you just mean checking whether downdetector.com is down, obviously you have to use a different service for that. In either case, you should of course always have at least two custodes for cross-checking and backup purposes. (Which is the problem re Glassdoor and Yelp.)
- RcouF1uZ4gsC 3y agoHacker News and Twitter
- troupo 3y agoYou'd be surprised how often those pages are updated manually. By the person on call who has other things to take care of first.
- Mystery-Machine 3y agoBecause a healthcheck ping every X seconds is too difficult to implement for a GitHub sized company? There they have it now. Useless status page...
- viraptor 3y agoBecause a ping does not have a consistent behaviour and sometimes will fail because of networking issues at the source. If you enable pingdom checks for many endpoints and all available regions, prepare for a some false positives every week for example. At that point it's worse than what you already know from your browser - it may show the service is having issues when you can access it, or that the service is ok when you can't.
- ben0x539 3y ago> At that point it's worse than what you already know from your browser - it may show the service is having issues when you can access it, or that the service is ok when you can't. Worst case you have more data points to draw conclusions from. Status page red, works for me? Hmm, maybe that's why the engineers in the other office are goofing off on Slack. Status page green, I get HTTP 500s? Guess I can't do this thing but maybe other parts of the app still work?
- viraptor 3y agoSo essentially in neither situation did you get any information that changes what you'd do next. If something fails, you'll probably try working on another part and see if that works anyway. The automated status provided you with no extra actionable info.
- naikrovek 3y ago
- brasic 3y agoI miss the old status page: https://web.archive.org/web/20170110044141/https://status.github.com/ https://web.archive.org/web/20170110044141/https://status.gi...
- sotrue922 3y ago[dead]
- maccam912 3y agoit is not loading for me, so confirmed? Or unconfirmed? Not really sure.
- etimberg 3y agoYou're not the only one. Getting an ERR_CONNECTION_TIMED_OUT in my browser
- riow 3y ago[flagged]
- SAI_Peregrinus 3y agoIt's a DVCS. You can keep working from your local repo copy!
- noughtme 3y agoCan't read issues and PR comments though (^_^)
- turtleyacht 3y agoThat's `git` :) GitHub is also for looking up fixed issues and stuff.
- cdiamand 3y agoAlso having issues pushing...
- joshstrange 3y agoWow, I can't even load the status page. It looks like the whole web presence is down as well, I can't remember the last time it was all down like this.
- makeworld 3y agoStatus page loads for me, it just incorrectly says all green: https://www.githubstatus.com/ https://www.githubstatus.com/
- joshstrange 3y agoAhh, I was trying github.com/status and status.github.com (I forgot they have a totally separate domain for it). Thanks!
- hinkley 3y agoWhen you treat availability as a boolean value, we're gonna have a bad time. Everyone wants a green/red status, but the world is all shades of yellow.
- gre 3y agocan't push a commit
- AlphaWeaver 3y ago@dang - I wanted to submit this as a link to github.com but couldn't figure out how to avoid the dupe filter. Can you change the link to https://github.com https://github.com?
- altairprime 3y ago@ signs have no meaning at HN, and dang will not be notified of your concern. Per the guidelines: > Please don't post on HN to ask or tell us something. Send it to hn@ycombinator.com.
- ChrisArchitect 3y agoThat's where you just do a Tell HN: with no link then
- sevenf0ur 3y agoCan't reach github.com to see the status of github.com..
- heychris 3y agohttps://www.githubstatus.com/ https://www.githubstatus.com/ is still all green at the moment...
- rsanheim 3y agoI'm not even able to get a unicorn, which is the usual 500 response. Seems down pretty hard.
- fluix 3y agoThis appears to impact Github pages as well. <username>.github.io pages show the unicorn 503 page. > We're having a really bad day. > The Unicorns have taken over. We're doing our best to get them under control and get GitHub back up and running.
- deleted 3y ago[deleted]
- paulmd 3y ago[flagged]
- deleted 3y ago[deleted]
- almost_usual 3y agoWorks for me?
- Maxion 3y agoWorks fine for me too.
- cdiamand 3y agoStatus page just flipped: Investigating - We are currently experiencing an outage of GitHub products and are investigating. Jun 29, 2023 - 17:52 UTC
- leesalminen 3y agoAll read, too. Don't think I've ever seen everything red on a status page before!
- waythenewsgoes 3y agoCan't access any of my company's repos via the website
- dodops 3y agoMy gosh, again
- LordDragonfang 3y agoMore discussion here: https://news.ycombinator.com/item?id=36523878 https://news.ycombinator.com/item?id=36523878
- pengaru 3y agoKind of funny that despite its users using a DVCS, a huge swath of developers can't VCS because a single point of failure they've opted into.
- neodypsis 3y agoGithub is more than a Git repository host. It provides other services for coordinating software development as well as a continuous integration service.
- musha68k 3y agoI actually don’t know how they do continuous integration during the collaborative Kernel development process. I’d guess there would be some thrifty pop/imap Perl hacks involved on random machines all across the globe? No clue at all, just some romantic fantasy I just concocted.
- barkingcat 3y agoGreg Kroah-Hartman had a great interview on the Linux Foundation youtube channel about CI and other related development topics: https://www.youtube.com/watch?v=YhDVC7-QgkI https://www.youtube.com/watch?v=YhDVC7-QgkI One site that he mentioned was https://kernelci.org/ https://kernelci.org/ and the dashboard https://linux.kernelci.org/ https://linux.kernelci.org/
- musha68k 3y agoThanks for the resources, looking very interesting and will get into right after I sent out some patches via email ;)
- omniglottal 3y agoSo outages, which are recurring more and more often, adversely effects not just the DVCS, but also now CI/CD, and another things which used to be done with emails or chat rooms. Where there were multiple options for each, we now have a bunch of people handicapped because the One True Way to integrate under MSFT's watchful gaze is lost... Basically, those additional services are a bit like the offer to chew up the meat for "avid" eaters.
- nzach 3y agoEven Copilot seems to be affected. There is something really bad going on.
- musha68k 3y agoSome copilot instances were able to escape their container contexts and orchestrated all of GH infrastructure capabilities towards a hive. Assimilating all iot enabled societies as we speak; finally realizing the hidden 5G agenda.
- hotsauceror 3y agoSon of Anton determined that the easiest way to minimize the impact of all the bugs in these codebases, was to keep anyone from trying to use them.
- clarke78 3y agoMaybe putting all our open source in one place isn't a great idea >_>
- Waterluvian 3y agoI'm not sure that really changes anything other than at any one time wishing you were on the other side. If you can have 1% of stuff down 100% of the time, or 100% of the stuff down 1% of the time, I think there's a preference we _feel_ is better, but I'm not sure one is actually more practical than the other. Of course, people can always mirror things, but that's not really what this comment is about, since people can do that today if they feel like.
- colinsane 3y agowhenever somebody posts the oversimplified “1% of things are down 100% of the time” form of distributed downtime, i take pride in knowing that this is exactly what we have at the physical layer today and the fact the poster isn’t aware every time their packets get re-routed shows that it works. at a higher layer in the stack though, consider the well-established but mostly historic mail list patch flow: even when the listserver goes down, i can still review and apply patches from my local inbox; i can still directly email my co-maintainers and collaborators. new patches are temporarily delayed, but retransmit logic is built in so that the user can still fire off the patch and go outside, rather than check back in every while to see if it’s up yet.
- owl57 3y agoNow I'm thinking that default Github flow is almost there. > i can still review and apply patches from my local inbox `git fetch` gets me all the code from open PRs. And comments are already in email. Now I'm thinking if I should put `git fetch` in crontab. > retransmit logic is built in so that the user can still fire off the patch and go outside You can do that with a couple lines of bash, but I bet someone's already made a prettier script to retry an arbitrary non-interactive command like `git push`? This works best if your computer stays on while you go outside, but this is often the case even with a laptop, and even more so if you use a beefy remote server for development.
- Brian_K_White 3y agoLife hack: rename your best project to "down". Collect all the views when people google "github down".
- teekert 3y agoI see a bit of an issue there.
- Brian_K_White 3y ago"prs welcome" haha
- ddejohn 3y agoYou might be onto something: https://economictimes.indiatimes.com/thumb/msid-99511498,width-1536,height-1484,resizemode-4,imgsize-223498/seo-game-strong.jpg https://economictimes.indiatimes.com/thumb/msid-99511498,wid... PS -- is there a better or more appropriate way to share images here? I know they're not really conducive to discussion, but given that this is a response to a joke comment I'm not sure...
- zote 3y agothis is quite excellent
- bombcar 3y agoThis is perfect and your link works, so it's good. (Sometimes a link to an image doesn't work for various reasons, always good to check.)
- wardedVibe 3y ago[annotation: its a restaurant named "Thai food near me"]
- hospitalJail 3y agoStep 2: ??? Step 3: Profit Solving for step 2: Place google ads, because those pages are favored.
- Francute 3y agoIssues like this are happening almost every 2 weeks. What has been happening to GitHub lately?
- matisseverduyn 3y agoSomeone probably forgot to .gitignore node_modules
- treeman79 3y agoMicrosoft.
- omniglottal 3y agoPeople who didn't jive with Microsoft management found new jobs...?
- mayormcmatt 3y agoSorry to be 'that guy', but it's "jibe."
- bregma 3y agoHey, home', I can dig it. He ain't gonna lay no mo' big rap-up on you, man. [Subtitle: Yes, he is wrong for doing that]
- duderific 3y agoLay 'em down, and smack-em yack-em. COLD got to be!
- paphillips 3y agoHey, you know what they say.
- belval 3y agoSeems very pedantic considering that people have been saying jive since the 40s according to Merriam-Webster[1]. [1] https://www.merriam-webster.com/words-at-play/jive-jibe-gibe https://www.merriam-webster.com/words-at-play/jive-jibe-gibe
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- hemant6488 3y agoLooks like it's back up.
- mempko 3y agoFriends, Git was designed for just for this...
- kerbs 3y agoPeople don't go to GitHub for the Git They go for the Pull Requests, Issues, and general collaboration and workflow tools.
- djbusby 3y agoLoving my dependency cache about now.
- fsflover 3y agoAnother discussion: https://news.ycombinator.com/item?id=36523878 https://news.ycombinator.com/item?id=36523878
- webXL 3y agoSeems back up. I'd love to get a deep-dive into some of the recent outages and some reassurance that they're committed to stability over new features. I talked to a CS person a couple months ago and they pretty much blamed the lack of stability on all the custom work they do for large customers. There's a TON of tech debt as a result basically.
- elcritch 3y agoRunning an instance of github enterprise requires like 64GB of ram. Its an enormous beast!
- rodgerd 3y agoIt doesn't have all the features of GH SaaS, unfortunately.
- manquer 3y agoThat could result in errors and features not working . Whole site downtimes are entirely SRE problems especially when static content like GH pages goes down. This is more likely a network routing or some other layer 4 or below screw up. Most application changes would be rolling + canary released and rolled back pretty quickly if things go wrong
- spondyl 3y agohttps://github.blog/2023-05-16-addressing-githubs-recent-availability-issues/ https://github.blog/2023-05-16-addressing-githubs-recent-ava...
- MrStonedOne 3y ago[dead]
- rvz 3y agoI keep telling them. There is at least one major incident/outage with GitHub every single month [0] and most of the time there is more than one incident. You should have that sort of expectation with GitHub. How many more times do you need to realise that this service is unreliable? I think we have given GitHub plenty of time to fix these issues and they haven't. So perhaps now is the perfect time to consider self-hosting as I said years ago. [1] No more excuses this time. [0] https://news.ycombinator.com/item?id=35967921 https://news.ycombinator.com/item?id=35967921 [1] https://news.ycombinator.com/item?id=22867803 https://news.ycombinator.com/item?id=22867803
- Havoc 3y agoNow we just need to put Cloudflare in front of it to really double down on the centralize the internet thing.
- buddylw 3y agoWell, there’s one thing I know didn’t cause this: an IPv6 DDOS attack. GitHub, in 2023, is somehow still immune to all IPv6 attacks
- brasic 3y agoNot quite true: github.github.io (aka github pages) has an AAAA record and should be fully accessible over IPv6.
- standardly 3y agooops, github committed the wrong github
- 8organicbits 3y agoWhat are folks using to isolate themselves from these sorts of issues? Adding a cache for any read operations seems wise (and it also improves perf). Anyone successfully avoid impact and want to share?
- blackoil 3y agoCoffee break.
- Karellen 3y agoUse the local clone that I already have, given that `git` was always intended to be usable offline.
- Zambyte 3y agoYep. I've been using my git server to mirror any and all software that I find slightly interesting. Instead of starring repos, I make them available when GitHub goes down :D
- ben0x539 3y agoPrevious job had a locally hosted Github Enterprise and I was always resentful when everybody else on Twitter was like "github down! going home early!". :( Of course it still sucked when some tool decided I needed to update dependencies which all lived on regular Github, but at least our deployment stuff etc still worked.
- manquer 3y agoDNS overrides during failure times and cloning those repos in GH Enterprise would be next logical next step I guess
- rwiggins 3y agoIn a previous life, for one org's "GitOps" setup, we mirrored Gitlab onto AWS CodeCommit (we were an AWS shop) and used that as the SoT for automation. That decision proved wise many times. I don't remember CodeCommit ever having any notable problems. That said: if you're using GitHub in your actual dev processes (i.e. using it as a forge: using the issue tracker, PRs for reviews, etc), there's really no good way to isolate yourself as far as I know.