9 ms·
GitHub is down
- openthc 3y agoMine looks like (on git push): remote: Resolving deltas: 100% (4/4), completed with 4 local objects. remote: fatal error in commit_refs To $REPO ! [remote rejected] main -> main (failure) error: failed to push some refs to '$REPO' Status Page: https://www.githubstatus.com/ https://www.githubstatus.com/ -- actually shows red -- it was almost updated as fast as HN.
- SketchySeaBeast 3y ago> Status Page: https://www.githubstatus.com/ https://www.githubstatus.com/ -- actually shows red -- it was almost updated as fast as HN. I think we should applaud this when it actually happens. Far too many services are terrible for this. While I'd rather the service not be down, it makes me feel a bit better if they don't lie about it.
- paulddraper 3y agoIn a world of green, an accurate status page is a radical act.
- geephroh 3y agoI'd buy this t-shirt
- BossingAround 3y agoIt's funny when a company like Amazon, which many would think is on the cutting edge of automation, implements their status page as a manually updated static page.
- Alupis 3y agoIt's not on accident. There's SLA's and other things on the line when the status page officially recognizes an outage.
- ehayes 3y agoHmm, will it turn out to be DNS?
- belter 3y agoDNS...Developers Need Solutions
- itsbenweeks 3y agoI'm guessing so, given the breadth of the outage. (It's always dns.)
- shagie 3y agoIn the various tools for determining blame: https://isitdns.com https://isitdns.com https://shouldiblamecaching.com https://shouldiblamecaching.com https://bofh.d00t.org https://bofh.d00t.org nc bofh.jeffballard.us 666 # I am surprised this is still working given its age; the CGI version at https://pages.cs.wisc.edu/~ballard/bofh/ https://pages.cs.wisc.edu/~ballard/bofh/ doesn't work anymore (but the page is still up)
- roughly 3y agoHearts out to all the GitHub SREs who get to fix this, and all the SREs at GitHub’s customers who get to spend the next week convincing management and their peers that moving git in house won’t result in fewer outages.
- andrewfromx 3y agoso true. At this moment everyone's like, well! This is UNACCEPTABLE. We'll just, we'll just... what, what are you going to do better than github?
- no_wizard 3y agoI always wondered if some % of GitHub orgs move to Gitlab when this happens. I'd be interested to see if there is any correlation there.
- slg 3y agoYeah, because Gitlab would never go down. https://hn.algolia.com/?q=gitlab+down https://hn.algolia.com/?q=gitlab+down
- dmonitor 3y agoGitlab can be self hosted
- dmattia 3y agoas can Github: https://docs.github.com/en/enterprise-server@3.5/admin/overview/about-github-enterprise-server https://docs.github.com/en/enterprise-server@3.5/admin/overv...
- slg 3y agoAnd we all know self-hosted software never goes down. Although I guess the benefit is that the CTO gets a psychological lift of looking over Jeremy from IT's shoulder when he tries to fix the issue rather than being at the mercy of some engineers which the company can't fire.
- bityard 3y agoI sense a disturbance in the force...
- belter 3y agoyou mean... I sense a disturbance in the source...
- andrewfromx 3y agoThe Source is what gives a hacker his power. It's an energy field created by all coded things. It surrounds us and penetrates us. It binds the galaxy together.
- Phemist 3y agoOut of an abundance of caution...
- andrewfromx 3y agoon May 4th too. Hmmmmm
- gondaloof 3y agoEverything is down. How can this happen? Many of these services are unrelated, like even the brand-new Codespaces and Copilot are completely down. Do they all have a single point of failure? (Presumably the git server)
- ehayes 3y agoYeah but they probably all share DNS or something related to networking
- zackkitzmiller 3y agoDNS. It's always DNS
- capableweb 3y agoWhen in doubt, and the outage is widespread it's either DNS or BGP, most likely DNS. My money is also on DNS in this case.
- TimWolla 3y agoI suspect you'd lose that money. Many of the recent GitHub outages were caused by database issues. Examples: https://github.blog/2023-05-03-github-availability-report-april-2023/ https://github.blog/2023-05-03-github-availability-report-ap.... In fact last march they specifically mentioned “Database stability”: https://github.blog/2022-04-06-github-availability-report-march-2022/ https://github.blog/2022-04-06-github-availability-report-ma...
- jallen_dot_dev 3y agoIt's not DNS There's no way it's DNS It was DNS
- brookst 3y agoDNS, BGP, PKI, probably other TLAs.
- WookieRushing 3y ago
- edgyquant 3y agoWonder why mine[1] got marked as a duplicate when it was posted first? 1. https://news.ycombinator.com/item?id=35817640 https://news.ycombinator.com/item?id=35817640
- andrewfromx 3y agoso unfair. This happens to me too sometimes. It's like not a perfect system.
- sva_ 3y agoTough making a living on imaginary internet points.
- andrewfromx 3y agohey, it's a livin. Mcconaughey agrees. I just keep on getting points.
- Reubachi 3y agoWhile I understand where you as a person are coming from, and my below thought is a tangent, I think this sentiment is like the origin story of editorialized headlines. If one gets dopamine/validation/click thru revenue from others consuming/clicking on media one shared,and further feel "unfairly gamed" when someone else gets that payout, it's a short time before one starts subconsciously appealing to the human lizard brain and sensationalizing media.
- andrewfromx 3y agowell put!
- hombre_fatal 3y agoMaybe because they linked to the incident instead of just the domain homepage that other people tried to post.
- 3y ago
- irrational 3y agoHuh, it is working for me. Is it region specific?
- buildbot 3y agoStatus updated 1 minute ago saying resolved so maybe they fixed it?
- mvanbaak 3y agoTheir twitter account states that allnis back to normal
- llama052 3y agoI wonder if they are migrating most of their platform to Azure.
- deleted 3y ago[deleted]
- inductive_magic 3y ago>This incident has been resolved. Posted 1 minute ago. May 04, 2023 - 16:23 UTC Everything resolved at the same time. DNS?
- romellem 3y agoAccording to the same status page, this is Resolved as of May 04, 2023 - 16:23 UTC.
- rvz 3y agoLast time GitHub went down was less than a month ago: [0] At this point maybe it is time to self-host rather than to continue to tolerate this since it's guaranteed to goes down every month. [1] Even open-source projects like RedoxOS, GNOME, KDE, ReactOS, etc are doing just fine without worrying about GitHub's unreliability as predicted years ago on not centralizing everything to GitHub. [2] [0] https://news.ycombinator.com/item?id=35611739 https://news.ycombinator.com/item?id=35611739 [1] https://news.ycombinator.com/item?id=35611862 https://news.ycombinator.com/item?id=35611862 [2] https://news.ycombinator.com/item?id=22867803 https://news.ycombinator.com/item?id=22867803
- joshmanders 3y ago> At this point maybe it is time to self-host rather than to continue to tolerate this since it's guaranteed to goes down every month. Not a single one of these outages has caused me enough pain to even consider moving to another hosted provider let alone self-hosting and the burden that goes into managing all that stuff. Can we stop with this whole "omg a major platform that gets used by millions of people a day can't guarantee 100% uptime, we need to all manage our own infra!!" mentality? Seriously I've been in GitHub all morning and decided to take a break and read HN and I was JUST learning about this now... It was resolved before I seen this thread too.
- deleted 3y ago[deleted]
- dom96 3y agoThe fact that GitHub being down can have such an impact means that Git’s decentralised features don’t go far enough. Why aren’t issues, PRs and all these other little things we depend on also stored in a git repo? Or something similar that can withstand GitHub being down. I set aside time today to go through my notifications and I can’t because of this. Notifications are trickier to decentralise but still, ugh.
- tcoff91 3y agoThose who use it by emailing patches around aren't affected, ala the linux kernel.
- Kwpolska 3y agoThey aren’t affected until the mailing list servers go down.
- capableweb 3y agoCouldn't find any stats online about it, but how often does the servers for the mailing daemons for the linux mailing list go down?
- SSLy 3y agoThen you can send PR's to the maintainers' direct inboxes, that'd still work.
- Domenic_S 3y agoheh, this was my argument back in like 2009 when everyone was all, "DECENTRALIZED!! We have to take months to migrate from SVN to git cause DECENTRALIZED!! You don't one a single point of failure do you?!" all I could say was, uhhhh github is a single point of failure. Unless you're mailing diffs around like Linus (which nobody I knew did), it's 6 of one, half-dozen of the other. there are lots of great things about git and especially git over e.g. svn (especially merges) but I never bought the decentralization argument the way it was spearheading the conversation back then.
- amelius 3y agoQuick, activate the Copilot-based sysadmin!
- whimsicalism 3y ago27 minutes to resolve - relatively fast
- buremba 3y agoI'm not really sure Github prioritizes the high availability anymore. These outages might be a way to motivate the customers switch to Github's self-hosted enterprise solution which is the revenue stream for Github. If they're not losing users because of these outages, (I'm sure they can find that out) these outages really make them more money in the long run..
- HereBeBeasties 3y agoAs a previous GitHub Enterprise Server customer on a site with hundreds of users, who's spoken with a number of GitHub employees about this, I do not think that is true. They would far rather you were on GitHub Enterprise Cloud. They view Server instances as a necessary evil for people who have compliance or security requirements they can't handle with the Cloud version, but it is fairly obviously a pain in the neck for support, and attempting to attain feature parity. They have been trying to bring their Server offering closer in architecture to the Cloud version since forever (e.g. moving to Nomad a few years ago to orchestrate it all as containers, so you can shard the services out over multiple hosts similar to github.com, etc.).
- mimd 3y agoI see GitHub is working hard to achieve nine fives.