11 ms·
GitHub issue - resolved
Seems like GitHub's struggling at the moment: https://downdetector.com/status/github/
I'm getting unicorns.
- mabbo 4y agoI feel like HN could just have a traffic monitor that adds a little icon to the main page. Like "I dunno what's up, but there's a LOT of y'all here right now, so something probably is".
- absoflutely 4y agoI think you meant "I dunno what's down..."
- mabbo 4y agoEven better
- deleted 4y ago[deleted]
- Bostonian 4y agoSuppose you start your day by downloading some code from GitHub to work on. This morning you would be stuck. Do you save code on your hard drive or company servers or GitLab to handle this risk?
- wizzwizz4 4y agoYou just clone the repo from one of your colleagues'.
- SketchySeaBeast 4y ago"GitHub is down" probably wouldn't even make the top 10 stupidest reasons that I couldn't work. Is the idea here that you're downloading brand new code? I would imagine most people would have a stale from Friday copy of the repo locally.
- bouncycastle 4y agoEver since Microsoft took over...
- onedr0p 4y ago...Github has gotten significantly better.
- donohoe 4y ago...and Github is down... https://www.githubstatus.com/ https://www.githubstatus.com/
- rvz 4y ago...and out every single month to no end to the outages...
- sebzim4500 4y agoDid Github never go down before the acquisition? Would be interesting to see stats.
- hk__2 4y agoand up again.
- deleted 4y ago[deleted]
- lima 4y agoProbably not a coincidence - changes are the #1 reason of outages. Moving faster does mean breaking more things.
- deleted 4y ago[deleted]
- MattIPv4 4y agoAlso seeing unicorns, mostly for "No server is currently available to service your request.". If I don't get a unicorn, the page takes a long old time to load.
- CipherThrowaway 4y agoBack up for me
- manmal 4y agoThe site is up but git push still doesn’t work for me as of now.
- CipherThrowaway 4y agoJust found that out the hard way.
- can16358p 4y agoYup. Tried pushing and got "remote: fatal error in commit_refs". Was trying to understand what I was doing wrong.
- vincentkriek 4y agoSame here! Was going crazy what I was doing wrong. Renamed my branch to see if that was causing anything.
- themanmaran 4y agoSmall world. I assumed I had created a duplicate branch name or something, and renamed the branch as well before hitting up google.
- activitypea 4y agoFor me it went down just as I was logging in, I thought I was banned or something :')
- Avicebron 4y agoSame, I thought they had finally come for me because I refuse to change master to main.
- czx4f4bd 4y agoSame here haha. StackOverflow informed me that it means GitHub is down, and indeed it was. I have to wonder if Git could somehow report this better. I guess it depends on exactly how GitHub is down, but "fatal error in commit_refs" made me worry that my local repo was somehow hosed.
- TheSoftwareGuy 4y agoI think it can? whenever I can't connect to my company's git server I get a `failed to connect to git.company.com` message. or something like that
- 4y ago
- MattIPv4 4y agohttps://www.githubstatus.com/incidents/52z0j6phhnjs https://www.githubstatus.com/incidents/52z0j6phhnjs
- tyingq 4y agoSeems better done than a lot of status pages, runs on separate infra, updated, has a way to subscribe, etc. However, saying "degraded performance" when you know it's "down for everyone" is an industry phrasing thing that's irritating. AWS also has "elevated response times" when everyone is seeing 5xx errors, or infinite response times.
- MattIPv4 4y agoFor what it's worth, it's powered by statuspage.io, which is relatively industry standard for status pages.
- kachnuv_ocasek 4y agoIt's not down for everyone. I can browse just fine, but my pushes get rejected, so that qualifies as "degraded performance" for me.
- tyingq 4y agoIt's got many named granular services marked as "degraded performance" or "degraded availability" that seemed to be down for everyone.
- pizza234 4y agoIt seems accurate to me - after some time, several "degraded performance" flags have been changed to "major outage".
- glogla 4y ago> However, saying "degraded performance" when you know it's "down for everyone" is an industry phrasing thing that's irritating. AWS also has "elevated response times" when everyone is seeing 5xx errors, or infinite response times. Another popular one is "elevated API error rates" when the error rate is 1.
- deleted 4y ago[deleted]
- TheLocehiliosan 4y agoAlso known as Monday.
- erkanerol 4y ago[flagged]
- deleted 4y ago[deleted]
- Phemist 4y agoThis was done out of an abundance of caution, I'm sure.
- LeonM 4y agoFor those who missed the context of this joke: https://news.ycombinator.com/item?id=35285390 https://news.ycombinator.com/item?id=35285390
- bqmjjx0kac 4y agoCan’t get hacked if you’re offline.
- knorker 4y agoExcept for your bitcoins, if your cold wallet had a bug: https://research.kudelskisecurity.com/2023/03/06/polynonce-a-tale-of-a-novel-ecdsa-attack-and-bitcoin-tears/ https://research.kudelskisecurity.com/2023/03/06/polynonce-a...
- rvz 4y agoIt goes down every month like I said before [0]. The last time this happened was 2 days ago [1] then weeks ago [2] and it is evident that it is falling apart in front of us. First, it was the RSA key leak in [1][3], then the site's key expired causing down time again [2] and now this. I don't think anyone can tell me with a straight face that GitHub was any more reliable or better when Microsoft acquired it. It is now worse off. Nothing has changed except for more outages and downtime. So so reliable. /s [0] https://news.ycombinator.com/item?id=35004629 https://news.ycombinator.com/item?id=35004629 [1] https://news.ycombinator.com/item?id=35295216 https://news.ycombinator.com/item?id=35295216 [2] https://news.ycombinator.com/item?id=35003741 https://news.ycombinator.com/item?id=35003741 [3] https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/ https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-k...
- johnlbevan2 4y agoNB: GitHub was purchased by MS 5 years ago. All these events are in the last month. It's apparant that it wasn't without issues prior to acquisition (e.g. a quick search for GitHub issues prior to 2018 gives this: https://techcrunch.com/2017/07/31/github-goes-down-and-takes-developer-productivity-with-it https://techcrunch.com/2017/07/31/github-goes-down-and-takes...) - reporting issues in 2017, 2015, and 2012. I don't have the data to comment on whether it was better before or after MS acquisition, but would suggest this isn't the best sample size to base any conclusions on.
- powerage 4y agoI have been getting error when I try git push. I taught the problem is me but I saw this thread.
- expotesticle 4y agoThanks for ruining my day
- hakanderyal 4y agoBecause this needs to happen as one prepares to push to day's last deploy. Back to backups. Have backups for critical systems, people. In my case, it's building docker containers locally and luckily deploying to one server via ssh.
- TechBro8615 4y agoBecause if you don't spend multiple days per year managing your own "backup" to GitHub, you might be left unable to push to GitHub for a few hours per year? btw, I hope none of your CI system relies on build steps that might include pulling code from GitHub or downloading packages from GitHub Package registry. Often when GitHub is down, my CI system on GitLab is broken too.
- hakanderyal 4y agoI like to reduce the single point of failure count in my projects as much as I can, with common sense. My backup system is running a few cli commands instead of git push, so it's a no brainer. YMMV.
- TechBro8615 4y agoHow sure are you that none of those commands has a dependency on GitHub infrastructure?
- hakanderyal 4y agoBecause I use Github only for git hosting and actions (building & deploying containers). In this case, code is already in my computer, and I can build and ssh to server and pull&run the new container. "docker buildx build ... -push" "ssh ...@..." "docker pull ..." "docker compose up ..."
- TechBro8615 4y agoThat assumes that none of your Docker images have a build step that includes interacting with GitHub, or if some do, that you have every affected layer already cached on your local computer.
- tomconte 4y agoTime for another coffee
- larsnystrom 4y agoHN should really have a “Down HN” category, as this is the most reliable place to get this information.
- jonas-w 4y agoEverytime something has degraded performance or is down for me: 1. check my internet connection 2. check HN 3. check official statuspage
- stavros 4y agoSurely you can skip the first step, as the second combines them.
- XCSme 4y ago> check my internet connection How do you do that?
- mpsprd 4y agoHN is de facto the fastest secondary source for any current event related to tech.
- sveeeeeene 4y agoI also use it for connectivity check. Most other sites have so much ads and cookies it creates long lags with adblock in some older systems I use. So I point to HN and get a really fast response that the web is working
- vrosas 4y agoOne time HN was itself down and that was a confusing experience
- ithkuil 4y agoI check HN first because that also checks my internet connection :-) that said, HN is also down a lot lately, but that the kind of outage that makes more more productive actually!
- henriquenunez 4y agoSeems it's fixed now?
- deleted 4y ago[deleted]
- rurban 4y agoread-only
- oneeyedpigeon 4y agoRelated to the Twitter source code leak, perchance?
- dataengineer56 4y agoGiven that looking at repos is just about the only thing that I can do then I'd say that seems unlikely.
- oneeyedpigeon 4y agoI was more implying that this could be a result of an attack borne out of retribution.
- SketchySeaBeast 4y agoIs the implication that someone affiliated with Twitter took down GitHub because of something someone hosted there? Does that honestly seem plausible to you?
- oneeyedpigeon 4y agoI'm not saying "that happened", I'm asking about the possibility. Does it seem plausible that someone sympathetic to Twitter launched an attack on GitHub in retaliation? Absolutely.
- SketchySeaBeast 4y agoHonestly, while it's certainly possible it doesn't seem plausible to me. I have a hard time seeing someone with the extreme emotional connection needed to lash out because of the leak also having the technical sophistication required to bring GitHub down.
- new_user_final 4y agoWhy GitHub is down so often? Why is it not possible to keep it up 100% of time (without counting physical failure)?. I haven't seen any down time for my system (it has hundreds of thousands of users online) in months since I have completed the setup.
- zensayyy 4y agoI find this strange too. GH seems to have more major incidents lately...
- raxor53 4y agoI'm sure your scale is similar.
- substation13 4y agoGoogle search is basically never down. AWS is basically never down. WhatsApp is basically never down. Time for GitHub to grow up?
- sdfhbdf 4y agoRO system is generally probably easier to keep up than a RW system that is constantly innovating.
- bhouston 4y agoI Think he was referring to Google Search in general. I've never witness since Google went live in 1998 any Google Search downtime. Probably happened but I can not remember it.
- sdfhbdf 4y agoGoogle Search is still a RO system - you are mostly just retrieving information from a search index.
- bachdx 4y ago[dead]
- 0xcb0 4y agoI cant push anymore! Anyone else ?
- gitgud 4y agoHmm this is quite an incident, I haven't seen Github reject all pushes to all repos like this in quite some time.... hopefully not SSH key related
- Madmallard 4y agook this makes sense i was just trying to push
- willejs 4y agoI used to get my github status updates in slack via an RSS feed, and just searched for the feed again, but its gone? Is there an alternative for this?
- bennyp101 4y agoClicking "Subscribe To Updates" top right on https://www.githubstatus.com/ https://www.githubstatus.com/ gives me a whole load of options for getting updates, one of those is RSS
- vidluther 4y agoI feel like events like these are the new "compiling".. https://xkcd.com/303/ https://xkcd.com/303/
- siva7 4y agoCodespaces is not connecting anymore for me
- kukimonsta 4y agoI am getting unicorns!
- tarruda 4y agoMaybe they are experimenting with GPT automating devops
- davemp 4y agoNow's a good opportunity to ask about alternative Git hosts. What other services do HNers use? I've been unwilling to host any personal projects on GH after Copilot launched and it because clear that GH/MS doesn't really respect the authors of the code they host. Honestly open source in general has gotten a little less compelling to me after Copilot. The recent security issue at GH has also turned me off even more on Git hosting services. For closed source projects maybe it's just best to store encrypted backups off site and spin up a self hosted option whenever collaboration is needed. Seems pretty inefficient though.
- eloisius 4y agoI’ve been happy with SourceHut
- blueflow 4y agoI have a vps with bare git repos, pushing via ssh. A selection of these has hooks set up to mirror all pushes onto github/gitlab/$platform, but these are write-only.
- davemp 4y ago> I have a vps with bare git repos, pushing via ssh. A selection of these has hooks set up to mirror all pushes onto github/gitlab/$platform, but these are write-only. I like it, self-hosting bare git repos has been pretty painless for me in the past on LANs. You could still add a hook to encrypt the repo and backup whenever you merge to dev or something as well. You pretty much only lose the hosted diff/review/ticketing tools which I've never enjoy much regardless.
- _0w8t 4y agoAt my previous work almost everything was self-hosted including Git and servers to run machine learning models. The only exception was Jira. The company owned the servers and rented the space in a datacenter. For code reviews we had Critic. At the new work we use BitBucket, but for reviews its UI is strictly worse than Critic. And on top of it it is strictly more expensive than self-hosting experience including paying for a competent sysadmin. I understand that cloud allows to offload a lot of headaches, but I really see no point in using cloud services for development. Even for a small company a dedicated server with another to spare in case of failures will be cheaper and it’s administration will be trivial.
- belter 4y agoGit Docs... - https://git-scm.com/book/en/v2/Getting-Started-About-Version-Control https://git-scm.com/book/en/v2/Getting-Started-About-Version... "...The next major issue that people encounter is that they need to collaborate with developers on other systems. To deal with this problem, Centralized Version Control Systems (CVCSs) were developed. These systems (such as CVS, Subversion, and Perforce) have a single server that contains all the versioned files, and a number of clients that check out files from that central place. For many years, this has been the standard for version control." "...However, this setup also has some serious downsides. The most obvious is the single point of failure that the centralized server represents. If that server goes down for an hour, then during that hour nobody can collaborate at all or save versioned changes to anything they’re working on. If the hard disk the central database is on becomes corrupted, and proper backups haven’t been kept, you lose absolutely everything..." "...This is where Distributed Version Control Systems (DVCSs) step in. In a DVCS (such as Git, Mercurial, Bazaar or Darcs), clients don’t just check out the latest snapshot of the files; rather, they fully mirror the repository, including its full history. Thus, if any server dies, and these systems were collaborating via that server, any of the client repositories can be copied back up to the server to restore it. Every clone is really a full backup of all the data...."
- williamvds 4y agoGit avoids the problem where the central service being down gets in the way of local development. Even in Git, that central service going down means degraded collaboration. Normally a Git remote is just an ssh-accessible machine, and so pretty resilient. But GitHub is a lot more complex, so apparently that simple service went down, along with all the features they built on top of it
- imglorp 4y agoYou don't actually need the central service. In an emergency, a team could work entirely over email via diffs like Linus does.
- williamvds 4y ago
- drewbug01 4y agoOne wonders if MSFT leadership will ever connect the dots between “hollowing out talent over the years,” “hiring freezes,” and “layoffs” to this outcome. One wonders sometimes if that’s the goal.
- mnsh 4y ago> We’ve identified an infrastructure change that has been rolled back and we are monitoring recovery. From github status
- tibanne 4y agoOne has to ask oneself... would this have happened without the Microsoft acquisition?
- pxc 4y agoDoes anyone have real, contemporary numbers on this? Your observation matches my intuition, and I've seen some stats around the first two years after the acquisition¹, but I don't know any good up-to-date analysis on the question. My intuition aligns too closely with my known biases here for me to be satisfied with that alone. 1: https://statusgator.com/blog/has-github-been-down-more-since-its-acquisition-by-microsoft/ https://statusgator.com/blog/has-github-been-down-more-since...
- yla92 4y agoEverything is green now!
- dark-star 4y agoseems to work fine for me...?
- robinhoodexe 4y agoIs anyone self-hosting the enterprise version of github as a backup? Or using it as the primary source and then using the cloud version as backup?
- lesquivemeau 4y agoIt seems like you can push again !
- tiffanyh 4y agoOnly code with Github is replicated locally, not everything else like wiki, issues, etc. That's something I love about Fossil. Everything with Fossil (wiki, issues, code) is replicated as well.
- ghostwriter 4y agoYou can keep issues within git itself: https://github.com/MichaelMure/git-bug https://github.com/MichaelMure/git-bug
- Fabricio20 4y agoLooks like it's back and the status page reports it as green, however GitHub Actions builds are not triggering on push as usual when this happens, so for me it doesn't seem to be fully recovered yet.
- dolmen 4y agoLooks OK from here (Paris, France).
- tomaszcekalo 4y agoI wonder if it has anything to do with them updating their RSA SSH host key (https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/ https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-k...) 3 days ago.
- xwdv 4y agoIt’s up now.
- 5mv2 4y agoAre there downtime safe GitHub alternatives?
- eitland 4y agoYou can always sync over local (or remote) ssh or fileshares or even zip up smaller repos, sneakernet them over to those who need them, unpack it there and pull from the unzipped folder. The pros who maintain git uses email(!), but I think that would take more time than just waiting out the outage.
- dainiusse 4y agoCommited code straight from copilot :)