6 ms·
GitHub was down again
GitHub is down once again!
- coleifer 6y agoLooks like it's affecting git operations for me. Yikes.
- qzio 6y agomhmm https://www.githubstatus.com/incidents/2y6v4ltq26g7 https://www.githubstatus.com/incidents/2y6v4ltq26g7 :(
- danra 6y ago> Currently observing issues affecting GitHub.com ?. Yes.
- JeroenKnoops1 6y ago:( a lot of colleagues are now sad... :(
- pupdogg 6y agotell me about it! I got an early start today wanting to push deploys before customer opened...well, now I can't!
- Epskampie 6y agoI'm so used to github being reliable that I was first checking my own machine when composer wouldn't install something.
- sergiomattei 6y agoHaha, I was trying to install a brew cask package and wondering why it wasn't working...
- pupdogg 6y agoSame here! I'm using WSL2 on Win10 and started getting "fatal: Could not read from remote repository" errors and thought it's definitely something to do with my WSL legacy to WSL2 upgrade.
- skytreader 6y agoSame. I'm of the opinion that GitHub's edge over GitLab is their uptime guarantee. But they've had issues for the past 3 days now.
- dubcanada 6y agoWhat are we at 5 downtimes this week?
- leeoniya 6y agoand the new notifications they rolled out are worse in some significant ways :(
- ivanfon 6y agoNot sure if they've fixed anything, but this blog post describes some of the issues: https://drewdevault.com/2020/03/13/GitHub-notifications.html https://drewdevault.com/2020/03/13/GitHub-notifications.html
- leeoniya 6y agoi reported a lot of this to them during beta, too. thank god they added back a button to group by repository. it was literally unusable otherwise - just a stream of disorganized crap. the repo name is still uselessly duplicated in every issue line despite the common heading when grouped. you can no longer "mark all as read" in a repo without going elsewhere to see the hidden notifs, pressing "select all" and clicking "done"...so next time you refresh the page without doing that, it all floats to the surface again. etc etc. ugh.
- Legogris 6y agoThey must have been anticipating this happening more often, given the replacement of the raging unicorn with a sad squidcat (or is it a catsquid?)
- cocktailpeanuts 6y agoi think they shuffle it up algorithmically. I am getting the unicorn, for example.
- Insanity 6y agoYup same here
- epoch_100 6y agoWhile I use (and mostly like!) GitHub, this is yet another reminder that centralizing so much infrastructure—from package managers to CI pipelines to static websites and more—around one company is a very bad idea, and will likely bite us in the end.
- aldanor 6y agoI always wondered about Rust packages - why GitHub? Were any alternatives ever discussed or proposed?
- jonsolo 6y agoIs that actually bad? A chain is only as strong as its weakest link, so if my system depends on many sites then even one outage can take me down. Centralizing like this can actually reduce risk.
- Epskampie 6y agoDepends on how it's done, if it's one authority with many mirrors it can be more reliable. Take for example `apt` in debian, you can always find a mirror online. With git's distributed nature, this would be very much possible.
- jonsolo 6y agoAgreed, that would be even better. I wonder what happened to GitHub that their entire HA system didn't work.
- dyingkneepad 6y agoWill you also mirror CI infrastructure, issues, pages, documentation, API-consulting scripts, etc? Also, pushing to a mirror is not exactly the thing you want. What makes apt easier is that pretty much everybody is just downloading from the source or from a mirror. When the source stops, mirrors don't get updated and everything still works. That's very different from the usage model that people have with Github. Gihub/Gitlab have extended the Git usage model very much, they are not just git. You can't easily migrate away from them to another git offering (and not even very easily between both).
- twistedpair 6y agoGeeze, so many outages/impacts (12+) in the last 90 days. The status page looks like a Pez dispenser. https://www.githubstatus.com/ https://www.githubstatus.com/ Usually companies put in place a release freeze or "Code Purple" when there are such demonstrated problems with releasing stable code.
- SirensOfTitan 6y agoSeriously considering moving our projects off of GitHub at this point. What other services are good? I loved using Phabricator at Facebook, but its cloud option is pricy.
- boudin 6y agoI'm considering self-hosting gitlab.
- reledi 6y agoHaving been down that road, it's also a mess. Best I can imagine now is using GitHub as a primary and other hosted services as a mirror. Mirroring the git repos will only get you so far though, you won't have the services built around them (e.g. issues, merge requests, comments, actions) or you may not be mirroring all git repos that you need access to. But a reduced service may be sufficient for short periods during an outage.
- leesalminen 6y agoHow so? We've been self-hosting Gitlab for 2.5 years now (maybe more?). Besides a regular update (through apt) we haven't had any trouble. Team of 5 developers w/ a dozen or so repos and basic CI/CD builds. I highly recommend it.
- reledi 6y agoWe didn't have a better SLA than hosted GitHub when self-hosting GitLab, and then there's the extra cost of maintaining the instance, the CI runners, the DB, the networking, configuration and so on. Wasn't worth it for us after using self-hosted for 2+ years with a tech team ranging between 10-25. On top of that, most of the team thinks GitHub has a better UX and enjoy using it more.
- boudin 6y agoCan you share your experience?
- chacha102 6y agoFortunately, Github is built on a distributed system that doesn't require us to stop coding just because we can't access a central server. Distributed systems for the win!
- leeoniya 6y agoexcept that it also serves as a communication platform, like issues, PRs, code review, CI. the fact that git is distributed is but a small part of the whole.
- slowmovintarget 6y agoWhich is why it gets targeted by "state actors". I'm not saying that's what this is, but it's happened before.
- YetAnotherNick 6y agoWhat is distributed here? If github is down no one can see what you changed, not even the build servers. Git is distributed in the sense that every client has complete history, but not in the sense that my laptop can act as replacement of github(like torrent).
- chrisseaton 6y agoGit supports lots of protocols for sharing your changes - even email!
- YetAnotherNick 6y agoDiffs can be shared over mail, right. Can I do a `git pull` via mail?
- chrisseaton 6y agoYou can't over mail. But each of you could run your own Git server if you wanted to be able to pull.
- chatmasta 6y agoHave they provided any insight as to the cause of the recent spike of downtime? Any post-mortems or anything like that?
- archi42 6y agoI am often a bit sad that we don't have all the fancy pants GitHub features. But right now I'm quite happy I can still use my git, the CI runs quite well, and everything else is quite dandy overall =)
- hirundo 6y agoSingle Point of Failure As A Service
- _nhh 6y agoSeeeelf hosting!
- reledi 6y agoJust switched back to using GitHub again and the timing couldn't be worse. Hope things are going okay for them, definitely some stressful weeks. On a side note, the status.github.com page is quite delayed. Following live updates from Twitter has been a better strategy for confirming that GitHub is having issues: https://twitter.com/search?q=github&src=typed_query&f=live https://twitter.com/search?q=github&src=typed_query&f=live
- dbingham 6y agoWhat was that we were all saying about Microsoft being on an upward trend? Years and years of stability to the point where it starts to taken for granted. About a year post Microsoft purchase, and here we are.
- adrianvoica 6y agoExactly.
- tylfin 6y agoI would really like to see a postmortem for the recent outages. While I doubt it's all the same root cause, it would be nice for some messaging around improving resiliency. I think the outages could be related to: 1. Github mobile just went public so they may of changed the scaling params to keep up with expected increase in traffic 2. The new notification system seems to be a lot heavier, and they could still be catching up to the changes 3. They were somewhat recently acquired by Microsoft so maybe they're migrating to Azure to reduce expenses Whatever the cause, 11 days with outages out of 90 is pretty rough when you rely on Github for project management, a central hub for viewing CI, and all VC concerns. Feel bad for the smaller companies that wholly adopted GitOps and are blocked deploying hotfixes during these outages.
- jlgaddis 6y ago> I would really like to see a postmortem for the recent outages. While I doubt it's all the same root cause, it would be nice for some messaging around improving resiliency. They recently posted a "post-incident analysis" [0] about several "service disruptions" in February that were all a result of database issues. Obviously, I have no idea if today's outage is related. --- [0]: https://github.blog/2020-03-26-february-service-disruptions-post-incident-analysis/ https://github.blog/2020-03-26-february-service-disruptions-...
- sbuccini 6y agoGitHub Enterprise as a quarterly Uptime SLA of 99.95% It's probably worth checking to see if they've violated it this quarter. The status page says that their uptime is 99.92% YTD but their support page says that the status page is not connected to their internal metrics. [0] [0] https://help.github.com/en/github/site-policy/github-enterprise-cloud-addendum#enterprise-cloud-uptime-sla https://help.github.com/en/github/site-policy/github-enterpr...
- kalev 6y agoBe glad you're not on Bitbucket though. Issues happen so often people stopped posting it to HN.
- bmgoau 6y agoI feel Bitbucket has been remarkably stable this year. The difference between the Github Statuspage and Bitbucket's is night and day. Of the two recent issues I can see in the last month they look like minor glitches and not outages. Bitbucket had an unfortunate series of interruptions for a couple of days in October last year, but Github has been having major outages every few days for 3 months now.
- chkaloon 6y agoWhy is this post flagged? Seems like an appropriate HN topic
- marceloabsousa 6y agoAgreed.
- dang 6y agoThe more repetitive a topic gets, the more users flag it. 'Is down' posts are repetitive as a category to begin with, and 'X is down again' posts are repetitive along two axes. There was one of these a few hours ago, before it got flagged: https://news.ycombinator.com/item?id=22955595 https://news.ycombinator.com/item?id=22955595 ... which was likely because users regarded it as a quasidupe of https://news.ycombinator.com/item?id=22935941 https://news.ycombinator.com/item?id=22935941. By 'quasidupe' I mean it's not strictly the same story, but the difference ('down again') isn't significant enough for the thread to be substantially different. (The submitted title on this one was "GitHub: Here We Go Again".)
- headalgorithm 6y agoI understand your argument, but the number of comments on this post shows people still want to discuss it. Is it possible to run an experiment. Remove the flag option but highlight the hide option to users. Users who are not interested in the topic can remove it from view but do not kill the discussion for those who are interested in it. Then see if the quality of posts/discussions decline.
- jlgaddis 6y agoEvery "GitHub is down [again]" post leads to the same discussion. With the number of recent GitHub "service disruptions", it starts to get a bit old.
- arghblarg 6y agoConsider self-hosting via gogs.io or gitea, and use them to mirror your key repo dependencies ...
- Havoc 6y ago>use them to mirror your key repo dependencies ... Is this possible with a self-hosted gitlab? Trying to get that to push to a cloud repo (GCP/Azure/whatever) as backup. Managed to selfhost gitlab but at the edge of my technical git knowledge here
- Yetanfou 6y agoIt almost feels like HoTMaiL all over again. For those too young to remember, Microsoft bought Hotmail (forget the silly CamelCasing, we know that HoTMaiL was HTML + mail by now) which was based on FreeBSD+Apache and was champing at the bit to use it to demonstrate the scalability and stability of their then relatively new NT operating system and the IIS web server. Let's just say that... things did not go the way Microsoft would have wanted and the demonstration more or less achieved the opposite of what they intended. It took them a long time to move the frontend to Windows and an even longer time to do the same to the backend. They won't make that mistake again but they might succumb to featuritis or wrongfooted attempts to steer Github-users further and further into the Microsoft world.
- duxup 6y agoI remember signing up for Hotmail and abandoning my isp email as if it was some sort of internet activism ;)
- deleted 6y ago[deleted]
- johndavid9991 6y agoExperienced this last week, too, we worked hard to complete a feature, and we were ready and excited to merge and push the update to production, then GitHub was suddenly down. We didn't sleep well that night. We could have done some workarounds but just too frustrated to do so.