12 ms·
GitHub was down
- prh8 9y agoThis is the 3rd major outage in a week, wow.
- sinaa 9y agoApp server availability is now 98.3% over the past month, which seems pretty bad!
- travisjungroth 9y agoGood ol' one 9 uptime.
- ki85squared 9y ago"One nine up-time" rolls right off the tongue! Must be a PR play!
- cheeze 9y agoPoor gals and guys. Godspeed to them.
- kchoudhu 9y agoDid they outsource their ops to Infosys? This is the third time this week.
- zakshay 9y agoNot sure yet, but it seems to be due to AWS (S3 likely).
- carsongross 9y agoWow. Once is an accident. Twice is a coincidence. Three times is an enemy action.
- zackify 9y agoBoth days a different error on pushing, making me think it wasn't down and it was my fault!
- deleted 9y ago[deleted]
- runn1ng 9y agoAs the last time, pull/push access to repos works fine for me, only the web service doesn't work. edit: confirming, push doesn't work either
- praneshp 9y agoTangential: Slack was down for a bit earlier today as well.
- anonu 9y agoWhat do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...
- H1Supreme 9y agoGitlab?
- Robin_H 9y agoDoes anyone know an alternative to githost.io? We don't want to use gitlab.org as it's painfully slow but also don't want to manage our own gitlab instance. edit: We don't like the recent changes made in githost. ( Replaced the $35 option with $149 ) We're a team of 15 people. edit2: should have clarified; I'm hooked to GitLab, just looking for another service that does managed hosting. Not looking for GitLab alternatives.
- ShakataGaNai 9y agoGitlab.com? BitBucket is the other major player that comes to mind, but googling "hosting git" returns enough results to melt ones mind. If you're really concerned, you run it yourself. GitLab insider a docker container isn't hard to manage. You can even 1-click Gitlab onto a DO Droplet in just a few minutes: https://www.digitalocean.com/community/tutorials/how-to-use-the-gitlab-one-click-install-image-to-manage-git-repositories https://www.digitalocean.com/community/tutorials/how-to-use-...
- m_sahaf 9y agoGitea[0] is extremely simple to host, if you want a self-hosted solution that's not as heavy as gitlab. [0] https://gitea.io https://gitea.io
- kuschku 9y agoGitea sadly has some issues (for example, the "Public Activity" tab on user profiles 500's atm: https://try.gitea.io/crackedmind?tab=activity; https://try.gitea.io/crackedmind?tab=activity; it throws 500's everywhere once you throw on a modern Xeon more than 30 users/second at it, because it doesn't cache database accesses, at all, etc).
- ben174 9y agoRough year for GitHub. GitLab is licking their chops right about now.
- Robin_H 9y agoUgh, time to go to bed then.
- deleted 9y ago[deleted]
- grzm 9y agohttps://status.github.com https://status.github.com shows > 18:19 CDT Major service outage.
- amgin3 9y agoThey shutdown the servers because they discovered that at least half of them didn't identify as female.
- sctb 9y agoPlease stop posting unsubstantive inflammatory comments (aka trolling). https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- Cofike 9y agoDoes this mean we get the day off?
- nicklaw5 9y agoI'm down for that :D
- Woofles 9y agoSeems like they're saying it's maintenance now? $ gl fatal: remote error: GitHub is offline for maintenance. See http://status.github.com for more info.
- jdelsman 9y agoThat is likely just their standard downtime message.
- Woofles 9y agoOnly reason I brought it up because it was originally showing the all to familiar unicorn on the website, then it changed to also saying maintenance.
- beager 9y agoThis seems like a pretty common response for a breaking incident for a scale app. Requests flow through to a failing system and trigger HTTP 500. Those requests may pachinko through the stack, making a variety of calls that can compound the degradation of a system weathering an unplanned failure state. Engineers stop the bleeding by 503'ing requests at the perimeter or putting up a static maintenance page. This allows things like caches or DBs or app servers to cool off while a rollback or a revert goes out. Then, when the system is stable again, let requests flow through again (slowly, of course).
- NicoJuicy 9y agoIt's sometimes up ( perhaps bad to say this --> everyone will refresh )...
- deleted 9y ago[deleted]
- morgzilla 9y agoAnd now the office goes to the keg.
- acchow 9y agoif only we had a decentralized version control system...
- ben174 9y agoExactly this. It's painful to realize that Git was invented to solve this very problem, but so many use GitHub as their source control server just like CVS and all the other old source control systems.
- EduardoBautista 9y agoThen what's the alternative? Push and pull directly from other users who have access to the repository? What if that user is currently offline? There are huge benefits to having a main centralized repo. If you need redundancy for some reason, it's fairly trivial to mirror the repository somewhere else.
- Yetanfou 9y agoPush and pull to your own server running Gitea/Gogs or Gitlab, and mirror to Github if you feel the urge? Or, if you insist on having your workflow around Github, do it the other way around: have your local server mirror your Github repositories. That way, if Github goes belly-up you can just switch to your local server for the time being.
- __jal 9y agoIf your workflow depends on centralization, run your own centralized service. If you can't live without the GUI, Github will even sell you Github Enterprise. Or just run Gitolite or one of the other tools.
- dordoka 9y agoYou can set up another git server and webhook pull from the main one whenever someone pushes there, so you have a backup replica. Coders just need to setup two remotes for the repos and use the second one when main one is unavailable.
- 9y ago
- adventist 9y agoUGHHUGHGUH!
- workerIbe 9y agoOh man was just browsing a 3 month old branch for a feature that was just revived today, did not clone... Please come back!
- dewiz 9y agoanybody knows if www.atom.io would be affected by this?
- pirocks 9y agoIt seems to have been fixed now.
- Retr0spectrum 9y ago> 00:41 BST Everything operating normally.
- deleted 9y ago[deleted]
- unpwn 9y agoAnd its back!
- deleted 9y ago[deleted]
- unpwn 9y agoAnd its back!
- hetman 9y agoEveryone is saying: just work on your local repo. But GitHub is way more than just git. There's bug tracking, code review, continuous integration, etc etc. Making your organisation too dependent on a remote service can indeed be a scary prospect and I'm not sure what GitHub offers to mitigate this.
- tuxxy 9y agoThey offer a local instance of GitHub actually.
- madeofpalk 9y agoWhich I can't imagine to be any more reliable than hosted GitHub. But hey, I guess there's the comfort in know that when it goes down it's our fault!
- gtirloni 9y agoReliability can always be improved... It just costs more.
- Klathmon 9y agoThe argument is with a self-hosted solution you can choose your "danger times". If you know you have a major deliverable coming up, you can choose not to touch your git server until it's at a somewhat safe moment to do so.
- viraptor 9y agoWhile it definitely sounds like a good idea, every single time I've seen it applied, the reason was the opposite. It was essentially "we've got so little control over our stuff that we'd rather not touch". Mostly goes along with lack of backups, lack of change management, and lack of automated rebuild procedure. If you have "danger times", then any unplanned disruption in that time will hurt so much more.
- sebringj 9y agoBlip, its back up. Nah, too much effort to host my own and I do need a coffee break from time to time. Ignore this if you are on a big team of course.
- badpizza 9y agoGone now! 18:41 CDT Everything operating normally.
- myrandomcomment 9y agoI would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We purchased systems from Sun because of the quality of the HW and the stability of Solaris. Now a stack of cheap linux boxes with the right design (IP CLOS, BGP fabric, load balancing and containers) who cares if a switch fails or a rack goes out? Most issues today are caused my humans and bugs from software upgrades for new code.
- brianwawok 9y agoKind of? How many servers did the average app depend on in 1998? 1? Get dual HD and you were in decent shape. Compare to a modern microservice app, that maybe depends on 100 internal services and 4-5 external services. A lot of things need to go right or mostly right for things to function.
- ProAm 9y agoSounds like an argument against micro services?
- jameskegel 9y agoI'd give a dollar to hear how you arrived at this conclusion.
- stouset 9y agoOnly if you don't stop to consider the alternative.
- ProAm 9y ago"The Wheel of Time turns, and Ages come and pass, leaving memories that become legend. Legend fades to myth, and even myth is long forgotten when the Age that gave it birth comes again."
- Animats 9y agoThis is different from yesterday's GitHub Major Service Outage.[1] [1] https://status.github.com/messages https://status.github.com/messages
- ram_rar 9y agoSuch incidents remind me of 2 things 1. No matter what 99.99999999% availability a service provides, its utterly useless if the time to get back up is unacceptable. 2. Do not have remote services, which you cannot fully control to be a part of your run time deployments.
- acchow 9y agoUptime % includes time to recovery. Since you're not up if you're still recovering...
- hkmurakami 9y agoYes. And recovery time should be tracked separately from uptime is the takeaway imo.
- ram_rar 9y agoBut that still does not provide much inside into recovery time. Its good to know, the median recovery time.
- solatic 9y agoThe only time that uptime would not give insight* into recovery time is if the service is not being regularly updated (remember, zero-downtime deployments should never affect your uptime numbers), then, the uptime numbers improve over time to something really impressive but go in the toilet once the shit hits the fan. I hardly think it's reasonable to consider services like GitHub or infrastructure like AWS as being irregularly updated...
- OJFord 9y agoI think GP's point is that you should assume the 0.{however many 0s}1% it's down is going to be a bad time for you, and then ask yourself how long you want to wait to be back up. A long down time might be worse for you than a shorter down time more frequently.
- 9y ago
- deleted 9y ago[deleted]
- davxhackxxx22 9y agokingzhackx@gmail.com...To change your school grade, Examination questions before Exams, Hackinto any Database, Get lost password for Facebook; Twitter, any email hacks; Whatsapp and other social media, Spy on your Partner to test their
- davxhackxxx22 9y agokingzhackx@gmail.com...To change your school grade, Examination questions before Exams, Hackinto any Database, Get lost password for Facebook; Twitter, any email hacks; Whatsapp and other social media, Spy on your Partner to test their
- davxhackxxx22 9y agokingzhackx@gmail.com...To change your school grade, Examination questions before Exams, Hackinto any Database, Get lost password for Facebook; Twitter, any email hacks; Whatsapp and other social media, Spy on your Partner to test their
- blkrokn 9y agoDid they misgender the server again?
- lessclue 9y agoIf you want to host your own repos with user and access management, bugs, issue, task tracking and tons more, take a look at Facebook's Phabricator https://www.phacility.com https://www.phacility.com (the self-hosted version, of course).
- 1zael 9y agoRun your own git server. Version-control locally, build automated systems to push triggered by uptime status.
- zebraflask 9y ago95.8%. What would AWS think about that?