30 ms·
GitHub was down
- xxkylexx 9y agoLooks like I am still able to push to/pull from my repos without issue.
- DylanBohlender 9y agoI've had a few CI jobs fail when attempting to pull, but I've also had some succeed. Seems that it isn't an all-or-nothing outage.
- vishesh92 9y agoIt just became a major service outage.
- adtac 9y ago>GitHub is having a minor service outage It's definitely not minor.
- deleted 9y ago[deleted]
- slap_shot 9y agoStatus now shows Major Service Outage: 12:32 EDTMajor service outage. https://status.github.com/ https://status.github.com/
- indubitably 9y agounicorns all around
- tevonsb 9y agoThoughts on the cause?
- cjbprime 9y agoDefinitely Bitcoin-related.
- midnitewarrior 9y agoSegWit's first victim.
- orf 9y agoNot enough mongodb. clearly they are not webscale yet.
- rubicon33 9y agohttps://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
- Ace17 9y agoLudum Dare? :-)
- btym 9y agoBug in their Ethereum-backed MongoDB instance.
- tabtab 9y agowhile (! github.works) { this.add(new Buzzword()); }
- komali2 9y agoDefcon let out on Sunday and there's a lot of bored hackers with leftover energy.
- mirekrusin 9y agoAI started backing itself up on github.
- leesalminen 9y agoI think it started as minor as I was receiving a unicorn once per 10 pages. It's currently happening on almost all. Of course, I'm trying to dig into a WebKit issue and need the issues to load!
- luhn 9y agoPages Builds Failure Rate spiked to over 2000%. I don't know how that's possible, but it seems pretty bad.
- frandroid 9y agoBad Metric is Bad.
- westbywest 9y agoGuessing absolute failure rate previously not encountered by current graphing tool, incorrect scaling factor. Or, it was decided at some point that always reporting 0.000XXX% failure rate, even if correct, didn't offer an intuitive metric, so zeroes were intentionally truncated.
- adrianpike 9y agoMaybe 20+ failing retries for any single page build?
- tedmiston 9y agoIt'd be nice to have an intuitive explanation on the status page for what PBFR means if it can go over 100%.
- Cafey 9y agoIt has leveled up to a major outage!
- ProAm 9y agoGithubs uptime is pretty bad. Isn't it under 95% for the year now?
- smudgymcscmudge 9y agono
- chippy 9y ago95% would mean it would have over 18 days offline! I wouldn't even put it in the 99% (3.7 days) uptime category either.
- kkirsche 9y agoThat would be 18 days out of a year. I'd be amazed if it was that low. I'd expect closer to 98-99% (7-3 days per year)
- dewiz 9y agohttps://status.github.com/graphs/past_month https://status.github.com/graphs/past_month
- valarauca1 9y agoNot really. It is just very public when they go out. Steam, Blizzard, Activation are down at least 6 hours a week. While Bank of America is offline 24hours/month. Scheduled downtime is still downtime.
- FireBeyond 9y ago> Scheduled downtime is still downtime. Except in the SLA. I've argued more than once when watching a vendor announce "emergency scheduled maintenance"(!) as an outage is looming.
- Cafey 9y agoThe stat you refer to is for the day only. Status page shows 99.5159% for the past month, I'd guess it would be in that range for the year aswell.
- 0xbear 9y agoHow many more can we expect before they develop appreciation for testing _before_ they push to prod?
- toyg 9y agoPushing changes on Monday morning... way to ruin everyone's week!
- cjbprime 9y agoNewly introduced errors in site code are only one of many sources of failure for a site like GitHub, and probably a fairly rare one.
- 0xbear 9y agoIt's a nearly weekly occurrence. Avoidable one, I might add. When was the last time a major Google service had a major outage?
- cjbprime 9y agoYou're still missing the point, by assuming that the outages are code related when you call them avoidable. Distributed systems at scale are terrifyingly hard to control. When I said "probably a fairly rare one", I didn't mean that the outages are rare. I meant that new code is probably a rare cause of the outages that happen. They have other causes unrelated to new code. (I'm also skeptical that GitHub major outages happen "nearly weekly", but I don't have data.)
- 0xbear 9y agoI'm not "missing" anything. I worked at Google for 7 years much of which was spent working on, you guessed it, distributed systems infrastructure. You guard against this by carefully canarying things and putting robust testing, monitoring, and deployment procedures in place. A release might take a few days, but you can be reasonably certain your users won't be your guinea pigs, and if shit does hit the fan, rollback is easy, and you can reroute traffic elsewhere while you roll back. Most of the time no rollback is needed: you just flip a flag and do a rolling restart on the job in Borg. For some types of outages (most of which users never even see) Google has bots that calculate monetary loss. And the figures can be quite staggering and motivating, so people do postmortems and try their best to make sure the outages don't happen again.
- deleted 9y ago[deleted]
- peterwwillis 9y agoDang. It's too bad their customers' source control files aren't distributed and decentralized, or they could keep working and ignore this.
- toyg 9y agoThe problem is not the files, it's the project management stuff (issues and PR tracking). That is a SPOF. If they could decentralize that, then nobody would care if they are down half the time.
- mschuster91 9y agoI wonder why you can't export them as a read-only git repository. (Read-only to avoid people messing around with the history)
- csense 9y agoBecause that would make it easier to migrate away from Github, which Github doesn't want you to do.
- daxelrod 9y agoYou can actually export them, but the format is undocumented, and is meant only for import into GitHub Enterprise: https://developer.github.com/v3/migration/migrations/ https://developer.github.com/v3/migration/migrations/
- peterwwillis 9y agoThey can - it's called Github Enterprise (or if you're a medium to large corporation, JIRA). I thought occasional downtime was just something people factored into free services. You can always file your issues later, or use a chat service (or, heaven forbid, email) if there's an immediate need for feedback.
- ibgib 9y agoAre there any other major sites that are down?
- thepumpkin1979 9y agoAll software depending on Github repo releases downloads are down, e.g: Rancher CLI.
- smudgymcscmudge 9y agoRelease downloads are affected?
- thepumpkin1979 9y agoyep, my Circle-CI deploy fails, the following URL is not available: https://github.com/rancher/cli/releases/download/v0.6.0/rancher-linux-amd64-v0.6.0.tar.gz https://github.com/rancher/cli/releases/download/v0.6.0/ranc...
- kmfrk 9y agoI had to change a username from capitalized to uncapitalized and use my updated remote afterwards, apologies if I broke it for everyone.
- cjbprime 9y agoIt's kmfrk's fault! Get them!
- drfuchs 9y ago"Git 'em!" FTFY.
- dguaraglia 9y agoDid you know that if you type "Google" in the Google search box you can break the internet? :| (BTW, this was meant as a joke... although I'm not discarding that kmfrk broke Github just yet!)
- mdaniel 9y agoTo the best of my knowledge, GitHub org and usernames in the URI are case-insensitive for both the website and clone URIs. I haven't tested ssh clone URIs to know if they are also insensitive, but I'd guess they are You would only need to change it if the "presentation" format bothers you (again, as far as I know)
- kaikai 9y agoI would love to see a chart of traffic to other sites when GitHub goes down. My bet is that HackerNews and Twitter both get significant spikes from all those bored developers.
- emilsedgh 9y agoIts not like your editor is down.
- mdellabitta 9y agoMy team was working on merging pull requests when this happened. We might actually be at fault! That would be kinda cool.
- revmoo 9y agoWell you all know to commit your vendor libs so you can still deploy when Github is down, right? RIGHT?!
- Osiris 9y agoIt's starting to work again for me. I was able to approve a PR and merge it.
- __s 9y agoIn the face of a lack of information, HN comments begin to throw around unfounded speculation & tongue-in-cheek jokes run rampant. I suppose that in the absence of information, many stay silent, & the remaining see a thread lacking comments & now we've got this meta one in the mix
- sashk 9y agoWhat was happening to Github for a week or so in late June - early July? I see "The status is still red at the beginning of the day" for a whole week. https://status.github.com/messages/2017-07-03 https://status.github.com/messages/2017-07-03
- nowarninglabel 9y agoThey seemed to be suffering some external attacks / DDoS but I never saw a post-mortem from them on it, hopefully one is forthcoming (or maybe is out but I missed it)
- richardwhiuk 9y agoThey don't do post mortems.
- chrisferry 9y agoMaybe not public ones, but I'm pretty sure they perform internal post mortems when they have outages.
- daxelrod 9y agoThey occasionally do, and they call them Incident Reports. https://github.com/blog/2106-january-28th-incident-report https://github.com/blog/2106-january-28th-incident-report https://github.com/blog/2273-incident-report-inadvertent-private-repository-disclosure https://github.com/blog/2273-incident-report-inadvertent-pri...
- moomin 9y agoI knew I shouldn't have released the new version of my project yesterday. :p Sorry everyone
- GrumpyNl 9y agoHow does this affects all your dependencies?
- kentor 9y agoWhy would it?
- s73ver 9y agoA lot of package manager tools download their packages from Github. If you happened to be refreshing your dependencies at the time, or doing a clean build, then you'd be SOL.
- GrumpyNl 9y agoThanks for the explanation.
- moosingin3space 9y agoReally? I was under the impression that most language package managers downloaded from a CDN. I know that pip, npm, yarn, cargo, and hex do, at least.
- s73ver 9y agoHomebrew I believe goes through GitHub. Many of the Vim plugin managers also do, or at least have the option. I think CocoaPods and Carthage, both for iOS development, do as well. I think it's somewhat common for a new package manager to use Github as a kind of CDN for a while, until they get big enough they can do their own.
- adam_seeliger 9y agoEveryone calm down. GitHub is working just fine.
- rcthompson 9y agoLooking at the status graphs, it seems like there was some clearly anomalous data starting around midnight, about 9 hours before the actual outage "began". Maybe a gradual botnet ramp-up, and 9:27 AM is when it got bad enough to overload some critical service? (Or really any other threshold-based failure scenario.)
- aksx 9y agoor a bad commit being deployed on their fleet.
- tambourine_man 9y agoInsert remark on why we use a centralized service for a distributed source control system, etc. No one seems to care, unfortunately
- twblalock 9y agoLots of people care, but we also recognize that the advantages of using a centralized system outweigh the disadvantages for many use cases.
- Goladus 9y agoThe reason is github did a fantastic job of implementing useful features. The visual design is unmatched and they have done a great job implementing developer oriented integrations and social features. A more federated approach to this sort of thing might have been nice, but so far nothing I have seen comes close to the value-add offered by github.
- xutopia 9y agoYou're sidestepping the main reason I believe it worked so well. It benefits from network effect. It is a collaborative tool and people like to have their work on there so others can collaborate with them.
- Goladus 9y agoThat's important but I think people over-estimate it. It's both. I predict that if you analyzed the github network, you'd find many hubs are based around companies that chose to move their workflows to github based on features other than network effects. Or at least, the existing network was only one of many reasons.
- manigandham 9y agoAs a business, we (and most other companies I know) chose github for features and performance. Nice that other open source stuff is there but doesnt matter for what we pay for.
- macawfish 9y agoWhatever happened to gittorrent?
- nicodjimenez 9y agoPretty shocking how accustomed we've all become to Github outages. Mission critical developer tools should have gratuitously high uptime. Github is a 100M company whose most important operational objective is just keeping the site running. I call shenanigans.
- yellowapple 9y agoMy apologies. I knew my Perl 6 wrapper for GLFW was bad, but never realized it'd be so bad that GitHub would choke to death on it.
- loomer 9y agoAnyone have any knowledge of what specifically happened?
- pramodzion 9y agoGithub is back online.
- alexchamberlain 9y agoWhen I break our GitHub webhooks, I joke it's time for people to practice our Disaster Recovery (DR) procedures. In all seriousness, this is a good opportunity to practice work without GitHub. Any service can go down; can you deploy a critical bug fix without it? If not, why not and what can you do to fix it?
- deleted 9y ago[deleted]
- pmoriarty 9y agoWhere is github hosted? Do they use AWS or another commercial cloud provider, or do they have their own servers in data centers (hopefully scattered around the globe)? If AWS, are their services spread among multiple availability groups? I'm just wondering how this could happen.
- moondev 9y agoI believe they manage their own hardware in a datacenter
- cormacrelf 9y ago... It may surprise some HN readers, but AWS outages aren't the only reason other sites go down. Having multi-AZ just means you're resilient to a localised, single-AZ AWS failure. It doesn't help for the 1000 other ways your service could go down.
- pm90 9y agoThey're hosted at Rackspace. I'm not sure if they're hosted on the Public Cloud or whether they have a dedicated hardware or what though.
- worldexplorer 9y agoNo they shifted from Rackspace. https://github.com/holman/ama/issues/553 https://github.com/holman/ama/issues/553
- sattoshi 9y agoI like how this comment has 3 replies saying different things. I am now feeling less informed than I was after the first reply.
- worldexplorer 9y agoAn ex-github employee who used to give talks on Github scaling challenges revealed they shifted from Rackspace. See yourself - https://github.com/holman/ama/issues/553 https://github.com/holman/ama/issues/553
- DevKoala 9y agoThis is happening too frequently now.
- relaxitup 9y agoDo these general Github outages affect GH Pages as well, or is that service portion segmented to some degree?
- bluehatbrit 9y agoPages are static sites served from separate infra from the main github.com from what I've heard, so you should be fine in almost all cases. I've never had one go down unless I pushed a dud build.
- _Marak_ 9y agoIf anyone is interested, I've been working with a git host that is actually distributed across a p2p network using SSB. see: https://github.com/clehner/git-ssb https://github.com/clehner/git-ssb https://github.com/noffle/git-ssb-intro https://github.com/noffle/git-ssb-intro It's been working fairly well so far. We are using git-ssb to manage a few projects instead of putting them into Github.
- swsieber 9y agoHey, that's really cool. Have you considered submitting it for a Show HN?
- vertex-four 9y agoIs there, at least theoretically, a way to prevent other people from pushing to my repo? That seems like it would suck re griefing for any project that might become even mildly politically sensitive for whatever reason.
- _Marak_ 9y agoEverything is key based, so only key holders push to your repo. It's all based on the SSB protocol. I'd suggest reading https://github.com/noffle/git-ssb-intro https://github.com/noffle/git-ssb-intro to get an idea.
- vertex-four 9y agoThat states: "git-ssb's permissionless model has an interesting consequence: anybody can push to anybody else's git repository." The guide doesn't show any key-sharing in order to do that. Are you saying that's incorrect?
- _Marak_ 9y agoI don't want to give you the wrong answer so I've forward this question to the SSBC network for one of the core developers to answer better. I'd be surprised if there was no security for pushes. The repos I've worked on did require an invite from the creator.
- AlphaWeaver 9y agoI saw a comment earlier mentioning that GitHub allegedly doesn't release post mortems publicly? If this is true, that's upsetting.
- deleted 9y ago[deleted]
- luhn 9y agoThey have published post-mortems before, but the previous bout of downtime went unacknowledged. Maybe it's a new policy. Here's a postmortem they did several years ago: https://github.com/blog/1261-github-availability-this-week https://github.com/blog/1261-github-availability-this-week
- Cacti 9y agoI don't see why you would expect them to. It's nice and all, but they're under no obligation to tell the world everything that happened every time they have a minor outage. Not every outage is newsworthy or even that interesting, and oftentimes they're little more than a PR piece for the company.