8 ms·
Github experiencing major service outages across all services.
- deleted 14y ago[deleted]
- experiment0 14y agoA little bit more info here: https://status.github.com/messages https://status.github.com/messages
- lacosaes0 14y agoAnd this is why I use Google Code: supports git and is more reliable than Github. It's a shame that they don't offer a paid service for closed-source software.
- rplnt 14y agoWere there any issues with Github prior to this? I don't recall any aws-like pattern. But I might have just missed it.
- cdelsolar 14y agoGithub is awesome, but yeah, they've been having lots of outages for a while now. A deploy based on a git fetch && git reset from Github might not be a good thing.
- kordless 14y agoA few. They lost a server about 10 days ago which took down a few repos including some gists. All in its been very stable.
- tzaman 14y agoWhere do you live, under a rock? Of course they offer paid service for closed-source software.
- stock_toaster 14y agoIndeed. Not only do they offer a paid service and private repos, but they offer a paid self-host version as well.
- cmelbye 14y agoI'm assuming by "they" he was referring to Google Code. (Wishing that Google Code offered a paid service.)
- jtchang 14y agoThis is why I love git (and distributed version control systems in general). For the most part a short downtime isn't the end of the world. When it comes back up I'll push my changes and that'll be that.
- shoopy 14y agoI can do the same thing with CVS?
- callahad 14y agoExactly. I'm sitting here thinking "poor GitHub engineers" rather than "oh crap, how am I going to work?" (Though I do wish GitHub Issues weren't as centralized...)
- kmfrk 14y agoYup. I wanted to do some coding before heading off, but I guess that's off the table now.
- deoxxa 14y agoWhat? That's the complete opposite of what OP was saying. You don't need github to be online to work with your code or even your repositories so long as you haven't gone and thrown all your digital eggs in the same basket (i.e. as long as you have other remotes or at least local copies of your repositories).
- vacipr 14y agoGood.I was starting to miss "Github is down" submissions. Quick let's look for alternatives.
- cagenut 14y agoThat's a really nice status dashboard, tracking and publicly displaying your 98th% is really cool. On the other hand stuff like this: "13:17 UTC We are seeing unicorns ..." Comes off as un-professional at exactly the wrong moment.
- tinco 14y agoHow is that un-professional? Unicorns mean errors for github, and they report them. This is more professional reporting than most companies do.
- recursive 14y ago> Unicorns mean errors for github I have no idea what this means, but I can see how the idea of coming up with an alternate language in which fantasy creatures represent things we already have established words for might seem unprofessional
- stock_toaster 14y agoQuite literally, the unhandled error page on github shows a unicorn. As seen on this google image search result[1]. It appears they have put up an octocat "we are down" temporary static page right now though... [1]: http://www.flickr.com/photos/beatak/4008687328/ http://www.flickr.com/photos/beatak/4008687328/
- 1qaz2wsx3edc 14y agoAdditionally they could be referring to their use of `Unicorn: Rack HTTP server for fast clients and Unix` http://webcache.googleusercontent.com/search?um=1&hl=en&safe=off&authuser=0&biw=1920&bih=982&q=cache%3Ahttp%3A%2F%2Fgithub.com%2Fblog%2F517-unicorn&ie=UTF-8&sa=N&tab=iw&ei=KlbWUNrGAoSp2gWKooCIBg http://webcache.googleusercontent.com/search?um=1&hl=en&...
- tinco 14y agoThe term unicorn actually comes from the fact that they run the 'unicorn' webserver. When a webapplication error occurs unicorn catches it and displays an error to the visitor. They prettied that error up with a pretty image of a unicorn and a small explanation that something went wrong. So when they say 'we saw unicorns' they literally saw (reports of people seeing) pictures of unicorns. Now I understand that might confuse someone who does not regularily use github, but it is just a term that is in their jargon and I see no reason why they should market-speak that up in an intermediate status message. (Note that these status messages are not press releases but reports that engineers make during the discovery process)
- tzaman 14y agoI guess Murphy is strong on this one, I was just right in the middle of bringing up new server (to host Rails app). Oh well, I'll go to bed early :D
- rolleiflex 14y agoWell handled, minus the unicorns. Tangentially relevant: I just wish Github offered some sort of an academic plan for students, no private repo means that I cannot use Github at all not because I'm building closed-source software, but because I (obviously) can't put my assignment work for public viewing before the assignment deadline. So I've been using Bitbucket, which is fine and all, but I would have loved to be a part of Github community.
- ricardobeat 14y agoFive dollars... You can also commit locally, and only push after the deadline.
- experiment0 14y agoThey do offer a free plan for students. I get 5 free private repos. I can't get the link now since GitHub is down ha, but when it's back up, just have a quick google and you'll find a quick form to fill in and voila. It's great.
- drivebyacct2 14y agoIt's actually a discount. That way you can have a better account for $5 less each month. Although I should probably go turn that off having just graduated.
- experiment0 14y agoWhen I signed up around a year ago it wasn't a discount. It was a free micro account that lasts two years. I'm not sure if that's changed however.
- drivebyacct2 14y agoRight. It is. But you can upgrade your account and it acts like a $5 discount (the micro plan is $5...)
- 14y ago
- zemanel 14y agoWhat about a script/service to mirror Bitbucket and Github (or others) through webhooks or etc? Was just getting my hands on Homebrew after a fresh OS install when i hit the Octobummer :-/
- ricardobeat 14y agoYou can push to both at the same time, no need for hooks.
- zalew 14y ago.git/config [remote "bitbucket"] url = git@bitbucket.org:you/repo.git fetch = +refs/heads/*:refs/remotes/bitbucket/* $ git push bitbucket //edit after tinco's comment +refs/heads/:refs/remotes/origin/ -> +refs/heads/:refs/remotes/bitbucket/
- zemanel 14y agoAh but i meant a global service for public repos, instead of each owner setting up manually
- 1SaltwaterC 14y agoA remote can have multiple url entries. 'git push origin master' in my case pushes to Bitbucket, then GitHub. Must be manually added through 'git config -e' though. The order is important as it pulls from the first remote URL, while the push is to all.
- rubynerd 14y agoI'm amazed at how much this has knocked me on my arse. I first attempted to redo the README for a service I've just open-sourced, before realising Github is down. Then, I attempted to fix the company CI server (OOM errors because of Carrierwave needing more than 500MB of memory to run 1 spec in, for some unknown reason), which failed because it couldn't check out the code. After giving up on that, I attempted to install Graphite to a company server, where I hit another roadblock because the downloads are hosted on Github, and so I had to use Launchpad, which I had an allergic reaction to. Also, when I was shelling into the server, oh-my-zsh failed to update because, you guessed it, Github was down. Still, shouts to the ops team in the trenches, we're rooting for you.
- tinco 14y agoYour company runs its source revisions through Github without a backup solution? Do you really put all your eggs in a basket you have no control over? I know that in theory a cloud solution should have a higher uptime than an amateuristic set up private server, but cloud solutions have a certain complexity and coherence that make them very vulnerable to these kinds of 'full failures' where nothing is in your control. Maybe you should take this time to learn from this, and analyze what you could do to reduce the impact of this failure. For example, you could research what it would take for your company to move to another Git provider, perhaps even on your own server or a VM slice at some cloud provider. I'm not saying you should drop github, because obviously they have great service, but be realistic about cloud service. Cloud service is like RAID: it is not a backup. The way RAID is nice for recovering from errors without downtime, there is a chance something bigger happens and you still lose your data cloud is nice for offering scalability and availability but there's a chance everything goes down and you still can't run your operations.
- ghshephard 14y agoAdd up your downtime over a three year period by relying on Git (or gmail, or AWS) versus the cost of trying to engineer some local-backup system, and the downtime associated with that going awry. Outages happens - as long as we're talking hours a year, pretty much everyone but life-safety systems, critical infrastructure, and payment/high-traffic commerce sites are probably better off just letting third-party cloud vendors manage their systems. Take the downtime and relax. (Now, if downtime consistently gets into the 10s of hours/year, it's time to look for a new cloud provider. )
- darkchasma 14y agoI never forget to pull before going remote, and today I did, and the one time I actually need github to be up and it's not. But I can't be upset, even if this wasn't my fault, it's github.
- revskill 14y agoMost of developers in my countries 're using SVN. I don't use SVN at all, and i'm failing to convince my partner that Git is better than SVN, just because i don't believe in hardware stability.
- niggler 14y agoWhat's funny is the extent to which Github is being used as a centralized repository for many projects. I don't just mean for project discovery; the issues and gists and other services aren't replicated as often or easily as the code. In fact, a lot of services depend on github for various reasons, all of which are probably borked now ...
- robomartin 14y agoI did a bunch of work on GitHub today just before it went down. Talk about getting lucky. I'm sure we will all learn a bunch from the post-mortem. These high-profile and very openly discussed failures are always good for learning all kinds of things. No issues at all on how GitHub is handling it so far. Eager to learn what happened. Hitting refresh on the status page every so often. Better than watching underwater basket-weaving competition at the olympics.
- syncsynchalt 14y agoI created a new org on GitHub today (through all the unicorns) and was about to push an existing repo to them so we could all start pulling it. Talk about unlucky.
- kh_hk 14y agoThe good part is you can do it yourself without GitHub (which, after the downtime, could come into the equation quite easily): -- Remote machine, or SSH port forwarded machine -- > adduser git (you either add pusher/puller's ssh pubkeys to its authorized_hosts, or use a shared password) > su git > cd ~ > git init --bare myproject.git -- Your local repo -- > git remote add <name> git@aforementionedmachine:myproject.git > git push <name> Setting up a git repo for confidential pushing and pulling is quite easy.
- syncsynchalt 14y agoAlready had the repo up and running elsewhere, thanks. Just needed to do some non-git stuff on github (create a new repo on GH to push it to, add team members, and so on). I've been doing private DVCS for years (mercurial) but this is my first project that's on git and I've been looking forward to the opportunity to host it on github and see what I've been missing.
- rossjudson 14y agoCan't read the clojure 1.5 RC1 release notes. Damn. Looks like services implemented in terms of github are not reliable (but there was no guarantee of that anyway).
- fomojola 14y agoInteresting: Github Pages is still up (got a blog on there and it doesn't appear to have experienced any outages). Static site generators for the win.
- buttscicles 14y agoPages was out briefly a couple of hours ago (certainly some repos, anyway) Seems to be fine now though.
- timothybone 14y agoI feel like such an uber goober! I was installing a package that relied on a github file, which for obvious reasons failed....little did i realize that that was the problem, all I saw was such and such python error callback....doh!
- csense 14y agoIt should be pretty far down their list of priorities at this point, but I just noticed the "Exception Percentage" value at https://status.github.com/graphs/past_day https://status.github.com/graphs/past_day is saying 483.704%. The fact that they're measuring this in percentages implies the maximum is 100%, but this isn't so.
- chuhnk 14y agoI imagine it was a percent measured against successful responses. The exceptions now far exceed those successful responses. You are correct in saying it should be 100% but people have a habit of scaling beyond it.
- csense 14y agoIf this is in fact how they're measuring...I was going to make this post a long rant about how ridiculous it is to measure it in that way, but then I realized that people do sometimes measure probabilities by quoting the win-to-loss or loss-to-win ratio. This is called "odds" and frequently used in gambling. Usually, though, someone says "4.7 to 1" or "47 to 10" (abbreviated 4.7:1 or 47:10) instead of 470%. Usually the larger number is stated first, and the direction is usually indicated by a word like "favorite" or "longshot." So one would say "Errors seem to be a 4.7:1 favorite today." It's slightly complicated by the fact that odds can measure one of several things: A. A probability ratio ("Red" is a slight underdog in the game of roulette [1]; the odds against hitting it are 20:18 since there are 20 non-red spaces and 18 red spaces) B. A payout ratio ("Red" pays 1:1, meaning the prize if you win this bet is equal to the amount of the bet) C. The current payout of a paramutuel pool [2] Odds are seldom used outside of a gambling context. [1] http://en.wikipedia.org/wiki/Roulette http://en.wikipedia.org/wiki/Roulette [2] http://en.wikipedia.org/wiki/Parimutuel_betting http://en.wikipedia.org/wiki/Parimutuel_betting
- cjbprime 14y agoIt's back now.
- pcerioli 14y agoGitHub outage is now over.
- 23david 14y agoTime to fork github. Bad bad sign that they can't get their operations in order. I guess $100 Million can't buy you uptime...
- rdl 14y agoI'm curious what actually went wrong.
- greggman 14y agoI'm really curious how much longer github can offer so much free service. It's not just git but effectively free web hosting as well, at least for statically served pages. It seems like it's only a matter of time before something will have to give. Either they'll have to start throttling web serving or cover the site in ads like sourceforge or something I guess I'd better sign up and start paying ASAP to help be part of the solution
- robin_reala 14y agoPlenty of companies pay GitHub multiple thousands of dollars a year for their services. It’s not just personal accounts.
- jamescun 14y agoI don't think GitHub's hosting of free repositories costs them that much. As you said yourself, it is just essentially free hosting so the largest cost will probably be storage (which is pretty cheap anyway now) and then the much minor cost of Push/Pull/Clone etc bandwidth and processing. Think of it less as a free service and think of it more as marketing. I have a few free repos which I used to open source some of my personal modules; however when it came time to select a source revisioning service for work, I remembered how easy GitHub was and went with their payed service. I am fairly sure us now paying for our repositories offset itself and my and quite a few other free repositories.
- kcbanner 14y agoGithub is used by large enterprises.
- pjungwir 14y agoIt's easy enough to set up a backup repo so a team can keep collaborating when Github is down, but does anyone have a way to deploy Rails apps with Capistrano+bundler? It's terrible not being able to deploy; sometimes that can be really urgent. With Cap I can just repoint config/deploy.rb to the backup git repo, but what about bundler?
- pjungwir 14y agoTo answer my own question: http://stackoverflow.com/questions/8693319/where-can-i-install-gems-from-when-rubygems-org-is-down Of course if you have private patches of gems with bundler's `:git` option, you'd need to repoint all of those, too (as well as keep them in two places). It'd be awesome if there was some simpler way, maybe a separate declaration in your Gemfile so that any gems added to your project also get installed on a corporate git server, and then bundler uses that as a fallback if `source` doesn't work.
- tronicron 14y agoSolution is to self-host your critical files. If you have SSH already on a server (ha!) it is pretty easy: http://www.verbosity.ca/hosting-your-own-git-based-shared-repositories-using-ssh http://www.verbosity.ca/hosting-your-own-git-based-shared-re...