7 ms·
"Major service outage"
- templaedhel 14y agoGithub is great, they're working on some serious problems, and every service has downtime. But this is getting pretty serious. This is the 3rd time in these last few weeks there has been a "significant service outage" So much of a typical dev workflow is based around github, but more importantly a lot of new package managers use github as a base. Not being able to pull dependencies is a fairly big problem. I will continue to use github, because they're awesome. At the same time, we're going to have to start building around it to ensure our uptime isn't reliant on their (less than optimal) uptime.
- anon1385 14y agohttps://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
- badgar 14y agoMany other companies are able to implement web services without the reliability issues GitHub has experienced. As your parent poster explained, some unreliability is always tolerable, but GitHub is starting to cross that threshold. Your list of fallacies does nothing to address that very real complaint.
- PommeDeTerre 14y agoI've found it somewhat odd how, although advocating the use of a decentralized version control system, much of the git community has ended up heavily centralized on GitHub.
- jeremyjh 14y agoIt is a distributed system. You still have to have shared remotes in order to integrate branches.
- snogglethorpe 14y agoYup, when github barfs, you can still happily keep on hacking. You're also not limited to one shared repo by any means, so even synchronization between multiple developers can continue when github is down. To avoid problems it's of course a good idea when one of the repos is a "master", and if using a secondary repo for synchronization when the master is down, you'll probably want to switch to unique temporary branches for the purpose.
- jaequery 14y agowell, lot of open-source projects depend on github as source of their code distribution. github is like the new sourceforge these days.
- yakiv 14y agoOne possible backup would be Bitbucket (https://bitbucket.orghttps://bitbucket.org/ https://bitbucket.orghttps://bitbucket.org/). (I'm not affiliated Bitbucket in any way. I don't profit from them doing well, aside from them sticking around to keep providing free hosting for me.)
- ajross 14y agoAnother possible backup is any host on the internet. Seriously, dump it into a static file directory on any old web server or run a trivial git server. Send deltas with patch files. It's not even harder than github is, just uglier.
- bitcartel 14y agoIf you want to use your own servers, Atlassian (who own Bitbucket) recently released Stash, a competitor to Github Enterprise. Pricing is very good and if you have a commercial license, you also get the source code.
- lflux 14y agoI've evaluated Stash a couple times, and I feel that Gitlab still has it beat.
- jaequery 14y agoperhaps there maybe a way to somehow sync-up github w/ bitbucket to act as a single master origin. so we can use either or and they'll be synchronized automatically. or some company that can just act as a mirror to github would be great.
- rsobers 14y agoSo setup 1 small git server that mirrors your most critical repositories which devs can failover to in the event GitHub is down _exactly_ when you need to integrate changes. I wouldn't call this "building around" GitHub. Git's nature promotes this kind of thing.
- michaelsbradley 14y agoIn a similar vein, one could failover to mirrors of critical repositories hosted on another GitHub-like service, e.g. Bitbucket: https://bitbucket.org/ https://bitbucket.org/
- daenz 14y agoThe beauty of decentralized version control is a node going down doesn't kill your workflow. In a pinch, you could use git bundle to exchange commits with other developers: http://git-scm.com/2010/03/10/bundles.html http://git-scm.com/2010/03/10/bundles.html
- bascule 14y agoThis is great until you depend on anything you actually have to fetch from Github. Bundling project dependencies from git comes to mind
- pbiggar 14y agoUnless you need to run a bundle install (or npm install, or whatever) and have github urls in your Gemfile.
- robryan 14y agoDoes Bundler allow for multiple gem sources to be specified (on a gem by gem basis)?
- pbiggar 14y agoI'm not 100% positive on the specifics, but I've never seen it used. I suspect its not possible. (context: I make https://circleci.com https://circleci.com - a continuous integration company for web apps, often Rails. We occasionally get support requests that allow/ask us to look at Gemfiles, so I've seen an above average number of Gemfiles. However, I more often see the stdout of the `bundle install` command, which shows GitHub being accessed).
- skateinmars 14y agoYes, you can specify a git repo in your Gemfile (and optional tags/branches) : gem "rails", :github => "rails/rails", :tag => 'v3.2.11' You can also specify multiple gem sources (http://gembundler.com/v1.2/gemfile.html http://gembundler.com/v1.2/gemfile.html), but usually only rubygems.org is used unless you need a private geù repository.
- darkchasma 14y agoThis is what I think may be their biggest problem. Because of the outages, and due to my own stupidity, I have been caught with my pants down twice because Github was always just up. And then I thought, well, lesson learned, they won't go down again, and ker-pow (I'm stupid like that). So I have had to create a secondary remote, and for private projects, it's really easy to do so. Which is kinda problematic, because those are the accounts that keep the lights on for Github. If I have to cut budget, it's a lot more likely now that I would consider cutting the private repos. I just hope they figure this out soon, so I can forget about how easy it was.
- deleted 14y ago[deleted]
- s1kx 14y agoI suppose this is related to the Rails vulnerability (http://news.ycombinator.com/item?id=5028218 http://news.ycombinator.com/item?id=5028218) and everyone updating/deploying their applications
- yogipatel 14y agoUpdate from GitHub: 23:49 UTC Everything operating normally
- cagenut 14y agoIf they keep this up I'm gonna have to buy github-enterprise! wait a minute...
- jonursenbach 14y agoGithub Enterprise runs on your own servers. Not theirs.
- X-Istence 14y agoThat's the joke ...
- jonursenbach 14y agoThere was a joke in that?
- gbog 14y agoWould it be possible to build a github mirror with only the repos and nothing of the web candies? Then maybe the reliability would be increased.
- Doih 14y agoWe use rhodecode to mirror our hg and git stuff it auto pulls stuff also.
- gjvc 14y agoi've downgraded to free
- DigitalSea 14y agoI get outages are a way of life with a widely used web application, but Github have really dropped the ball lately. This is one of many service outages lately and as a paying customer it's disheartening and worrying because I use Github in my day-to-day workflow, I and many others have come to rely on it. Don't get me wrong here, I love Github and couldn't live without it, but they really need to sort out these problems and it's not like they don't have the funds to address the issues anyway. My knowledge of distributed computing is somewhat limited, but I would have thought they'd just spin up a few extra virtual machines to handle the database spike (maybe it's not that simple with Github's setup, I'm not sure).
- 727374 14y agoDatabases typically don't scale horizontally with so much ease. Scaling high "write" data stores is not a trivial problem. But I feel your pain.
- hunvreus 14y agoGithub have their own datacenter and hardware, they are not relying on any cloud provider out there. It makes it harder to handle spikes of load, but they actually have an I/O intense service which justifies that choice.
- druiid 14y agoIncorrect: https://github.com/blog/493-github-is-moving-to-rackspace https://github.com/blog/493-github-is-moving-to-rackspace
- DigitalSea 14y agoMy favourite line from that linked blog post from 2009 is this: "We're aware of the current stability and performance issues, and we want to let you know what we're doing about it." - issues they were having nearly 4 years ago are still happening unless the problems they've faced lately are completely different.
- Nux 14y agoJust run your own shit, people.
- mikec3k 14y agoI use BitBucket for most of my projects since they offer free private repository hosting.
- willvarfar 14y agoGood to see that they've upgraded/patched their Ruby On Rails attack surface then... :)
- cmwelsh 14y agoMirror your projects to BitBucket to reduce SPOF. That's what I'm doing with my private repositories.
- rauar 14y agoIs this an automated status page ? The messages are so ... focused. If yes, I love it. Need to build one too.
- Doih 14y agoJust use rhodecode as a backup and don't worry anymore
- outside1234 14y agoCorrelated with thousands of redeploys for the Rails security issue?
- emperorcezar 14y agoLooks like it was a whole 10 minutes. EVERYONE GRAB THE PITCHFORKS!
- rtkwe 14y ago2:13 if you count from the first message about high exceptions to the all clear "everything is working normally."
- darkarmani 14y agoThis is why I should deploy using git (via github), right? I can always rollback if i break something...