9 ms·
GitHub availability this week
- akoumjian 14y agoI would love to know more about this two pass migration strategy.
- jnewland 14y agoWe use https://github.com/soundcloud/large-hadron-migrator/ https://github.com/soundcloud/large-hadron-migrator/
- druiid 14y agoWell, I have to say... replication related issues like this are why I/we are now using a Galera backed DB cluster. No need to worry about which server is active/passive. You can technically have them all live all the time. In our case we have two live and one failover that only gets accessed by backup scripts and some maintenance tasks. Once we got the kinks worked out it has been performing amazingly! Wonder if GitHub looked into this kind of a setup before selecting the cluster they did.
- aaronblohowiak 14y agoany details on the kinks you worked out?
- druiid 14y agoSure. Maybe I should do a writeup for it on my blog at some point in the near future :). The two main issues we encountered both had to do with search for products/categories on our sites. The first was that Galera/WSREP doesn't support MyISAM replication (It has beta support, but I wouldn't trust it). This meant that we had to transition our fulltext data to something else. The something else in this case was Solr which has been a much better solution anyway (fulltext based search was legacy anyway so this I can kind of count as a win). The second issue and the one that was causing random OOM crashes was partly due to a bug, partly due to the way the developer responsible for the search changes implemented things. The bug part is that galera doesn't specifically differentiate between a normal table and a temp table. When you have very very small/fast temporary tables that are created and truncated before the creation of the table is replicated across the cluster it can leave some of these tables open in memory (memory leak whoo!). We were able to fix for this and have been happy ever since. If there's any interest I can do a larger writeup about actual implementation of the cluster, caveats and the like.
- pbiggar 14y agoI know that they have to be apologetic like this, but the simple fact is that GitHub's uptime is fantastic. I run http://CircleCi.com http://CircleCi.com, and so we have upwards of 10,000 interactions with GitHub per day, whether API calls, clones, pulls, webhooks, etc. A seriously seriously small number of them fail. They know what they're doing, and they do a great job.
- aaronblohowiak 14y agoIf Github hasn't gotten their custom HA solution right, will you? Digging into their fix, they disabled automatic failover -- so all DB failures will now require manual intervention. While addressing this particular (erroneous) failover condition, it does raise minimum down time for true failures. Also, their mysql replicant's misconfiguration upon switching masters is also tied to their (stopgap) approach to preventing the hot failover. So, the second problem was due to a mis-use/misunderstanding of maintenance-mode. How is it possible that the slave could be pointed at the wrong master and have nobody notice for a day? What is the checklist to confirm that failover has occurred correctly? There is also lesson to be learned in the fact that their status page had scaling issues due to db connection limits. Static files are the most dependable!
- jaggederest 14y agoIt blows my mind that they aren't simply using Jekyll to generate and update the status page. I mean... they wrote it, right?
- autotravis 14y ago"There is also lesson to be learned in the fact that their status page had scaling issues due to db connection limits. Static files are the most dependable!" Seriously, why would a status page need to query a db?
- gsibble 14y agoI assume that the status server is not actively checking every Github server/service whenever someone pings it. It probably polls the servers every X seconds. The best place to store that type of data is in a DB. Where else would you put it?
- aaronblohowiak 14y ago> It probably polls the servers every X seconds. And then you could write out a new static file, just once, and send it to your edge server of choice.
- cwb71 14y agoThe part of this post that really blew my mind: We host our status site on Heroku to ensure its availability during an outage. However, during our downtime on Tuesday our status site experienced some availability issues. As traffic to the status site began to ramp up, we increased the number of dynos running from 8 to 64 and finally 90. This had a negative effect since we were running an old development database addon (shared database). The number of dynos maxed out the available connections to the database causing additional processes to crash. Ninety dynos for a status page? What was going on there?
- mbell 14y agoAnyone tested S3's static page hosting under heavy load? I would think you could just update the static file as a result of some events fired by your internal monitoring process.
- WestCoastJustin 14y agoS3 is great for static content. I was taking the AWS ops course and the instructor mentioned some very large organizations redirect their site to S3 when under DDOS so they can remain on-line. In fact, he said that AWS recommended this solution to them?! Can you fathom someone who is under DDOS, and you tell them, hey, just redirect that our way ;)
- fierarul 14y agoYou pay for the bandwidth on AWS. Of course they would be glad to redirect a DDOS their way. It's pure gold for AWS.
- biot 14y agoUse Jekyll and push the site to S3: https://github.com/mojombo/jekyll/wiki https://github.com/mojombo/jekyll/wiki https://github.com/laurilehmijoki/jekyll-s3#readme https://github.com/laurilehmijoki/jekyll-s3#readme
- dustym 14y ago
- cschep 14y agoInteresting to read about github using MySQL instead of Postgres. Anyone know why? I am just curious because of all the MySQL bashing I hear in the echo chamber.
- technoweenie 14y agoMostly because of legacy reasons, at this point.
- lonnyk 14y agoDo you have a source for this information?
- technoweenie 14y agoI have the source code, yes :)
- jaggederest 14y agohttps://github.com/technoweenie https://github.com/technoweenie He's on the Github team. I assume he's speaking Ex Cathedra.
- boundlessdreamz 14y agoThat sounds like you would have chosen differently if you had to choose now. Is that so?
- autotravis 14y agoThey use both, according to Zach Holman (http://zachholman.com/talk/unsucking-your-teams-development-environment http://zachholman.com/talk/unsucking-your-teams-development-...)
- technoweenie 14y agoThe only postgres we use is from internal Heroku apps. We use Mongo in a few places too.
- WestCoastJustin 14y agoHere are the makings of a bad week (Monday of all things) - MySQL schema migration causes high load, automated HA solution causes cascading database failure - MySQL cluster becomes out of sync - HA solution segfaults - Redis and MySQL become out of sync - Incorrect users have access to private repositories! Cleanup and recovery takes time, all I can say is, I'm glad it was not me who had that mess to clean up. I'm sure they are still working on it too! This brings to mind some my bad days.. OOM killer decides your Sybase database is using too much memory. Hardware error on DRBD master causes silent data corruption (this took a lot of recovery time on TBs of data). I've been bitten by the MySQL master/slave become out of sync. That is a bad place to be in.. do you copy your master database to the slaves.. that takes a long time even of a fast network.
- cageface 14y agoThis kind of thing is one of the main reasons I prefer to do app development instead of backend work now. I don't get calls at 3am any more.
- cagenut 14y agoI'd like to welcome the github ops/dbas to the club of people who've learned the hard way that automated database failover usually causes more downtime than it prevents. Here's sortof the seminal post on the matter in the mysql community: http://www.xaprb.com/blog/2009/08/30/failure-scenarios-and-solutions-in-master-master-replication/ http://www.xaprb.com/blog/2009/08/30/failure-scenarios-and-s... Though it turns into an MMM pile-on the tool doesn't matter so much as the scenarios. Automated failover is simply unlikely to make things better, and likely to make things worse, in most scenarios.
- aaronblohowiak 14y agoautomated failover in the case of too much load is usually not what you want to do. automated failure in the case of hw/network failure is usually what you want to do. differentiating the former from the latter is left as an exercise for the reader.
- koglerjs 14y agohttp://www.shoebuy.com/gym-shorts.htm http://www.shoebuy.com/gym-shorts.htm
- ghshephard 14y agoAutomated database failover is absolutely mandatory for HA environments (as in, there is no way to run a 5 9s system without it) but, poorly done, results in actually reducing your uptime (which is a separate concept from HA). I've been in a couple of environment in which developers have successfully rolled out automated database failover, and, my takeaway, is that's it usually not worth the cost - and with very, very few exceptions, most organizations can take the downtime of several minutes to do manual failover. In general, when rolling out these operational environment, they are only ready when you've found, and demonstrated 10-12 failure cases, and come up with workarounds. In other words - if you can't demonstrate how your environment will fail, then it's not ready for an HA deployment.
- Xorlev 14y agoEvery HA deployment I've done, the HA manager inevitably had issues to begin with. It takes time, patience, and a few late nights.
- jluxenberg 14y ago"16 of these repositories were private, and for seven minutes from 8:19 AM to 8:26 AM PDT on Tuesday, Sept 11th, were accessible to people outside of the repository's list of collaborators or team members" ouch!
- nslocum 14y agoOne of those repos was mine. :( Fortunately it was a fresh Rails app without anything important. However, it does make me rethink the security of storing my code on github.
- mckoss 14y agoI store proprietary code on github, but I would never recommend storing actual secrets (like keys or passwords).
- code0 14y agoI am really curious about the technical reasons how this might have happened.
- andrewljohnson 14y agoThe lack of any negative response on this thread is a testament both to the thoroughness of the post-mortem, and the outstanding quality of GitHub in general. In GitHub we trust. I can't imagine putting my code anywhere else right now.
- lokotecla1 14y agopara que sirve esta pagina soy nuevo
- dumbluck 14y agoThis was the awesome kind of explanation about what went wrong and what was learned that I wish everyone would do.
- donavanm 14y agoUpdate strategy of master first is interesting. I've always seen the other way with update standby, flip to standby, verify, update original master. Auto inc db keys once again cause horribleness. Nothing new there I suppose. And as mentioned the multi dyno + DB read status page is craaaazy. Why oh why isnt this a couple static objects. Automagically generate and push if you want. Give 'em a 60 second TTL and call it a day. Put them behind a different CDN & DNS then the rest of your site for bonus points.
- jyap 14y ago"As traffic to the status site began to ramp up, we increased the number of dynos running from 8 to 64 and finally 90." Wait, why isn't there some caching layer? eg. Generate a static page or use Varnish. This part makes no sense at all. At most you're then firing up another 5 dynos (or none) to handle the traffic. 90 is ridiculous.
- deleted 14y ago[deleted]
- gbog 14y agoGenuine question: github is built upon git, which is a rock solid system for storing dataand in these reports we read that github relies a lot on MySQL, so... Did the github guys ponder using git as their data store? Just an example, in git one can add comments on commits, would it be possible to use it for the github comment function? Or maybe it is?
- holman 14y agoGenerally, Git will be way too slow for that. Git is typically our bottleneck, since you're dealing with so much overhead and disk access to perform functions. Databases are best for, well, performing relational queries. In the case of commenting on a commit, if you store them only in the repository it becomes non-trivial to ask "show me all of the comments by this user" unless you have an intermediary cache layer (in which case you're back where you started).
- gbog 14y agoThanks for answering. Tell me if I'm wrong but MySQL would be behind a caching layer anyway, so the choice would be between cached git or cached git + mysql. In git, logging commits on a file from an author is also a kind of join, and it is surprisingly fast, so using git as a data store is a weird idea that I cannot take out of my head.