8 ms·

Cause of today's Github outage

214 points by jlangenauer 16y ago

geophile 16y ago
Anyone know what database system they use?
- blutonium 16y ago
  MySQL. Not that using anything else would have prevented this or would help restore the table faster...
  - trezor 16y ago
    Incorrect. The article states that they lost data between backup time and whenever the incident happened. Any reasonable quality transactional database system has a transaction log for a reason. If you were using commercial DB system like Microsoft SQL Server or Oracle (even a decade ago), this would not be an issue. No data loss. Businesses should care about their data, and I guess this is why the commercial databases are still doing fine in a landscape increasingly dominated by FOSS everywhere. I realize licensing costs do matter, but I can't fathom why people put up with the sad excuse of a RDBMS which is MySQL. For any nontrivial tasks it is slow, it's unreliable, attempting to secure your data by taking database backups (which seemingly can't even provide you with transactional safety!!!) renders the DB unusable and locked while backup is performed, having databases bigger than what you can store in memory makes it perform like a flat-file, etc etc ad infitum. Surely there must be something better people can use which is still free? Postgres for instance?
    - bigiain 16y ago
      FWIW, even MySQL let's you store "binlogs", which allow you to "rerun" all commands which changed the data since your last backup - if you've configured it to. Which makes me think, last time I had anything to do with adminning MySQL, binlogs were required for replication, I'm guessing this means github aren't replicating that database anywhere either...
    - pilif 16y ago
      There are many reasons for using Postgres over mysql, but having a backupable bin-log isn't one of them. MySQL does this too. This is an oversight in github's MySQL configuration (and their post states that they want to fix it)
    - jmulho 16y ago
      You are right on. The cardinal rule of traditional relational databases is that data loss is unacceptable. A database administrator who allows data loss is by definition incompetent. A database that allows data loss is by definition not a database. Commercial databases such as Oracle, SQL Server, and DB2 are built from the ground up to make it possible to recover everything up through the last committed transaction before the incident. I have always presumed that the leading open source traditional relational databases (mySQL and PostgreSQL) provide this same capability. In the github case (mySQL) it seems that full recovery may have been possible, but it was not possible within an acceptable timeframe. This may be a flaw in the software (lack of features to make quick recovery possible), or a flaw in the strategy (not configured to make quick recovery possible). However, any of these databases can easily be administered in a manner that allows data loss. The github event is not necessarily an indication of a problem with mySQL. It is unquestionably an indication of a problem with the administration of the database. Can anyone confirm that full recovery from this type of incident (within a reasonable timeframe) should be possible with both mySQL and PostgreSQL if administered properly?
  - jbellis 16y ago
    fwiw, Cassandra automatically takes a snapshot (which is basically instantaneous -- see http://www.riptano.com/blog/whats-new-cassandra-066 http://www.riptano.com/blog/whats-new-cassandra-066 for a brief explanation why) before truncating or dropping a column family, exactly because it makes recovering from a foot/gun incident like this so much less painful.
  - devmonk 16y ago
    Rails makes this way too easy to do by accident.
    - jlangenauer 16y ago
      Actually, in Rails 3, there are explicit guards against running the test suite against the production environment. See https://rails.lighthouseapp.com/projects/8994/tickets/5685-test_help-is-unusable-in-an-project-that-has-more-than-one-test-environment https://rails.lighthouseapp.com/projects/8994/tickets/5685-t...
      - catch23 16y ago
        However in this case, it happened on their continuous integration server -- which probably does more than just "rake test". On our hudson scripts, it does something like "rake db:reset" because some concurrent scripts require data to be persisted in the database and not rolled-back per test. So when it's running something like "rake db:reset", which simply recreates all tables & seeds the database, it just looks at your environment and assumes you know what you're doing. Hudson (and other continuous integration systems) are pretty automated so unless it prompted you, it would run the db scripts quietly.
    - gregwebs 16y ago
      If your database is on a socket then that would mean tests are running on a production machine, which just seems crazy. If it is on TCP than that means access isn't restricted properly. At any rate how is this Rails specific?
      - devmonk 16y ago
        "how is this Rails specific?" I spent years in Java prior to Ruby and through the various frameworks in Java, I never remember making the mistake or working with anyone who made the mistake of blowing away the production data when running tests. Since working in Rails, have had at least 2-3 experiences on the team with truncated/dropped tables or mangled data in production, and it isn't just the current team. Many other languages and frameworks don't have it as part of the normal process used by most developers to accidentally do this as much as it has seemed to happen in Rails 2, due to the conventions used. With Rails, the same project for the most part is in dev, test, and prod. Developers want database.yml revisioned with the rest of the app. Many Rails developers at some point run production against production DB locally, in order to run migrations on production prior to Capistrano deploys, etc., or maybe just to run a production console to run reports or tweak data. So, there is a very real chance for wild things to happen in such a risky environment. I'm not anti-Rails by any means, but I've not heard of this happening as much as it has historically with Rails in other frameworks.
  - ethangunderson 16y ago
    No system could prevent it, but there are databases(Oracle and MongoDB off the top of my head, and I'm sure there are others) that implement slave delays specifically to help mitigate human error like this.
  - chesser 16y ago
    > MySQL. Not that using anything else would have prevented this or would help restore the table faster... The slowness of the restore sounds like they have to insert a text dump back into the database, rather than simply copy over the binary files. Even copying over 1 TB from a single hard drive or between EC2 instances would only take a few hours, not days. And they weren't using binlogs. And they didn't separate out their test from production environment, which is understandable when you're in demo mode but not after a couple years and half a million users.
burgerbrain 16y ago
Good thing git is distributed. I've been working on my code all day and never even noticed!
- axod 16y ago
  So what's the point of using github if you don't notice it's offline?
  - burgerbrain 16y ago
    Github provides a convenient place for me to publish my work to the world, allowing others to pull it as they desire (my work machine is a laptop, and can never be relied upon to be up, or at any particular address). Because git is distributed my workflow is not effected in the slightest by "somebody else's" (github's) completely separate repo being down. Contrast this with something older like subversion and sourceforge. If sourceforge went down you were shit out of luck.
seanmcq 16y ago
Lesson, don't let your CI machine talk to your production servers (firewalls are good at this).
- nettdata 16y ago
  In my environment, our Dev's (individuals or environments/subnets) don't have access to PROD or QA, and our CIT boxes are in DEV. Likewise, QA and PROD only have access to their own environments. We have a build master that promotes a reviewed deployment package to QA and/or PROD environments, where the appropriate QA or PROD operations folks do the actual deployment. It's a luxury to have the resources available for this, but it's a life saver, because it really is stupidly easy to make a simple mistake and totally screw things up. The last time something similar happened to me, it happened to be at the end of a REALLY long day. And what do you know... that day was then made 24 hours longer, interspersed with the occasional cat nap while backups were being restored and verified. Fun times. Not.
  - smokinn 16y ago
    I'm not so sure it's a luxury. Maybe it is if your startup is servicing a group of techies who have knowledge of what problems lay in the background but most clients don't know and don't care. It's not that hard to keep up really. It's a question of a day or two of setup and then the hardish part of constant discipline to not take the "easy way out" and poke holes into the segregation you've set up. Mainly it takes a single team lead or CTO or whatever to be really explicit that you just don't break the steps and you'll avoid a LOT of problems. You'll still have problems, problems are inevitable, but in general you'll have mitigated them and with a proper backup and merge procedure you'll minimize downtime.
    - nettdata 16y ago
      Agreed, but to be clear, the luxury I was referring to was specifically having the DEV and PROD staff being totally separate teams/individuals, not just the separation of environments. In other words, nobody on the DEV team did any PROD operations, except in the case of bug investigation, tuning advice, etc.
deleted 16y ago
[deleted]
jlangenauer 16y ago
I think it's a measure of the goodwill in the community for Github (and perhaps, the fact that a lot of us have done something similar in the past) that they won't cop much flak at all for this.
- binaryfinery 16y ago
  I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises some basic questions about password storage. I don't run a site anything like github, but my production and test databases have different passwords and none of them are stored in a way that the test environment could get access to the live db, nor could the tests be run on the production environment. There's bad luck and theres asking for trouble.
  - alextgordon 16y ago
    If github spent their time so that their site raised absolutely no basic questions, then they'd still be in beta by now.
    - chesser 16y ago
      To me, that sounds like encouraging a race to the bottom. When you're in the business of storing other people's data, things like transactions, binary backups, QUICK RESTORES, and separating dev from production shouldn't be afterthoughts. They are core attributes. I would understand this for a demo, but not after a couple of years and half a million users. They have paying customers. "Reliable code hosting We spend all day and night making sure your repositories are secure, backed up and always available."
      - alextgordon 16y ago
        If we're going to use hindsight, we might as well look at the cause and the effect. Yes, they missed out some pretty obvious things. And what happened? A few hours of downtime because their restore was slow, plus a tiny bit of inconsequential data loss. Hardly a catastrophe. The fact is that every site has some sort of problems. Many of them will be completely obvious like this one. And while github could have gone through and attempted to fix them all, I much prefer they spend their time doing things that have more than a few hours of impact on my life. No doubt they will fix the issues involved today.
woan 16y ago
rookie mistake, better security if they isolate their networks too...
- gregwebs 16y ago
  I don't know why you first got down-voted. I have never worked on a project where a testing environment could access the production db.
  - sigzero 16y ago
    Yeah that is just insane. We typically run prod, stage, test, dev. The test and dev instances are even in another dmz.
  - storborg 16y ago
    I think he got downvoted for calling the GitHub team "rookies". Sure, they may not have tons of experience running 5-nines systems, but they've clearly built a great product that a lot of people love and respect.
    - chesser 16y ago
      They may not be rookies as Ruby hackers, but the evidence clearly points to them being rookies from the standpoint of data robustness. That's embarrassing enough when it's not your core business offering. They even brag about it on their main page.
aaronbrethorst 16y ago
Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing. From Chris' Twitter stream (http://twitter.com/#!/defunkt http://twitter.com/#!/defunkt): Seriously, I blame whoever wrote our crappy continuous integration software. Oh that's me
- nigelsampson 16y ago
  Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.
  - wccrawford 16y ago
    I did it for the main test database, and STILL felt like crap. SO many headaches.
  - mathgladiator 16y ago
    I once messed up a table that was responsible for a custom-distributed auto-increment field. If you roll that back and don't reset some servers, you have new data overwriting old data. Very bad stuff.
  - SkyMarshal 16y ago
    > Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Or sent a test email to thousands of customers in your prod database encouraging them to use web check-in for their non-existent flight tomorrow. Yeah, did that five years ago, talk about heart-attack-inducing. Quickly remedied by sending a second email to the same test set, thankfully, but that's the kind of mistake you never forget.
    - resdirector 16y ago
chunkbot 16y ago
I thought systems written in Erlang never go down! ;-)
tlb 16y ago
Forthright and classy. Compare to register.com, which had a big DNS outage Friday (affecting anybots.com) and never admitted to a problem.
- invisible 16y ago
  I believe they had a message on their homepage during the event about being DDoSed, but yeah no after-remarks is kind of ugly in my opinion. They could at least post and hide it from the general customer. 1) http://seclists.org/nanog/2010/Nov/415 http://seclists.org/nanog/2010/Nov/415
bytesong 16y ago
Minutes before the outage, my account would seem non-existent and all my repositories gone. They really scared the hell out of me. This will be a good reminder for me to always keep a local copy.
- SkyMarshal 16y ago
  Since it's git-based, is it even possible to not keep a local copy? I thought everything that goes into github has to be committed to your local git repo first?
  - bytesong 16y ago
    Well, there is always a local copy since you have to commit to the local repo first before you push it to github. That is, of course, as long as you don't remove it for no apparent reason.
- damncabbage 16y ago
  I was walking through someone's capistrano scripts, and suddenly everything came back "Not Found". My first thought was that the guy had just starting cleaning up his account. :)
latch 16y ago
I agree with what's been said so far. 1 - Shit happens 2 - We've all done stupid stuff 3 - Testing environment shouldn't have access to production What hasn't been said is how refreshing it is to see an honest and quick explanation. I know this type of approach is getting more and more common (see the foursquare outage), but in the grand scale of things, its still quite rare.
- chesser 16y ago
  I have a slightly higher threshold for considering something refreshing. It took several levels of basic mistakes for this to happen AND for the restore to be as slow as it is. Using MySQL with no transactions, no binary backups, no way to do a quick restore, and no separation of dev/production. DROP as opposed to DELETE cannot be rolled back and is therefore scary -- unless you aren't using transactions in the first place, in which case, WHEE!
  - matwood 16y ago
    I don't have much MySQL experience, but have lots of experience with other RDBMSs. Large restores take a long time no matter what. They shouldn't take days, but since we don't know how large the events table was we have no way of knowing if a faster restore was possible. A hot restore which it sounds like what they are doing may take longer by simple fact that it's restoring while the table is in use. And DROPs (or truncates) are almost always used if the goal is to remove all the data in a table like you would want if rebuilding the entire system. Something like 100M record transaction is not generally considered a good thing.
    - chesser 16y ago
      They take a long time if you have to rebuild the indexes, which is what you have to do if you only have a text dump. For MySQL, using the standard format (MyISAM), you can just do a file copy if you bothered to do a proper backup of the binary files.
1337p337 16y ago
It kind of makes me wish NILFS2 would become production-ready faster. Give MySQL its own partition, and just roll back to a previous checkpoint if you wipe everything. Not a substitute for backups, but a pretty speedy way to recover for a minor snafu like this.
- jacques_chester 16y ago
  NILFS is very nifty for write-oriented work, but I'm not sure if that's their workload. But if you're logging stuff, NILFS rocks: http://lists.luaforge.net/pipermail/kepler-project/2009-June/003452.html http://lists.luaforge.net/pipermail/kepler-project/2009-June...
ammmir 16y ago
simply checking if you're talking to a production instance could avert something like this. having some metadata in the db about whether the data stored there is acting as production and at what version and deployment level, so tests can have a sanity check before destructive activities.
- ammmir 16y ago
  why the downvote?
  - seldo 16y ago
    Presumably because the comment is obvious, unhelpful, and somewhat mean-spirited. I don't think anybody here needs a lesson in how to avoid accidentally dropping your production database.
    - ammmir 16y ago
      thanks for the explanation, i didn't mean to be evil. my suggestion might be obvious to some, but definitely not everyone, considering this isn't a singular occurrence.
jacques_chester 16y ago
Well done on coming clean. However this is why the dinosaur pens have such arduous red tape -- to try and catch serious errors before they hit production. A mate of mine works in that world and he regularly stops code going into production that would hose mission-critical government data. I prefer my agility to remain on the dev-and-test side of the fence.
dacort 16y ago
Second time I've heard of this happening fairly recently. Another incident, same cause. http://www.bigdoor.com/blog/bigdoor-api-service-has-been-restored/ http://www.bigdoor.com/blog/bigdoor-api-service-has-been-res...
- spudlyo 16y ago
  This was pretty painful. Their backups were too old, so it was necessary to do InnoDB data recovery rather than a straight-forward restore from backup. Since InnoDB table spaces never shrink, 80G of their truncated data was still all available in a single monolithic ibdata file. An InnoDB recovery tool named page_parser read their 80G ibdata file and spits out a maze of 16k InnoDB page files organized by an arbitrary index id. There are two internal InnoDB meta tables called SYS_INDEXES and SYS_TABLES which can give you a mapping from table name to PK index id. Unfortunately after the mass TRUNCATE all the tables got new index mappings, so it became a bit of table hide-and-seek. The InnoDB recovery tools lack a certain polish and maturity. You need to create a C header file for each table you want recover from the pile of 16k page files. You end up having to build a separate version of the constraints_parser binary for each table. There were bugs with the output of negative numbers, unicode handling, VARCHAR types with >= 127 characters, and some edge cases where InnoDB chooses to store CHAR(15) types as VARCHARs internally. Aleksander at Percona really saved the day, he was able to find and fix these bugs pretty quickly. I remember that magic moment when I finally was able to successfully checksum a huge block of the recovered data against the too-old-to-be-useful backup. "I love the smell of CRC32 in the morning. It smells like... victory."
peterwwillis 16y ago
If someone ever produces a good book of best practices for sysadm/syseng, please provide examples like these of why it's important to follow these best practices. Yes, we've all made silly mistakes. But if you're in that design meeting and somebody asks, should we do ABC in case of XYZ, try not to think about how complicated or time consuming it might be to do ABC. Think about the worst case. If not doing it could at some point bring down the whole business, perhaps you should ponder it some more. Actually, screw a book... Does anyone else want to start some wiki pages their experiences with screw-ups, the causes and the solutions? Does this exist in a comprehensive way and I just haven't found it?
- ojilles 16y ago
  Given the topic, it would be good to make mirrors of these pages before doing anything else with them.
- michael_dorfman 16y ago
  While a book of screw-ups might be amusing, I think it might be more instructive to look at the "old school" ways that screw-ups like this were avoided. I worked, back in the dark ages, for a health insurer that had two parallel environments-- one for testing/development, and one for production. None of the developers were allowed anywhere near the production environment. There was one full time employee, a former developer, whose primary job was to move code into production-- which he would only do when he received a signed document authorizing the change. Said document included the telephone numbers where the developer responsible for the code would be for the next 24 hours, so that one of the operators could call you in the middle of the night if your code caused any problems to the system. At the time, I used to think that this was ridiculous. After managing a staff of programmers myself, I'm not so sure.
  - viraptor 16y ago
    It depends on the software a bit, but I got a taste of "programmers not allowed on production" environment (in a very small scale). The problem I run into is that you have no idea what's going on on the production environment. Some characteristics are maybe reproducible on the dev site, but actual users will always do something differently. Sometimes not being able to poke the live system in a specific way while it's live, will cost you weeks of guessing in the dev environment. Then again, there is no perfect solution for this, is there? If you tried installing new version with more debugging points, you'd be deploying something unstable over something previously unstable and trying to push more data out which might be problematic in itself. I'm not even going into "rollback to stable didn't work" scenarios :(
jrockway 16y ago
I try to make my apps work against SQLite and the production database, so I can run all my tests against an in-memory SQLite database. This makes the tests run Really Fast, and it prevents a configuration error from causing my production data to go away. (It's not possible to do this in every case, especially if you make heavy use of stored procedures and triggers, but I don't. If I need client-independent behavior or integrity checks on top of the database, I just use a small RPC server. This makes testing and scaling easier, since there are just simple components that speak over the network. Much easier than predicting everything that could possibly happen to the database.)
sankara 16y ago
May be it's foresight or may be it's just paranoid. We've always used a entirely different username/password in prod and the password for prod never sits in the config files. Kind of saved us a couple of times. Sometimes it doesn't require a highly sophisticated setup to prevent a catastrophe.
random42 16y ago
I am a software developer, so I know "shit happens", but having the same configuration for database as testing environment, (same superuser name and password), which is not isolated from test environment, is pretty criminal even for a first time mistakeIMHO, especially for a product like github whom business, small and big trust with there business critical piece ("repository"). If I were running some critical code, I would have seriously reconsider github, or at-least ask for a detailed explanation on their engineering practices and fail-safe mechanisms.
- thibaut_barrere 16y ago
  Can you share more details about how you usually work ?
- scorpion032 16y ago
  I think the fact that they have been able to recover all critical data soon and will even update the event table over the next few days clearly displays their competence. Github isn't facebook and not seeing the last week's "activities" timeline for a few days isn't really that much of a problem.
  - chesser 16y ago
    > I think the fact that they have been able to recover all critical data soon and will even update the event table over the next few days clearly displays their competence. Taking a few days to restore a table indicates they only had a text dump of it. The details of how they screwed up indicate they were new to the idea of data robustness, despite it being the core of their business. Being a competent home user and being a competent data-robustness company involve slightly different levels of expertise.
    - risotto 16y ago
      Git itself handles robustness for what counts -- the source code and revision history. This outage did not affect repos as far as I have heard. Backing up and restoring mysql is an orthogonal problem. The fact that github can operate at all in a degraded mode very much indicates their competence. Many architectures can not operate without the entire db. Plus I bet they'll shore up their db operations next. Github is very much a young company that will take a lick or two in their way to greatness. All the startups here will.
jorangreef 16y ago
I moved my repos off GitHub to my EC2 server a month or two back since they're private and I was only using GitHub for keeping a copy of my code offsite. It's faster for simple push/pull and considering the sunk cost of my EC2 server, also free. I was trying to browse some repos on GitHub yesterday during the downtime and was thankful that my own were still available.
- jorangreef 16y ago
  Also, if you just need something for replicating code, Amazon is offering free micro servers on new accounts for up to a year I think.
deleted 16y ago
[deleted]
mkramlich 16y ago
Kudos to them for having the guts to say publicly that they accidentally destroyed their production database. It's been a great service, and I think as long as this kind of thing is rare, and none of my code repositories get corrupted or destroyed, I plan to stick with them.
donw 16y ago
This is why it's important to isolate production from other environments. Three rules have kept me from ever borking a production database: 1. Production DB credentials are only stored on the production appservers, and copied in at deploy time. 2. The production DB can only be accessed from the IPs of the production webfarm. 3. Staging, Testing, Development, and Everything Else live on separate networks and machines than production.
- oddi 16y ago
  While this is part of the problem, to me it seems like they didn't have proper restore procedures or at least hadn't tested them enough. There are countless ways to corrupt a database and restoring from backup is part of most remedies.
fizx 16y ago
Mysql 5.6+ has delayed replication. Until then, there's always tools like: http://www.maatkit.org/doc/mk-slave-delay.html http://www.maatkit.org/doc/mk-slave-delay.html

214 points by jlangenauer 16y ago

geophile 16y ago

Anyone know what database system they use?

blutonium 16y ago
MySQL. Not that using anything else would have prevented this or would help restore the table faster...
- trezor 16y ago
  Incorrect. The article states that they lost data between backup time and whenever the incident happened. Any reasonable quality transactional database system has a transaction log for a reason. If you were using commercial DB system like Microsoft SQL Server or Oracle (even a decade ago), this would not be an issue. No data loss. Businesses should care about their data, and I guess this is why the commercial databases are still doing fine in a landscape increasingly dominated by FOSS everywhere. I realize licensing costs do matter, but I can't fathom why people put up with the sad excuse of a RDBMS which is MySQL. For any nontrivial tasks it is slow, it's unreliable, attempting to secure your data by taking database backups (which seemingly can't even provide you with transactional safety!!!) renders the DB unusable and locked while backup is performed, having databases bigger than what you can store in memory makes it perform like a flat-file, etc etc ad infitum. Surely there must be something better people can use which is still free? Postgres for instance?
  - bigiain 16y ago
    FWIW, even MySQL let's you store "binlogs", which allow you to "rerun" all commands which changed the data since your last backup - if you've configured it to. Which makes me think, last time I had anything to do with adminning MySQL, binlogs were required for replication, I'm guessing this means github aren't replicating that database anywhere either...
  - pilif 16y ago
    There are many reasons for using Postgres over mysql, but having a backupable bin-log isn't one of them. MySQL does this too. This is an oversight in github's MySQL configuration (and their post states that they want to fix it)
  - jmulho 16y ago
    You are right on. The cardinal rule of traditional relational databases is that data loss is unacceptable. A database administrator who allows data loss is by definition incompetent. A database that allows data loss is by definition not a database. Commercial databases such as Oracle, SQL Server, and DB2 are built from the ground up to make it possible to recover everything up through the last committed transaction before the incident. I have always presumed that the leading open source traditional relational databases (mySQL and PostgreSQL) provide this same capability. In the github case (mySQL) it seems that full recovery may have been possible, but it was not possible within an acceptable timeframe. This may be a flaw in the software (lack of features to make quick recovery possible), or a flaw in the strategy (not configured to make quick recovery possible). However, any of these databases can easily be administered in a manner that allows data loss. The github event is not necessarily an indication of a problem with mySQL. It is unquestionably an indication of a problem with the administration of the database. Can anyone confirm that full recovery from this type of incident (within a reasonable timeframe) should be possible with both mySQL and PostgreSQL if administered properly?
- jbellis 16y ago
  fwiw, Cassandra automatically takes a snapshot (which is basically instantaneous -- see http://www.riptano.com/blog/whats-new-cassandra-066 http://www.riptano.com/blog/whats-new-cassandra-066 for a brief explanation why) before truncating or dropping a column family, exactly because it makes recovering from a foot/gun incident like this so much less painful.
- devmonk 16y ago
  Rails makes this way too easy to do by accident.
  - jlangenauer 16y ago
    Actually, in Rails 3, there are explicit guards against running the test suite against the production environment. See https://rails.lighthouseapp.com/projects/8994/tickets/5685-test_help-is-unusable-in-an-project-that-has-more-than-one-test-environment https://rails.lighthouseapp.com/projects/8994/tickets/5685-t...
    - catch23 16y ago
      However in this case, it happened on their continuous integration server -- which probably does more than just "rake test". On our hudson scripts, it does something like "rake db:reset" because some concurrent scripts require data to be persisted in the database and not rolled-back per test. So when it's running something like "rake db:reset", which simply recreates all tables & seeds the database, it just looks at your environment and assumes you know what you're doing. Hudson (and other continuous integration systems) are pretty automated so unless it prompted you, it would run the db scripts quietly.
  - gregwebs 16y ago
    If your database is on a socket then that would mean tests are running on a production machine, which just seems crazy. If it is on TCP than that means access isn't restricted properly. At any rate how is this Rails specific?
    - devmonk 16y ago
      "how is this Rails specific?" I spent years in Java prior to Ruby and through the various frameworks in Java, I never remember making the mistake or working with anyone who made the mistake of blowing away the production data when running tests. Since working in Rails, have had at least 2-3 experiences on the team with truncated/dropped tables or mangled data in production, and it isn't just the current team. Many other languages and frameworks don't have it as part of the normal process used by most developers to accidentally do this as much as it has seemed to happen in Rails 2, due to the conventions used. With Rails, the same project for the most part is in dev, test, and prod. Developers want database.yml revisioned with the rest of the app. Many Rails developers at some point run production against production DB locally, in order to run migrations on production prior to Capistrano deploys, etc., or maybe just to run a production console to run reports or tweak data. So, there is a very real chance for wild things to happen in such a risky environment. I'm not anti-Rails by any means, but I've not heard of this happening as much as it has historically with Rails in other frameworks.
- ethangunderson 16y ago
  No system could prevent it, but there are databases(Oracle and MongoDB off the top of my head, and I'm sure there are others) that implement slave delays specifically to help mitigate human error like this.
- chesser 16y ago
  > MySQL. Not that using anything else would have prevented this or would help restore the table faster... The slowness of the restore sounds like they have to insert a text dump back into the database, rather than simply copy over the binary files. Even copying over 1 TB from a single hard drive or between EC2 instances would only take a few hours, not days. And they weren't using binlogs. And they didn't separate out their test from production environment, which is understandable when you're in demo mode but not after a couple years and half a million users.

burgerbrain 16y ago

Good thing git is distributed. I've been working on my code all day and never even noticed!

axod 16y ago
So what's the point of using github if you don't notice it's offline?
- burgerbrain 16y ago
  Github provides a convenient place for me to publish my work to the world, allowing others to pull it as they desire (my work machine is a laptop, and can never be relied upon to be up, or at any particular address). Because git is distributed my workflow is not effected in the slightest by "somebody else's" (github's) completely separate repo being down. Contrast this with something older like subversion and sourceforge. If sourceforge went down you were shit out of luck.

seanmcq 16y ago

Lesson, don't let your CI machine talk to your production servers (firewalls are good at this).

nettdata 16y ago
In my environment, our Dev's (individuals or environments/subnets) don't have access to PROD or QA, and our CIT boxes are in DEV. Likewise, QA and PROD only have access to their own environments. We have a build master that promotes a reviewed deployment package to QA and/or PROD environments, where the appropriate QA or PROD operations folks do the actual deployment. It's a luxury to have the resources available for this, but it's a life saver, because it really is stupidly easy to make a simple mistake and totally screw things up. The last time something similar happened to me, it happened to be at the end of a REALLY long day. And what do you know... that day was then made 24 hours longer, interspersed with the occasional cat nap while backups were being restored and verified. Fun times. Not.
- smokinn 16y ago
  I'm not so sure it's a luxury. Maybe it is if your startup is servicing a group of techies who have knowledge of what problems lay in the background but most clients don't know and don't care. It's not that hard to keep up really. It's a question of a day or two of setup and then the hardish part of constant discipline to not take the "easy way out" and poke holes into the segregation you've set up. Mainly it takes a single team lead or CTO or whatever to be really explicit that you just don't break the steps and you'll avoid a LOT of problems. You'll still have problems, problems are inevitable, but in general you'll have mitigated them and with a proper backup and merge procedure you'll minimize downtime.
  - nettdata 16y ago
    Agreed, but to be clear, the luxury I was referring to was specifically having the DEV and PROD staff being totally separate teams/individuals, not just the separation of environments. In other words, nobody on the DEV team did any PROD operations, except in the case of bug investigation, tuning advice, etc.

deleted 16y ago

[deleted]

jlangenauer 16y ago

I think it's a measure of the goodwill in the community for Github (and perhaps, the fact that a lot of us have done something similar in the past) that they won't cop much flak at all for this.

binaryfinery 16y ago
I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises some basic questions about password storage. I don't run a site anything like github, but my production and test databases have different passwords and none of them are stored in a way that the test environment could get access to the live db, nor could the tests be run on the production environment. There's bad luck and theres asking for trouble.
- alextgordon 16y ago
  If github spent their time so that their site raised absolutely no basic questions, then they'd still be in beta by now.
  - chesser 16y ago
    To me, that sounds like encouraging a race to the bottom. When you're in the business of storing other people's data, things like transactions, binary backups, QUICK RESTORES, and separating dev from production shouldn't be afterthoughts. They are core attributes. I would understand this for a demo, but not after a couple of years and half a million users. They have paying customers. "Reliable code hosting We spend all day and night making sure your repositories are secure, backed up and always available."
    - alextgordon 16y ago
      If we're going to use hindsight, we might as well look at the cause and the effect. Yes, they missed out some pretty obvious things. And what happened? A few hours of downtime because their restore was slow, plus a tiny bit of inconsequential data loss. Hardly a catastrophe. The fact is that every site has some sort of problems. Many of them will be completely obvious like this one. And while github could have gone through and attempted to fix them all, I much prefer they spend their time doing things that have more than a few hours of impact on my life. No doubt they will fix the issues involved today.

woan 16y ago

rookie mistake, better security if they isolate their networks too...

gregwebs 16y ago
I don't know why you first got down-voted. I have never worked on a project where a testing environment could access the production db.
- sigzero 16y ago
  Yeah that is just insane. We typically run prod, stage, test, dev. The test and dev instances are even in another dmz.
- storborg 16y ago
  I think he got downvoted for calling the GitHub team "rookies". Sure, they may not have tons of experience running 5-nines systems, but they've clearly built a great product that a lot of people love and respect.
  - chesser 16y ago
    They may not be rookies as Ruby hackers, but the evidence clearly points to them being rookies from the standpoint of data robustness. That's embarrassing enough when it's not your core business offering. They even brag about it on their main page.

aaronbrethorst 16y ago

Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing. From Chris' Twitter stream (http://twitter.com/#!/defunkt http://twitter.com/#!/defunkt): Seriously, I blame whoever wrote our crappy continuous integration software. Oh that's me

nigelsampson 16y ago
Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.
- wccrawford 16y ago
  I did it for the main test database, and STILL felt like crap. SO many headaches.
- mathgladiator 16y ago
  I once messed up a table that was responsible for a custom-distributed auto-increment field. If you roll that back and don't reset some servers, you have new data overwriting old data. Very bad stuff.
- SkyMarshal 16y ago
  > Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Or sent a test email to thousands of customers in your prod database encouraging them to use web check-in for their non-existent flight tomorrow. Yeah, did that five years ago, talk about heart-attack-inducing. Quickly remedied by sending a second email to the same test set, thankfully, but that's the kind of mistake you never forget.
  - resdirector 16y ago

chunkbot 16y ago

I thought systems written in Erlang never go down! ;-)

tlb 16y ago

Forthright and classy. Compare to register.com, which had a big DNS outage Friday (affecting anybots.com) and never admitted to a problem.

invisible 16y ago
I believe they had a message on their homepage during the event about being DDoSed, but yeah no after-remarks is kind of ugly in my opinion. They could at least post and hide it from the general customer. 1) http://seclists.org/nanog/2010/Nov/415 http://seclists.org/nanog/2010/Nov/415

bytesong 16y ago

Minutes before the outage, my account would seem non-existent and all my repositories gone. They really scared the hell out of me. This will be a good reminder for me to always keep a local copy.

SkyMarshal 16y ago
Since it's git-based, is it even possible to not keep a local copy? I thought everything that goes into github has to be committed to your local git repo first?
- bytesong 16y ago
  Well, there is always a local copy since you have to commit to the local repo first before you push it to github. That is, of course, as long as you don't remove it for no apparent reason.
damncabbage 16y ago
I was walking through someone's capistrano scripts, and suddenly everything came back "Not Found". My first thought was that the guy had just starting cleaning up his account. :)

latch 16y ago

I agree with what's been said so far. 1 - Shit happens 2 - We've all done stupid stuff 3 - Testing environment shouldn't have access to production What hasn't been said is how refreshing it is to see an honest and quick explanation. I know this type of approach is getting more and more common (see the foursquare outage), but in the grand scale of things, its still quite rare.

chesser 16y ago
I have a slightly higher threshold for considering something refreshing. It took several levels of basic mistakes for this to happen AND for the restore to be as slow as it is. Using MySQL with no transactions, no binary backups, no way to do a quick restore, and no separation of dev/production. DROP as opposed to DELETE cannot be rolled back and is therefore scary -- unless you aren't using transactions in the first place, in which case, WHEE!
- matwood 16y ago
  I don't have much MySQL experience, but have lots of experience with other RDBMSs. Large restores take a long time no matter what. They shouldn't take days, but since we don't know how large the events table was we have no way of knowing if a faster restore was possible. A hot restore which it sounds like what they are doing may take longer by simple fact that it's restoring while the table is in use. And DROPs (or truncates) are almost always used if the goal is to remove all the data in a table like you would want if rebuilding the entire system. Something like 100M record transaction is not generally considered a good thing.
  - chesser 16y ago
    They take a long time if you have to rebuild the indexes, which is what you have to do if you only have a text dump. For MySQL, using the standard format (MyISAM), you can just do a file copy if you bothered to do a proper backup of the binary files.

1337p337 16y ago

It kind of makes me wish NILFS2 would become production-ready faster. Give MySQL its own partition, and just roll back to a previous checkpoint if you wipe everything. Not a substitute for backups, but a pretty speedy way to recover for a minor snafu like this.

jacques_chester 16y ago
NILFS is very nifty for write-oriented work, but I'm not sure if that's their workload. But if you're logging stuff, NILFS rocks: http://lists.luaforge.net/pipermail/kepler-project/2009-June/003452.html http://lists.luaforge.net/pipermail/kepler-project/2009-June...

ammmir 16y ago

simply checking if you're talking to a production instance could avert something like this. having some metadata in the db about whether the data stored there is acting as production and at what version and deployment level, so tests can have a sanity check before destructive activities.

ammmir 16y ago
why the downvote?
- seldo 16y ago
  Presumably because the comment is obvious, unhelpful, and somewhat mean-spirited. I don't think anybody here needs a lesson in how to avoid accidentally dropping your production database.
  - ammmir 16y ago
    thanks for the explanation, i didn't mean to be evil. my suggestion might be obvious to some, but definitely not everyone, considering this isn't a singular occurrence.

jacques_chester 16y ago

Well done on coming clean. However this is why the dinosaur pens have such arduous red tape -- to try and catch serious errors before they hit production. A mate of mine works in that world and he regularly stops code going into production that would hose mission-critical government data. I prefer my agility to remain on the dev-and-test side of the fence.

dacort 16y ago

Second time I've heard of this happening fairly recently. Another incident, same cause. http://www.bigdoor.com/blog/bigdoor-api-service-has-been-restored/ http://www.bigdoor.com/blog/bigdoor-api-service-has-been-res...

spudlyo 16y ago
This was pretty painful. Their backups were too old, so it was necessary to do InnoDB data recovery rather than a straight-forward restore from backup. Since InnoDB table spaces never shrink, 80G of their truncated data was still all available in a single monolithic ibdata file. An InnoDB recovery tool named page_parser read their 80G ibdata file and spits out a maze of 16k InnoDB page files organized by an arbitrary index id. There are two internal InnoDB meta tables called SYS_INDEXES and SYS_TABLES which can give you a mapping from table name to PK index id. Unfortunately after the mass TRUNCATE all the tables got new index mappings, so it became a bit of table hide-and-seek. The InnoDB recovery tools lack a certain polish and maturity. You need to create a C header file for each table you want recover from the pile of 16k page files. You end up having to build a separate version of the constraints_parser binary for each table. There were bugs with the output of negative numbers, unicode handling, VARCHAR types with >= 127 characters, and some edge cases where InnoDB chooses to store CHAR(15) types as VARCHARs internally. Aleksander at Percona really saved the day, he was able to find and fix these bugs pretty quickly. I remember that magic moment when I finally was able to successfully checksum a huge block of the recovered data against the too-old-to-be-useful backup. "I love the smell of CRC32 in the morning. It smells like... victory."

peterwwillis 16y ago

If someone ever produces a good book of best practices for sysadm/syseng, please provide examples like these of why it's important to follow these best practices. Yes, we've all made silly mistakes. But if you're in that design meeting and somebody asks, should we do ABC in case of XYZ, try not to think about how complicated or time consuming it might be to do ABC. Think about the worst case. If not doing it could at some point bring down the whole business, perhaps you should ponder it some more. Actually, screw a book... Does anyone else want to start some wiki pages their experiences with screw-ups, the causes and the solutions? Does this exist in a comprehensive way and I just haven't found it?

ojilles 16y ago
Given the topic, it would be good to make mirrors of these pages before doing anything else with them.
michael_dorfman 16y ago
While a book of screw-ups might be amusing, I think it might be more instructive to look at the "old school" ways that screw-ups like this were avoided. I worked, back in the dark ages, for a health insurer that had two parallel environments-- one for testing/development, and one for production. None of the developers were allowed anywhere near the production environment. There was one full time employee, a former developer, whose primary job was to move code into production-- which he would only do when he received a signed document authorizing the change. Said document included the telephone numbers where the developer responsible for the code would be for the next 24 hours, so that one of the operators could call you in the middle of the night if your code caused any problems to the system. At the time, I used to think that this was ridiculous. After managing a staff of programmers myself, I'm not so sure.
- viraptor 16y ago
  It depends on the software a bit, but I got a taste of "programmers not allowed on production" environment (in a very small scale). The problem I run into is that you have no idea what's going on on the production environment. Some characteristics are maybe reproducible on the dev site, but actual users will always do something differently. Sometimes not being able to poke the live system in a specific way while it's live, will cost you weeks of guessing in the dev environment. Then again, there is no perfect solution for this, is there? If you tried installing new version with more debugging points, you'd be deploying something unstable over something previously unstable and trying to push more data out which might be problematic in itself. I'm not even going into "rollback to stable didn't work" scenarios :(

jrockway 16y ago

I try to make my apps work against SQLite and the production database, so I can run all my tests against an in-memory SQLite database. This makes the tests run Really Fast, and it prevents a configuration error from causing my production data to go away. (It's not possible to do this in every case, especially if you make heavy use of stored procedures and triggers, but I don't. If I need client-independent behavior or integrity checks on top of the database, I just use a small RPC server. This makes testing and scaling easier, since there are just simple components that speak over the network. Much easier than predicting everything that could possibly happen to the database.)

sankara 16y ago

May be it's foresight or may be it's just paranoid. We've always used a entirely different username/password in prod and the password for prod never sits in the config files. Kind of saved us a couple of times. Sometimes it doesn't require a highly sophisticated setup to prevent a catastrophe.

random42 16y ago

I am a software developer, so I know "shit happens", but having the same configuration for database as testing environment, (same superuser name and password), which is not isolated from test environment, is pretty criminal even for a first time mistakeIMHO, especially for a product like github whom business, small and big trust with there business critical piece ("repository"). If I were running some critical code, I would have seriously reconsider github, or at-least ask for a detailed explanation on their engineering practices and fail-safe mechanisms.

thibaut_barrere 16y ago
Can you share more details about how you usually work ?
scorpion032 16y ago
I think the fact that they have been able to recover all critical data soon and will even update the event table over the next few days clearly displays their competence. Github isn't facebook and not seeing the last week's "activities" timeline for a few days isn't really that much of a problem.
- chesser 16y ago
  > I think the fact that they have been able to recover all critical data soon and will even update the event table over the next few days clearly displays their competence. Taking a few days to restore a table indicates they only had a text dump of it. The details of how they screwed up indicate they were new to the idea of data robustness, despite it being the core of their business. Being a competent home user and being a competent data-robustness company involve slightly different levels of expertise.
  - risotto 16y ago
    Git itself handles robustness for what counts -- the source code and revision history. This outage did not affect repos as far as I have heard. Backing up and restoring mysql is an orthogonal problem. The fact that github can operate at all in a degraded mode very much indicates their competence. Many architectures can not operate without the entire db. Plus I bet they'll shore up their db operations next. Github is very much a young company that will take a lick or two in their way to greatness. All the startups here will.

jorangreef 16y ago

I moved my repos off GitHub to my EC2 server a month or two back since they're private and I was only using GitHub for keeping a copy of my code offsite. It's faster for simple push/pull and considering the sunk cost of my EC2 server, also free. I was trying to browse some repos on GitHub yesterday during the downtime and was thankful that my own were still available.

jorangreef 16y ago
Also, if you just need something for replicating code, Amazon is offering free micro servers on new accounts for up to a year I think.

deleted 16y ago

[deleted]

mkramlich 16y ago

Kudos to them for having the guts to say publicly that they accidentally destroyed their production database. It's been a great service, and I think as long as this kind of thing is rare, and none of my code repositories get corrupted or destroyed, I plan to stick with them.

donw 16y ago

This is why it's important to isolate production from other environments. Three rules have kept me from ever borking a production database: 1. Production DB credentials are only stored on the production appservers, and copied in at deploy time. 2. The production DB can only be accessed from the IPs of the production webfarm. 3. Staging, Testing, Development, and Everything Else live on separate networks and machines than production.

oddi 16y ago
While this is part of the problem, to me it seems like they didn't have proper restore procedures or at least hadn't tested them enough. There are countless ways to corrupt a database and restoring from backup is part of most remedies.

fizx 16y ago

Mysql 5.6+ has delayed replication. Until then, there's always tools like: http://www.maatkit.org/doc/mk-slave-delay.html http://www.maatkit.org/doc/mk-slave-delay.html