7 ms·

To be fair, Atwood thought his hosting provider (CrystalTech) was backing up his system. As it turns out, their entire VM backup solution failed silently, so ev

• dnsworks 17y ago
  If you fail to audit and test your disaster recovery procedures on a regular basis, then you fail at competently maintaining your infrastructure. No excuses.
  • brk 17y ago
    Yes, but I'm not sure if your comment is targeted at CrystalTech or codinghorror? You can't outsource liability. If YOU are not testing YOUR backups. YOU fail.
    • jrockway 17y ago
      It's a blog. How much time and money is worth spending on this? (I backup my Slicehost account with their official service. If they die and my backup dies with it, that's fine; I don't have the time or money to do anything better.)
      • robryan 17y ago
        I'd say a lot to Jeff, not only is it a popular blog standing alone and probably generates a decent income but it's also very important to him for cross promotion.
        • bad_user 17y ago
          And do you think Jeff's blog just got less popular?
          • gaius 17y ago
            He's not serving any AdWords off that page :-P
          • diN0bot 17y ago
            his position as looking down on coding horror has somewhat moved to being part of the team. not necessarily a bad thing.
      • samdk 17y ago
        You don't need to put much time or money into it. The bare minimum of a weekly cron-automated rsync job can be done in a few minutes, even if you've never done it before.
      • PStamatiou 17y ago
        If I was in his shoes, yeah I'd be freaking out too and trying to get everything recovered ASAP. Then again my blog pays more than my (startup) job so that's why I care about it.
    • Confusion 17y ago
      You can't outsource liability. So you've tested whether your toaster has proper grounding and other safety precautions, in case it shortcircuits? I bet you haven't and that why you should stop repeating that stupid soundbite. We all 'outsource' liability all the time: we pay others to perform services for us and hold them responsible for the proper execution of those services. This includes hosting content and backing up that content.
      • brk 17y ago
        So you've tested whether your toaster has proper grounding and other safety precautions, in case it shortcircuits? Nope. I have a fully paid and verified home owners insurance policy though that covers any probable loss from a toaster fire. It's sort of like having an offsite backup of my important personal stuff. Your example is really not a parallel to a data backup. It's difficult to fully test all of the toasters failure modes in a non-destructive way. It's easy to setup an automated backup to a remote location. Given the gmail storage limits, you could tar and gzip your files and email your gmail account with the backup data. A better example in a "homeowner" realm is the flood pans that you can buy to put under your water heater or washing machine. You'd like to be able to trust that the manufacturer made a waterproof device, but at the same time it's cheap and easy to insure yourself against the most common failure modes. The sound bite is not stupid. People who believe you can outsource all the messyness of keeping a website alive are continually bitten in the ass by EC2, Rackspace, etc. failures.
        • sailormoon 17y ago
          An insurance policy is equivalent to an offsite backup? So you don't have any personal items of any intrinsic value at all; you could lose it all, get a cheque in return, and be happy? Wow. Well, awesome, but I doubt that is common. And look, I agree with you in many ways - people need to take responsibility for their own backups, sure. But there is a division of responsibility. I mean, even if all my backups are 100% perfect, I am still trusting the hosting provider to, well, keep the power on. Pay the peering bill. Keep the server temp down. Not go bankrupt tomorrow. You can just follow this chain as far as you want. Whether you like it or not, you're utterly dependent on the DNS root server admins. There is nothing at all you can do to prepare yourself for their failure. I bet you can't generate your own electricity or grow your own food, either. All of civilisation is built on co-dependency and delegation of responsibility. It's the only way to do anything complex. At some point, you must delegate. Atwood should have checked - but he was paying his host to do it. That's like having an employee whose job it is to do backups. At some point, you just have to let go and trust them. Otherwise you can never really do anything; you're caught up in checking minutia; I can give examples of this kind of leadership failure until my keyboard breaks.
  • nkohari 17y ago
    I'm not disagreeing, I'm just saying that before people get drunk on schadenfreude they should put themselves in Atwood's position. If you pay someone for a service you generally expect it to work when you need it.
    • mcantelon 17y ago
      Yeah, but if you pay for a mission critical service and never bother testing it, you've pretty much passively decided to fail.
      • pyre 17y ago
        The same could be said for CrystalTech
        • mcantelon 17y ago
          I don't think anyone's saying that CrystalTech didn't also fail.
      • krschultz 17y ago
        Is codinghorror really mission critical? He will restore from some other backups. Downtime on a blog is not that big a deal other than people like to hate on Jeff Atwood. Now if he lost customer credit card records or something like that, then it would be warranted.
        • houseabsolute 17y ago
          Doesn't Jeff derive a significant portion of his personal income from the site? If so, I'd say mission critical is a good way to describe it.
        • dinkumthinkum 17y ago
          If codinghorror somehow allows Jeff to live without working then I would say it is pretty mission critical. Mission critical doesn't mean the Earth will blow up or something like that.
  • Confusion 17y ago
    Do you also test that your airbags work properly? I don't audit and test the safety measures on most of the equipment in my house: that's what I pay the supplier for and there'll be hell to pay when they screw it up. Similarly, I don't audit and test my disaster recovery procedures, because that is what I pay my hosting partner for.
    • klodolph 17y ago
      "Reasoning by analogy" The people who made your car were legally required to test your model of car by crashing it into another one and making sure the airbags work. This is also too expensive to test yourself. One does not modify an airbag system at all without retesting it. Your hosting provider has no legal requirement to test their backup system. At best, they have a contractual obligation. And they don't know how to test recovering your site, because testing whether it works is a different process for each site. Additionally, it's cheap to test backup procedures. Most people have a spare computer somewhere (maybe not in the data center), and it should only take a few hours to restore a copy of your site. Once a year. For Chris's sake, you could probably do it in the background and put a movie on.
    • nomoresecrets 17y ago
      When you test airbags, you have to then replace them. This is possibly the worst car analogy ever :-)
• deleted 17y ago
  [deleted]
• DTrejo 17y ago
  If anything, I'd say this is a sign not to work with CrystalTech. After a mistake like this, it may be a sign that they will never mess up again.
  • potatolicious 17y ago
    I disagree - it's a sign that they will never mess up their VM backup system again. It's also a sign that they lack experience and competence. So while your odds of suffering this problem are significantly lower, your odds of the infinite number of other problems are still troubling.
  • nkohari 17y ago
    We actually had our app hosted at CrystalTech until a couple months ago. Earlier this year, they had a critical power outage, which took our site offline. That's one of the reasons we're not there anymore.
  • klodolph 17y ago
    Well, I'm a bit more cynical than you. However, Jack in the Box is probably one of the least likely chains to experience an E. Coli O157:H7 outbreak (since 1993, at least).
• cag_ii 17y ago
  I'd argue that the general idea expressed in the post below applies as much to system administration as it does to programming. Rule #1: Take responsibility. http://74.125.93.132/search?q=cache:xGcmkwq9zUkJ:www.codinghorror.com/blog/archives/001079.html http://74.125.93.132/search?q=cache:xGcmkwq9zUkJ:www.codingh...
• pvg 17y ago
  It doesn't seem to be particularly fair to the hosting provider, really, to put their name on the failpage and to tweet about how it's half their fault. Especially after you've advocated redundant backups, implied that you have them, written about the advantages of hosting your images on S3. When a mishap reveals that you've actually done none of these things, it is more than a little disingenuous to try to emphasize your hosting provider's relatively minor role in making you look foolish.
  • akkartik 17y ago
    Relatively minor?! They failed to do what they said they would. They destroyed data and borked up the backups. If it wasn't Jeff Atwood but me, would it be more their fault? Or would I just be less disingenuous? Nobody's calling them names, or threatening to take business elsewhere or anything. But it's good to get sunlight in there, show them consequences, when they fail you. When a company providing you with service fails, you're allowed to scream it from the rooftops if you feel like it.
    • pvg 17y ago
      Their role in making him look foolish (what I said, incidentally) is indeed quite minor. If he'd actually done the things he advocated and, frankly, implied that he'd done, he'd still have his images, his downtime would have been close to zero and he'd probably get to write a triumphant article about the value of following his own sage advice. Someone else pointed out in another comment, there's a good analogy to be drawn to Atwood's own words here: http://www.codinghorror.com/blog/archives/001079.html http://www.codinghorror.com/blog/archives/001079.html
      • akkartik 17y ago
        "Their role in making him look foolish (what I said, incidentally) is indeed quite minor." Fair enough. But he's not complaining about looking foolish (nor is grandparent as far as I can see). He's complaining about losing data. In that link he isn't saying "it's your fault no matter what the world throws at you, suck it up." He's saying, "look harder at yourself before you decide somebody else is to blame." Not an issue here; can we agree it is their fault?
• InclinedPlane 17y ago
  Also to be fair the backup process Atwood uses for Stack Overflow proper (et al) appears to be much more robust (regular offsite backups, etc.) The failures in this case appear to be: 1. Not taking the robustness of his blogs seriously enough (a LOT of people make this mistake, especially with their own content). 2. Being overly trusting of the procedures of his hosting provider. He thought he could trust his hosting provider's "backups" since they are a big company with lots of customers and he paid money for said backups, turns out he got ripped off. 3. Forgetting the maxim that you have to own your core competencies, and existence should always be a core competency.