6 ms·

If you fail to audit and test your disaster recovery procedures on a regular basis, then you fail at competently maintaining your infrastructure. No excuses.

brk 17y ago
Yes, but I'm not sure if your comment is targeted at CrystalTech or codinghorror? You can't outsource liability. If YOU are not testing YOUR backups. YOU fail.
- jrockway 17y ago
  It's a blog. How much time and money is worth spending on this? (I backup my Slicehost account with their official service. If they die and my backup dies with it, that's fine; I don't have the time or money to do anything better.)
  - robryan 17y ago
    I'd say a lot to Jeff, not only is it a popular blog standing alone and probably generates a decent income but it's also very important to him for cross promotion.
    - bad_user 17y ago
      And do you think Jeff's blog just got less popular?
      - gaius 17y ago
        He's not serving any AdWords off that page :-P
      - diN0bot 17y ago
        his position as looking down on coding horror has somewhat moved to being part of the team. not necessarily a bad thing.
  - samdk 17y ago
    You don't need to put much time or money into it. The bare minimum of a weekly cron-automated rsync job can be done in a few minutes, even if you've never done it before.
  - PStamatiou 17y ago
    If I was in his shoes, yeah I'd be freaking out too and trying to get everything recovered ASAP. Then again my blog pays more than my (startup) job so that's why I care about it.
- Confusion 17y ago
  You can't outsource liability. So you've tested whether your toaster has proper grounding and other safety precautions, in case it shortcircuits? I bet you haven't and that why you should stop repeating that stupid soundbite. We all 'outsource' liability all the time: we pay others to perform services for us and hold them responsible for the proper execution of those services. This includes hosting content and backing up that content.
  - brk 17y ago
    So you've tested whether your toaster has proper grounding and other safety precautions, in case it shortcircuits? Nope. I have a fully paid and verified home owners insurance policy though that covers any probable loss from a toaster fire. It's sort of like having an offsite backup of my important personal stuff. Your example is really not a parallel to a data backup. It's difficult to fully test all of the toasters failure modes in a non-destructive way. It's easy to setup an automated backup to a remote location. Given the gmail storage limits, you could tar and gzip your files and email your gmail account with the backup data. A better example in a "homeowner" realm is the flood pans that you can buy to put under your water heater or washing machine. You'd like to be able to trust that the manufacturer made a waterproof device, but at the same time it's cheap and easy to insure yourself against the most common failure modes. The sound bite is not stupid. People who believe you can outsource all the messyness of keeping a website alive are continually bitten in the ass by EC2, Rackspace, etc. failures.
    - sailormoon 17y ago
      An insurance policy is equivalent to an offsite backup? So you don't have any personal items of any intrinsic value at all; you could lose it all, get a cheque in return, and be happy? Wow. Well, awesome, but I doubt that is common. And look, I agree with you in many ways - people need to take responsibility for their own backups, sure. But there is a division of responsibility. I mean, even if all my backups are 100% perfect, I am still trusting the hosting provider to, well, keep the power on. Pay the peering bill. Keep the server temp down. Not go bankrupt tomorrow. You can just follow this chain as far as you want. Whether you like it or not, you're utterly dependent on the DNS root server admins. There is nothing at all you can do to prepare yourself for their failure. I bet you can't generate your own electricity or grow your own food, either. All of civilisation is built on co-dependency and delegation of responsibility. It's the only way to do anything complex. At some point, you must delegate. Atwood should have checked - but he was paying his host to do it. That's like having an employee whose job it is to do backups. At some point, you just have to let go and trust them. Otherwise you can never really do anything; you're caught up in checking minutia; I can give examples of this kind of leadership failure until my keyboard breaks.
nkohari 17y ago
I'm not disagreeing, I'm just saying that before people get drunk on schadenfreude they should put themselves in Atwood's position. If you pay someone for a service you generally expect it to work when you need it.
- mcantelon 17y ago
  Yeah, but if you pay for a mission critical service and never bother testing it, you've pretty much passively decided to fail.
  - pyre 17y ago
    The same could be said for CrystalTech
    - mcantelon 17y ago
      I don't think anyone's saying that CrystalTech didn't also fail.
  - krschultz 17y ago
    Is codinghorror really mission critical? He will restore from some other backups. Downtime on a blog is not that big a deal other than people like to hate on Jeff Atwood. Now if he lost customer credit card records or something like that, then it would be warranted.
    - houseabsolute 17y ago
      Doesn't Jeff derive a significant portion of his personal income from the site? If so, I'd say mission critical is a good way to describe it.
    - dinkumthinkum 17y ago
      If codinghorror somehow allows Jeff to live without working then I would say it is pretty mission critical. Mission critical doesn't mean the Earth will blow up or something like that.
Confusion 17y ago
Do you also test that your airbags work properly? I don't audit and test the safety measures on most of the equipment in my house: that's what I pay the supplier for and there'll be hell to pay when they screw it up. Similarly, I don't audit and test my disaster recovery procedures, because that is what I pay my hosting partner for.
- klodolph 17y ago
  "Reasoning by analogy" The people who made your car were legally required to test your model of car by crashing it into another one and making sure the airbags work. This is also too expensive to test yourself. One does not modify an airbag system at all without retesting it. Your hosting provider has no legal requirement to test their backup system. At best, they have a contractual obligation. And they don't know how to test recovering your site, because testing whether it works is a different process for each site. Additionally, it's cheap to test backup procedures. Most people have a spare computer somewhere (maybe not in the data center), and it should only take a few hours to restore a copy of your site. Once a year. For Chris's sake, you could probably do it in the background and put a movie on.
- nomoresecrets 17y ago
  When you test airbags, you have to then replace them. This is possibly the worst car analogy ever :-)

brk 17y ago

Yes, but I'm not sure if your comment is targeted at CrystalTech or codinghorror? You can't outsource liability. If YOU are not testing YOUR backups. YOU fail.

jrockway 17y ago
It's a blog. How much time and money is worth spending on this? (I backup my Slicehost account with their official service. If they die and my backup dies with it, that's fine; I don't have the time or money to do anything better.)
- robryan 17y ago
  I'd say a lot to Jeff, not only is it a popular blog standing alone and probably generates a decent income but it's also very important to him for cross promotion.
  - bad_user 17y ago
    And do you think Jeff's blog just got less popular?
    - gaius 17y ago
      He's not serving any AdWords off that page :-P
    - diN0bot 17y ago
      his position as looking down on coding horror has somewhat moved to being part of the team. not necessarily a bad thing.
- samdk 17y ago
  You don't need to put much time or money into it. The bare minimum of a weekly cron-automated rsync job can be done in a few minutes, even if you've never done it before.
- PStamatiou 17y ago
  If I was in his shoes, yeah I'd be freaking out too and trying to get everything recovered ASAP. Then again my blog pays more than my (startup) job so that's why I care about it.
Confusion 17y ago
You can't outsource liability. So you've tested whether your toaster has proper grounding and other safety precautions, in case it shortcircuits? I bet you haven't and that why you should stop repeating that stupid soundbite. We all 'outsource' liability all the time: we pay others to perform services for us and hold them responsible for the proper execution of those services. This includes hosting content and backing up that content.
- brk 17y ago
  So you've tested whether your toaster has proper grounding and other safety precautions, in case it shortcircuits? Nope. I have a fully paid and verified home owners insurance policy though that covers any probable loss from a toaster fire. It's sort of like having an offsite backup of my important personal stuff. Your example is really not a parallel to a data backup. It's difficult to fully test all of the toasters failure modes in a non-destructive way. It's easy to setup an automated backup to a remote location. Given the gmail storage limits, you could tar and gzip your files and email your gmail account with the backup data. A better example in a "homeowner" realm is the flood pans that you can buy to put under your water heater or washing machine. You'd like to be able to trust that the manufacturer made a waterproof device, but at the same time it's cheap and easy to insure yourself against the most common failure modes. The sound bite is not stupid. People who believe you can outsource all the messyness of keeping a website alive are continually bitten in the ass by EC2, Rackspace, etc. failures.
  - sailormoon 17y ago
    An insurance policy is equivalent to an offsite backup? So you don't have any personal items of any intrinsic value at all; you could lose it all, get a cheque in return, and be happy? Wow. Well, awesome, but I doubt that is common. And look, I agree with you in many ways - people need to take responsibility for their own backups, sure. But there is a division of responsibility. I mean, even if all my backups are 100% perfect, I am still trusting the hosting provider to, well, keep the power on. Pay the peering bill. Keep the server temp down. Not go bankrupt tomorrow. You can just follow this chain as far as you want. Whether you like it or not, you're utterly dependent on the DNS root server admins. There is nothing at all you can do to prepare yourself for their failure. I bet you can't generate your own electricity or grow your own food, either. All of civilisation is built on co-dependency and delegation of responsibility. It's the only way to do anything complex. At some point, you must delegate. Atwood should have checked - but he was paying his host to do it. That's like having an employee whose job it is to do backups. At some point, you just have to let go and trust them. Otherwise you can never really do anything; you're caught up in checking minutia; I can give examples of this kind of leadership failure until my keyboard breaks.

nkohari 17y ago

I'm not disagreeing, I'm just saying that before people get drunk on schadenfreude they should put themselves in Atwood's position. If you pay someone for a service you generally expect it to work when you need it.

mcantelon 17y ago
Yeah, but if you pay for a mission critical service and never bother testing it, you've pretty much passively decided to fail.
- pyre 17y ago
  The same could be said for CrystalTech
  - mcantelon 17y ago
    I don't think anyone's saying that CrystalTech didn't also fail.
- krschultz 17y ago
  Is codinghorror really mission critical? He will restore from some other backups. Downtime on a blog is not that big a deal other than people like to hate on Jeff Atwood. Now if he lost customer credit card records or something like that, then it would be warranted.
  - houseabsolute 17y ago
    Doesn't Jeff derive a significant portion of his personal income from the site? If so, I'd say mission critical is a good way to describe it.
  - dinkumthinkum 17y ago
    If codinghorror somehow allows Jeff to live without working then I would say it is pretty mission critical. Mission critical doesn't mean the Earth will blow up or something like that.

Confusion 17y ago

Do you also test that your airbags work properly? I don't audit and test the safety measures on most of the equipment in my house: that's what I pay the supplier for and there'll be hell to pay when they screw it up. Similarly, I don't audit and test my disaster recovery procedures, because that is what I pay my hosting partner for.

klodolph 17y ago
"Reasoning by analogy" The people who made your car were legally required to test your model of car by crashing it into another one and making sure the airbags work. This is also too expensive to test yourself. One does not modify an airbag system at all without retesting it. Your hosting provider has no legal requirement to test their backup system. At best, they have a contractual obligation. And they don't know how to test recovering your site, because testing whether it works is a different process for each site. Additionally, it's cheap to test backup procedures. Most people have a spare computer somewhere (maybe not in the data center), and it should only take a few hours to restore a copy of your site. Once a year. For Chris's sake, you could probably do it in the background and put a movie on.
nomoresecrets 17y ago
When you test airbags, you have to then replace them. This is possibly the worst car analogy ever :-)