8 ms·

Let's Stop Talking About "Backups"

83 points by JoelSutherland 17y ago

aw3c2 17y ago
Pointless write-up about linguistics. He says "restore" is the important thing, not the "backup". Well, duh.
- itgoon 17y ago
  Not really. That's actually one of my favorite "health check" questions: when was the last time you restored? Most places have very reliable backup procedures. Most of those have very poor restore procedures - I'd say about half fail when put to the test.
  - camccann 17y ago
    Running a restore is also a very effective way to discover that your backup procedure wasn't actually very reliable after all.
  - aw3c2 17y ago
    To me that is like saying "You can put money on your bank account but you should make sure you get it back". Of course you have to test your backups.
- michael_dorfman 17y ago
  If you think this post is about linguistics, you're missing the point Since the restore is the important thing, that's the one you have to test. And if you haven't tested restoring, your backup is (quite possibly) worthless.
- flogic 17y ago
  It's a well duh, but still most people fail to test restore. This may be the greatest thing about distributed version control. Every clone is a restore. A limited restore but still a restore.
gaius 17y ago
It's not us you need to tell Joel, it's your business partner. And if this is your way of telling him, isn't it a little passive-aggressive to do it in a blog post?
- mechanical_fish 17y ago
  Unless I've missed further developments in the story, his business partner trusted the "backups" of the hosting company. Which did not properly restore. So this is aimed one step farther up the chain. Or, rather, it's aimed at all of us. It is a lesson that a lot of people need to learn.
  - andreyf 17y ago
    Well, Jeff was asking his host "do you back up?", not "can you restore?". That's what the post is about. (Reading comprehension FTW!)
- jseifer 17y ago
  I think this was directed more at his business partners hosting company.
  - gaius 17y ago
    Who chose them and what due diligence was performed? This is business 101.
    - mahmud 17y ago
      Give me a break! (Hosting) companies will throw more references and specs at you than you know what to do with, and you have no choice but to take them for their word. The only thing you can do on your part is keep your own backups, independent of them.
      - axod 17y ago
        Doing your own backups isn't rocket science. Why you would outsource such a valuable thing is beyond me.
jfoutz 17y ago
backups are for suckers. keep the data on a few different spinning disks. if you can solve data synch between two sites, just keep your data synched. it's much better to ask yourself how long to replicate your existing system then how to back up. pxe boot to a kernel that you can install over the network with, bcfg2 to get the thing up to spec, start copying data. a lot of machines can be back and configured in 5 minutes. that said, i'm not you. i don't have terabytes of data to do statistics on. maybe there are other horrible details i'm forgetting. fast rebuilding is a pretty awesome strategy for a lot of cases.
- gaius 17y ago
  if you can solve data synch between two sites, just keep your data synched. LOL! And if you get a corrupt block on your primary site, what're you going to do? All your standbys are instantly tainted! Better leave this one to the grownups.
- spolsky 17y ago
  What if you hit rm -rf * and that gets synched? I don't mean to be rude, I don't know anything about you, but if you were a system administrator working for me, today would be your last day.
  - jawngee 17y ago
    Of course it would be his last day, he's a sysadmin and he just typed rm -rf * on /
    - mechanical_fish 17y ago
      You know, you tell yourself that you understand rm -rf. And yet somehow you always find a way to type it on something you shouldn't. Computers are perverse objects. It's like reading an airline crash report, which I'm told often ends up sounding like a comedy of errors. Most airline crashes have an entire handful of causes, all of which are individually innocent, but on the one extremely rare occasion when they all happen at the same time they add up to disaster.
hexis 17y ago
I tend to be a little paranoid about backups, so I have a few different disks backing up my main, desktop, machine. But I also use one of my backups to sync data to my laptop, not quite a full "restore" due to a big size difference in the respective drives. But, generally speaking, the two machines are in sync and I can be sure that at least one of my backups works reasonably well.
orblivion 17y ago
As if I didn't lose enough sleep pondering backups already.
mark_l_watson 17y ago
Common sense and obvious. In the 1980s I worked on a large DARPA project where a huge hit was taken because our admins never tried to restore from backups. It is the kind of lesson that is (hopefully) learned with just one bad experience. This is another reason why I like EC2 deployments: it is fairly easy to take your backups (automated deployment scripts, application, data) and spin up another copy of your whole system (except for flipping the DNS). Make sure those EBS-backed EC2 AMIs are really bootable and functioning :-)
- DenisM 17y ago
  What happens if Amazon runs out of machines, has a system-wide corruption that was noticed too late or decides to be dicks about something or other?
  - mark_l_watson 17y ago
    Good questions. I would think that if they are making money with AWS then they will keep buying more servers. I usually trust S3 for backups and restores but periodically back my own data off of AWS to local storage. I expect both Amazon and Google infrastructure services to experience outages from time to time. However, they have far more resources and expertise than I do to provide scalable services for a low cost.
    - DenisM 17y ago
      I was thinking about making a set up where my S3 backups are automatically mirrored to rackspace or something like that. That would be a really neat setup.
wglb 17y ago
This is good food for thought. Let's also add to this a concept from a different realm: Everybody has at least two dns servers listed in their /etc/resolv.conf, right? The reason is that in case one of them goes down, there is the other one. So this seems like a good lesson to take about backups. Mebbe three? One by your hosting provider, one at tarsnap, one on a separate dat tape, one on a usb stick? A good point is though that even something as big as a dat tape looks pretty small by the standards of what we need to back up today.
lazyant 17y ago
Testing your (off-site) backups is an obvious first item in many lists http://watsec.com/article/49 http://watsec.com/article/49
jsz0 17y ago
Full disk image backups are a good solution for this problem. No worries about partial backups or a complex restoration process. It's totally inefficient but storage is cheaper than man hours.
- peterwwillis 17y ago
  I'm assuming you're talking about some kind of atomic file-or-block-level backup such as LVM snapshots? Large files such as databases can change while reading them over a long period of time, so a standard disk image or file copy wouldn't be reliable for a live system.
  - ryanpetrich 17y ago
    zfs send/receive is amazing. I wish other filesystems had it
DenisM 17y ago
I find it disturbing how many people decided to comment or up-vote on Joel-Atwood angle, or insult each other. Remember: great people discuss ideas, normal people discuss events, shallow people discuss other people.
DannoHung 17y ago
This whole ordeal is getting me motivated to actually buy a cloud backup service (personal use, not business use). I was thinking of Carbonite or Backblaze. Anyone have any experience with those?
- slig 17y ago
  I used carbonite in early 2006 and the software was horrible. Eventually, I tried to uninstall and the process failed. I got a half installation that wouldn't work and couldn't be removed. I didn't try too hard after that, because I was planning to format and start over.
ghshephard 17y ago
What I find someone charming/quaint about Joel's Posts on "Operations" over the last six years that I've been reading him, is that he is, slowly but surely, discovering the "Art of Operations" - albeit at a glacial pace. Most People who work in Production Operations environments of any scale, discover what has taken Joel the better part of a decade, in the first two-three years of their career. I almost feel like that Airplane passenger sitting beside Brooks Jr - Brooks saw him reading his book, "Mythical Man Month" - and asked the guy (who had no idea who he was sitting beside) what he thought of the book - The gentleman responded that it was basically a summary of things he knew already. Joel is a giant in the industry, but he does have a tendency to discover/restate the obvious. "It's not backups, but the restores that matter" - is kind of the mantra of every single person who has ever been responsible for backups. Then, you go to _any_ class on running a production environment, and you discover things like RPO, RTO, Dress-Rehearsals, etc.. and the whole "It's restores that matter" begins to look quaint.
- lifeisstillgood 17y ago
  the "Art of Operations" suggests a book or similar that you are referring to - is there one, or am I reading too much into some double quotes?
  - josephkern 17y ago
    Yes there is a book: The Practice of System and Network Administration, Second Edition. This is the best Operations book I've ever bought. Worth every penny.
    - vdm 17y ago
      Because Google isn't links: http://www.amazon.com/Practice-System-Network-Administration-Second/dp/0321492668/ http://www.amazon.com/Practice-System-Network-Administration...
- josephkern 17y ago
  Totally agree on these points. The Joel-Atwood experience is not that of two programmers starting a company. It's a story of two programmers learning about System Administration.
Goladus 17y ago
It's helpful to boost signal for this message. It's an old message, but planning and practicing system restores can be as expensive in terms of equipment and manpower as actually making the backups. This leads to a lot of neglect.
Sukotto 17y ago
Also this, similar take on the horror of backups gone wrong: http://www.penny-arcade.com/comic/2005/8/10/ http://www.penny-arcade.com/comic/2005/8/10/ Don't blindly rely on your partner to do it... Trust, but verify.
loupgarou21 17y ago
... shouldn't it be common practice to test your backup system to make sure that the restore procedure meets the requirements of the client (company, etc?) The IT company that I work for creates a backup system based on the requirements of our clients and then demonstrates the whole backup and restore procedure to make sure that it falls in line with what our client actually wants. It's really not difficult to do. Sure, some of the restore procedures may be slower (depending on other requirements, such as cost,) but the client knows that will be the case and signs off on it.
dnsworks 17y ago
What he's saying is, "We failed miserably at having good process and procedures. Because of this we are going to lecture others, and point the blame at everybody but ourselves, in hopes that they'll stop pointing out how much credibility we lost over this."

aw3c2 17y ago

Pointless write-up about linguistics. He says "restore" is the important thing, not the "backup". Well, duh.

itgoon 17y ago
Not really. That's actually one of my favorite "health check" questions: when was the last time you restored? Most places have very reliable backup procedures. Most of those have very poor restore procedures - I'd say about half fail when put to the test.
- camccann 17y ago
  Running a restore is also a very effective way to discover that your backup procedure wasn't actually very reliable after all.
- aw3c2 17y ago
  To me that is like saying "You can put money on your bank account but you should make sure you get it back". Of course you have to test your backups.
michael_dorfman 17y ago
If you think this post is about linguistics, you're missing the point Since the restore is the important thing, that's the one you have to test. And if you haven't tested restoring, your backup is (quite possibly) worthless.
flogic 17y ago
It's a well duh, but still most people fail to test restore. This may be the greatest thing about distributed version control. Every clone is a restore. A limited restore but still a restore.

gaius 17y ago

It's not us you need to tell Joel, it's your business partner. And if this is your way of telling him, isn't it a little passive-aggressive to do it in a blog post?

mechanical_fish 17y ago
Unless I've missed further developments in the story, his business partner trusted the "backups" of the hosting company. Which did not properly restore. So this is aimed one step farther up the chain. Or, rather, it's aimed at all of us. It is a lesson that a lot of people need to learn.
- andreyf 17y ago
  Well, Jeff was asking his host "do you back up?", not "can you restore?". That's what the post is about. (Reading comprehension FTW!)
jseifer 17y ago
I think this was directed more at his business partners hosting company.
- gaius 17y ago
  Who chose them and what due diligence was performed? This is business 101.
  - mahmud 17y ago
    Give me a break! (Hosting) companies will throw more references and specs at you than you know what to do with, and you have no choice but to take them for their word. The only thing you can do on your part is keep your own backups, independent of them.
    - axod 17y ago
      Doing your own backups isn't rocket science. Why you would outsource such a valuable thing is beyond me.

jfoutz 17y ago

backups are for suckers. keep the data on a few different spinning disks. if you can solve data synch between two sites, just keep your data synched. it's much better to ask yourself how long to replicate your existing system then how to back up. pxe boot to a kernel that you can install over the network with, bcfg2 to get the thing up to spec, start copying data. a lot of machines can be back and configured in 5 minutes. that said, i'm not you. i don't have terabytes of data to do statistics on. maybe there are other horrible details i'm forgetting. fast rebuilding is a pretty awesome strategy for a lot of cases.

gaius 17y ago
if you can solve data synch between two sites, just keep your data synched. LOL! And if you get a corrupt block on your primary site, what're you going to do? All your standbys are instantly tainted! Better leave this one to the grownups.
spolsky 17y ago
What if you hit rm -rf * and that gets synched? I don't mean to be rude, I don't know anything about you, but if you were a system administrator working for me, today would be your last day.
- jawngee 17y ago
  Of course it would be his last day, he's a sysadmin and he just typed rm -rf * on /
  - mechanical_fish 17y ago
    You know, you tell yourself that you understand rm -rf. And yet somehow you always find a way to type it on something you shouldn't. Computers are perverse objects. It's like reading an airline crash report, which I'm told often ends up sounding like a comedy of errors. Most airline crashes have an entire handful of causes, all of which are individually innocent, but on the one extremely rare occasion when they all happen at the same time they add up to disaster.

hexis 17y ago

I tend to be a little paranoid about backups, so I have a few different disks backing up my main, desktop, machine. But I also use one of my backups to sync data to my laptop, not quite a full "restore" due to a big size difference in the respective drives. But, generally speaking, the two machines are in sync and I can be sure that at least one of my backups works reasonably well.

orblivion 17y ago

As if I didn't lose enough sleep pondering backups already.

mark_l_watson 17y ago

Common sense and obvious. In the 1980s I worked on a large DARPA project where a huge hit was taken because our admins never tried to restore from backups. It is the kind of lesson that is (hopefully) learned with just one bad experience. This is another reason why I like EC2 deployments: it is fairly easy to take your backups (automated deployment scripts, application, data) and spin up another copy of your whole system (except for flipping the DNS). Make sure those EBS-backed EC2 AMIs are really bootable and functioning :-)

DenisM 17y ago
What happens if Amazon runs out of machines, has a system-wide corruption that was noticed too late or decides to be dicks about something or other?
- mark_l_watson 17y ago
  Good questions. I would think that if they are making money with AWS then they will keep buying more servers. I usually trust S3 for backups and restores but periodically back my own data off of AWS to local storage. I expect both Amazon and Google infrastructure services to experience outages from time to time. However, they have far more resources and expertise than I do to provide scalable services for a low cost.
  - DenisM 17y ago
    I was thinking about making a set up where my S3 backups are automatically mirrored to rackspace or something like that. That would be a really neat setup.

wglb 17y ago

This is good food for thought. Let's also add to this a concept from a different realm: Everybody has at least two dns servers listed in their /etc/resolv.conf, right? The reason is that in case one of them goes down, there is the other one. So this seems like a good lesson to take about backups. Mebbe three? One by your hosting provider, one at tarsnap, one on a separate dat tape, one on a usb stick? A good point is though that even something as big as a dat tape looks pretty small by the standards of what we need to back up today.

lazyant 17y ago

Testing your (off-site) backups is an obvious first item in many lists http://watsec.com/article/49 http://watsec.com/article/49

jsz0 17y ago

Full disk image backups are a good solution for this problem. No worries about partial backups or a complex restoration process. It's totally inefficient but storage is cheaper than man hours.

peterwwillis 17y ago
I'm assuming you're talking about some kind of atomic file-or-block-level backup such as LVM snapshots? Large files such as databases can change while reading them over a long period of time, so a standard disk image or file copy wouldn't be reliable for a live system.
- ryanpetrich 17y ago
  zfs send/receive is amazing. I wish other filesystems had it

DenisM 17y ago

I find it disturbing how many people decided to comment or up-vote on Joel-Atwood angle, or insult each other. Remember: great people discuss ideas, normal people discuss events, shallow people discuss other people.

DannoHung 17y ago

This whole ordeal is getting me motivated to actually buy a cloud backup service (personal use, not business use). I was thinking of Carbonite or Backblaze. Anyone have any experience with those?

slig 17y ago
I used carbonite in early 2006 and the software was horrible. Eventually, I tried to uninstall and the process failed. I got a half installation that wouldn't work and couldn't be removed. I didn't try too hard after that, because I was planning to format and start over.

ghshephard 17y ago

What I find someone charming/quaint about Joel's Posts on "Operations" over the last six years that I've been reading him, is that he is, slowly but surely, discovering the "Art of Operations" - albeit at a glacial pace. Most People who work in Production Operations environments of any scale, discover what has taken Joel the better part of a decade, in the first two-three years of their career. I almost feel like that Airplane passenger sitting beside Brooks Jr - Brooks saw him reading his book, "Mythical Man Month" - and asked the guy (who had no idea who he was sitting beside) what he thought of the book - The gentleman responded that it was basically a summary of things he knew already. Joel is a giant in the industry, but he does have a tendency to discover/restate the obvious. "It's not backups, but the restores that matter" - is kind of the mantra of every single person who has ever been responsible for backups. Then, you go to _any_ class on running a production environment, and you discover things like RPO, RTO, Dress-Rehearsals, etc.. and the whole "It's restores that matter" begins to look quaint.

lifeisstillgood 17y ago
the "Art of Operations" suggests a book or similar that you are referring to - is there one, or am I reading too much into some double quotes?
- josephkern 17y ago
  Yes there is a book: The Practice of System and Network Administration, Second Edition. This is the best Operations book I've ever bought. Worth every penny.
  - vdm 17y ago
    Because Google isn't links: http://www.amazon.com/Practice-System-Network-Administration-Second/dp/0321492668/ http://www.amazon.com/Practice-System-Network-Administration...
josephkern 17y ago
Totally agree on these points. The Joel-Atwood experience is not that of two programmers starting a company. It's a story of two programmers learning about System Administration.

Goladus 17y ago

It's helpful to boost signal for this message. It's an old message, but planning and practicing system restores can be as expensive in terms of equipment and manpower as actually making the backups. This leads to a lot of neglect.

Sukotto 17y ago

Also this, similar take on the horror of backups gone wrong: http://www.penny-arcade.com/comic/2005/8/10/ http://www.penny-arcade.com/comic/2005/8/10/ Don't blindly rely on your partner to do it... Trust, but verify.

loupgarou21 17y ago

... shouldn't it be common practice to test your backup system to make sure that the restore procedure meets the requirements of the client (company, etc?) The IT company that I work for creates a backup system based on the requirements of our clients and then demonstrates the whole backup and restore procedure to make sure that it falls in line with what our client actually wants. It's really not difficult to do. Sure, some of the restore procedures may be slower (depending on other requirements, such as cost,) but the client knows that will be the case and signs off on it.

dnsworks 17y ago

What he's saying is, "We failed miserably at having good process and procedures. Because of this we are going to lecture others, and point the blame at everybody but ourselves, in hopes that they'll stop pointing out how much credibility we lost over this."