4 ms·

backups are for suckers. keep the data on a few different spinning disks. if you can solve data synch between two sites, just keep your data synched. it's much

by jfoutz 17y ago

backups are for suckers. keep the data on a few different spinning disks. if you can solve data synch between two sites, just keep your data synched. it's much better to ask yourself how long to replicate your existing system then how to back up. pxe boot to a kernel that you can install over the network with, bcfg2 to get the thing up to spec, start copying data. a lot of machines can be back and configured in 5 minutes. that said, i'm not you. i don't have terabytes of data to do statistics on. maybe there are other horrible details i'm forgetting. fast rebuilding is a pretty awesome strategy for a lot of cases.

gaius 17y ago
if you can solve data synch between two sites, just keep your data synched. LOL! And if you get a corrupt block on your primary site, what're you going to do? All your standbys are instantly tainted! Better leave this one to the grownups.
spolsky 17y ago
What if you hit rm -rf * and that gets synched? I don't mean to be rude, I don't know anything about you, but if you were a system administrator working for me, today would be your last day.
- jawngee 17y ago
  Of course it would be his last day, he's a sysadmin and he just typed rm -rf * on /
  - mechanical_fish 17y ago
    You know, you tell yourself that you understand rm -rf. And yet somehow you always find a way to type it on something you shouldn't. Computers are perverse objects. It's like reading an airline crash report, which I'm told often ends up sounding like a comedy of errors. Most airline crashes have an entire handful of causes, all of which are individually innocent, but on the one extremely rare occasion when they all happen at the same time they add up to disaster.
    - ajross 17y ago
      I fat-fingered the destination directory in a script once and accidentally shrunk my entire photo archive to 128x128 thumbnails. Thankfully I did have backups of all but a handful. I'm always amazed at the number of people who spend their time thinking about replication strategies without understanding that data can be accidentally deleted in production too. I guess backups look "easy", so it's not as sexy an area of architecture planning.
      - pyre 17y ago
        People just don't always grasp the big picture. People that see RAID as a backup strategy are only looking at 'hard drive failure' as potential scenario. The same for machine replication. They are only trying to prevent against what happens if 'the machine dies.' As long as they realize that this doesn't protect against, "I accidentally deleted the files," then it's all good.
        mechanical_fish 17y ago
        In a previous job I had a so-called system administrator laugh at my paranoia when I suggested that his awesome backup system -- protecting the equivalent of several dozen person-years of scientific data, millions of dollars' worth -- wouldn't protect us against a fire in the building. Or, for that matter, a thief who liked to steal computer hardware. Job security note for sysadmins: When someone suggests a disaster scenario, don't open your response with a laugh.
        pyre 17y ago
        Well, you know that you're talking to someone that doesn't plan for the future when, "What happens if the building burns down?" is responded to with, "You're just being paranoid." I guess the appropriate response would be to point out that using his logic tons of taxpayer money could be saved by nixing the fire departments (and tons of corporate money could be saved by not paying for fire protection -- alarms, detectors, escapes, etc).
mechanical_fish 17y ago
maybe there are other horrible details i'm forgetting Yes. Why should I bother to write this? I'll outsource the task to the authors of High Performance MySQL, Second Edition, page 475: Backup Myth #1: "I Use Replication As a Backup" This is a mistake we see quite often. A replication slave is not a backup. Neither is a RAID array. To see why, consider this: will they help you get back all your data if you accidentally execute DROP DATABASE on your production database? RAID and replication don't pass even this simple test. Not only are they not backups, they're not a substitute for backups. Nothing but backups fill the need for backups.
- gvb 17y ago
  Obligatory xkcd link: "Bobby Tables" http://xkcd.com/327/ http://xkcd.com/327/
- jfoutz 17y ago
  Completely reasonable response, forgive me for being an inarticulate noob in my post. I thought delayed replication was one of the main strategies they advocated in that book. I don't have it on hand. my mistake.
- diego 17y ago
  This is the incremental backup script I use on my Linux box at home, a quick-and-dirty imitation of what Time Machine does. Obviously $HOME/backup is a different physical disk. Feel free to improve on this. -------- #!/bin/bash HOME= date=`date "+%Y-%m-%dT%H:%M:%S"` rsync -aP --link-dest=$HOME/backup/current /home $HOME/backup/back-$date rsync -aP --link-dest=$HOME/backup/current /etc $HOME/backup/back-$date rm $HOME/backup/current ln -s back-$date $HOME/backup/current #see if the disk is getting full FREE=`df -lk|grep sdb1|awk -F" " '{print $5}'|awk -F"%" '{print $1}'` #alert me if the backup disk is getting full. T=80 if [ "$FREE" -gt "$T" ] then df -lk| mail $myaddress -s"disk alert $T% capacity" else echo "backup disk is less than $T full" fi
  - Locke1689 17y ago
    You should think about keeping an offsite backup if possible. A second disk won't necessarily help if your computer gets dropped in a pool, house catches on fire, etc.
idlewords 17y ago
Replication doesn't defend you against deleting the wrong file, or messing up an update to your database, or any of a large class of PEBKAC issues. It's also not useful if your main files get corrupted and you diligently propagate the corruption. See: ma.gnolia
- jfoutz 17y ago
  Setting aside the database, file problems are fairly easily solved via svn, .snapshot or a good configuration engine. As far as the database goes, I'm big on stored procs + archive tables, but i'll leave that to the grown ups ;)
  - masomenos 17y ago
    ...until you forget to check in important changes, or neglect to add a seemingly trivial configuration file to your SCM. Whereas, a policy of automatically backing up everything except for your exclude list would have saved your bacon.
apowell 17y ago
A real-time mirror won't help if your data gets corrupted (via a fat-fingered shell command, errant script, or a compromised system). When it's appropriate, it's awesome, but I don't think it's a replacement for a static "offline" backup,
jws 17y ago
Replication is not backup. It only protects against hardware failure. Spinning disks are good though. A lovely spot for backups. Just put them in a different building. Edit: In the time it took me to write this 5 other people also lambasted this poor fellow. Ouch.
- InclinedPlane 17y ago
  It doesn't matter if your data is replicated across 3 different continents. If all of your data is accessible behind the same admin login, or if it is all vulnerable to the corruption in the original, then it still has only 1 point of failure. A proper "spinning" backup is an immutable journaling system that can't be trimmed except via extremely secure methods (e.g. physical access to the machine).
InclinedPlane 17y ago
Sorry, this is not only WRONG but actually HARMFUL advice. Redundancy is not a backup. If someone with full admin control to your system can destroy all of your data then you do not have backups. A proper backup is physically separate from your primary data and, preferably, can't be destroyed with mere admin access to the system. The number of sites that have had catastrophic data loss due to relying on mirroring instead of true backups is quite significant.

by jfoutz 17y ago

gaius 17y ago

if you can solve data synch between two sites, just keep your data synched. LOL! And if you get a corrupt block on your primary site, what're you going to do? All your standbys are instantly tainted! Better leave this one to the grownups.

spolsky 17y ago

What if you hit rm -rf * and that gets synched? I don't mean to be rude, I don't know anything about you, but if you were a system administrator working for me, today would be your last day.

jawngee 17y ago
Of course it would be his last day, he's a sysadmin and he just typed rm -rf * on /
- mechanical_fish 17y ago
  You know, you tell yourself that you understand rm -rf. And yet somehow you always find a way to type it on something you shouldn't. Computers are perverse objects. It's like reading an airline crash report, which I'm told often ends up sounding like a comedy of errors. Most airline crashes have an entire handful of causes, all of which are individually innocent, but on the one extremely rare occasion when they all happen at the same time they add up to disaster.
  - ajross 17y ago
    I fat-fingered the destination directory in a script once and accidentally shrunk my entire photo archive to 128x128 thumbnails. Thankfully I did have backups of all but a handful. I'm always amazed at the number of people who spend their time thinking about replication strategies without understanding that data can be accidentally deleted in production too. I guess backups look "easy", so it's not as sexy an area of architecture planning.
    - pyre 17y ago
      People just don't always grasp the big picture. People that see RAID as a backup strategy are only looking at 'hard drive failure' as potential scenario. The same for machine replication. They are only trying to prevent against what happens if 'the machine dies.' As long as they realize that this doesn't protect against, "I accidentally deleted the files," then it's all good.
      - mechanical_fish 17y ago
        In a previous job I had a so-called system administrator laugh at my paranoia when I suggested that his awesome backup system -- protecting the equivalent of several dozen person-years of scientific data, millions of dollars' worth -- wouldn't protect us against a fire in the building. Or, for that matter, a thief who liked to steal computer hardware. Job security note for sysadmins: When someone suggests a disaster scenario, don't open your response with a laugh.
        pyre 17y ago
        Well, you know that you're talking to someone that doesn't plan for the future when, "What happens if the building burns down?" is responded to with, "You're just being paranoid." I guess the appropriate response would be to point out that using his logic tons of taxpayer money could be saved by nixing the fire departments (and tons of corporate money could be saved by not paying for fire protection -- alarms, detectors, escapes, etc).

mechanical_fish 17y ago

maybe there are other horrible details i'm forgetting Yes. Why should I bother to write this? I'll outsource the task to the authors of High Performance MySQL, Second Edition, page 475: Backup Myth #1: "I Use Replication As a Backup" This is a mistake we see quite often. A replication slave is not a backup. Neither is a RAID array. To see why, consider this: will they help you get back all your data if you accidentally execute DROP DATABASE on your production database? RAID and replication don't pass even this simple test. Not only are they not backups, they're not a substitute for backups. Nothing but backups fill the need for backups.

gvb 17y ago
Obligatory xkcd link: "Bobby Tables" http://xkcd.com/327/ http://xkcd.com/327/
jfoutz 17y ago
Completely reasonable response, forgive me for being an inarticulate noob in my post. I thought delayed replication was one of the main strategies they advocated in that book. I don't have it on hand. my mistake.
diego 17y ago
This is the incremental backup script I use on my Linux box at home, a quick-and-dirty imitation of what Time Machine does. Obviously $HOME/backup is a different physical disk. Feel free to improve on this. -------- #!/bin/bash HOME= date=`date "+%Y-%m-%dT%H:%M:%S"` rsync -aP --link-dest=$HOME/backup/current /home $HOME/backup/back-$date rsync -aP --link-dest=$HOME/backup/current /etc $HOME/backup/back-$date rm $HOME/backup/current ln -s back-$date $HOME/backup/current #see if the disk is getting full FREE=`df -lk|grep sdb1|awk -F" " '{print $5}'|awk -F"%" '{print $1}'` #alert me if the backup disk is getting full. T=80 if [ "$FREE" -gt "$T" ] then df -lk| mail $myaddress -s"disk alert $T% capacity" else echo "backup disk is less than $T full" fi
- Locke1689 17y ago
  You should think about keeping an offsite backup if possible. A second disk won't necessarily help if your computer gets dropped in a pool, house catches on fire, etc.

idlewords 17y ago

Replication doesn't defend you against deleting the wrong file, or messing up an update to your database, or any of a large class of PEBKAC issues. It's also not useful if your main files get corrupted and you diligently propagate the corruption. See: ma.gnolia

jfoutz 17y ago
Setting aside the database, file problems are fairly easily solved via svn, .snapshot or a good configuration engine. As far as the database goes, I'm big on stored procs + archive tables, but i'll leave that to the grown ups ;)
- masomenos 17y ago
  ...until you forget to check in important changes, or neglect to add a seemingly trivial configuration file to your SCM. Whereas, a policy of automatically backing up everything except for your exclude list would have saved your bacon.

apowell 17y ago

A real-time mirror won't help if your data gets corrupted (via a fat-fingered shell command, errant script, or a compromised system). When it's appropriate, it's awesome, but I don't think it's a replacement for a static "offline" backup,

jws 17y ago

Replication is not backup. It only protects against hardware failure. Spinning disks are good though. A lovely spot for backups. Just put them in a different building. Edit: In the time it took me to write this 5 other people also lambasted this poor fellow. Ouch.

InclinedPlane 17y ago
It doesn't matter if your data is replicated across 3 different continents. If all of your data is accessible behind the same admin login, or if it is all vulnerable to the corruption in the original, then it still has only 1 point of failure. A proper "spinning" backup is an immutable journaling system that can't be trimmed except via extremely secure methods (e.g. physical access to the machine).

InclinedPlane 17y ago

Sorry, this is not only WRONG but actually HARMFUL advice. Redundancy is not a backup. If someone with full admin control to your system can destroy all of your data then you do not have backups. A proper backup is physically separate from your primary data and, preferably, can't be destroyed with mere admin access to the system. The number of sites that have had catastrophic data loss due to relying on mirroring instead of true backups is quite significant.