4 ms·

18 years ago, while being a teenager, I misconfigured CCC only to find out that only the active user folders where synced after a full erase/reinstall upgrade o

by Twisell 10y ago

18 years ago, while being a teenager, I misconfigured CCC only to find out that only the active user folders where synced after a full erase/reinstall upgrade of MacOS. This was my epic "there is to kind of people those who lost data and those who will loose data" story. All my dad and mum files where gone, luckily nothing professional and no pictures (analog camera still ruled back then). This is just a reminder that no tool is ever perfect. CCC is great, TimeMachine is sluggish but "dumbproof" to a certain extend (but you have to keep faith in a black box). Personally I'm pretty paranoid so I perform my monthly off-site backup by rebooting my Mac on recovery partition and I perform a full disc copy to an external drive using DiskUtils (because ultimately this will be the recovery tools I'll use if things goes bad so I need to use my off-site backup...). It really sluggish so I run it while sleeping. I get that for professionals with more than 1To you will need to get to some more serious stuff anyway.

laumars 10y ago
I appreciate this was a long time ago and you've learned a lot since, but your story perfectly illustrates the importance of not just performing backups, but periodically testing them as well. That way you eliminate your blind faith in a black box by ensuring you actually have a working solution to fallback on.
- Twisell 10y ago
  I my case (and I can't emphasis more on my teenager status back then) it was a permission issue, everything looked fine from my perspective, but CCC somewhat didn't have the right permissions to sync folder of others users. So their home folder where actually empty. From my perspective the backup was a success... This traumatizing error is when I first learned about superuser and permissions. I guess CCC has gone a long way since then and added more safeguard. But I must confess that I've sticked to a "use the goddam standard tools" since then. Sure there must be a lot of more customizable or faster tools out there. But Time Machine stay "the backup tool my mum can't use wrong" I can't give enough to credit to Apple for that. And putting aside NSA/FBI problem, iCloud backup for iPhone is exactly as smooth. Literally every time I go to a Genius Bar, someone is about to hug a genius because of a successful restore from the iCloud of their broken/stolen iPhone. This is how you build a "faithful" customer base and no other tech company understand that better than Apple so far.
  - laumars 10y ago
    Those points don't change the fact that you (and everyone else) should test back ups periodically. If it's not file system permissions nor other configuration problems then it will be a failing storage medium or just dumb user error. So the only way to be sure you have a working back up is to test that back up - ideally before you actually need to use it.
    - Twisell 10y ago
      To be clear I can't agree more! I got a little lost while writing. But indeed my point was double check! Because in my case everything looked fine at first sight. This is even why I don't use the same process for my two backup (daily/monthly). Because to be fair you can't constantly check that your daily/hourly backup isn't corrupted, you rely on its build-in safeguard and occasionally you check it. But by using two different methods for daily vs monthly(offsite) backup, you significantly reduce the odds that the two different methods failed at the same time while you need them. (Also a monthly full clone is way easy to check than a Time Machine Disk)
    - PhantomGremlin 10y ago
      Yeah, once upon a time at a startup, I wanted a file restored. We were small; the CEO was actually the one doing the backups (to QIC tape cartridges). Of course, the recovery failed. In fact, the backups hadn't been working properly for months. I lost a few hours of work; if our file server had actually failed, not having a good backup could have literally been catastrophic. As in would the company have survived?! I recall an ancient quip, more or less: "if you don't test your backups, you don't have backups, you have dreams".
- bemmu 10y ago
  I keep hearing this, but as a Mac user who uses Time Machine and CCC, what's a good way to test the backups? After all deleting my only system to attempt a restore seems even more dangerous.
  - PhantomGremlin 10y ago
    what's a good way to test the backups? Here's a way to do it. You need to feel comfortable around the OS X terminal command line. You only rely on OS X's built in programs, so it's an independent way to check if your backup program is doing the right thing. As root, I used to do something like this: cd / find -x \ . \ -type f -print0 \ | xargs -0 -n 100 -x md5 \ | sort > /tmp/src.md5 Then I would run the same type of script in the destination directory and diff the results. There are some annoyances. E.g. (from memory) the ~/Library/Caches files aren't backed up, so they will be missing at the destination. I wrote some sed commands to first remove some of these from the result checksums to keep my diff's more manageable. Instead of the above, I currently use a Python script that I wrote, that let me fine tune things. At the heart of it is (as root, of course) using Python's os.walk to traverse a directory tree. For each file I use hashlib.sha256 to generate a checksum. I also don't descend into certain directories. Etc. Using a few Python scripts to generate and process the source and destination checksums allows me to, at the end, use a simple vimdiff source.checksums destination.checksums to quickly see the few differences that remain. Keep in mind that the perfect is the enemy of the "good enough". Just using the basic 'find' (and not a custom Python script) is plenty good. I used that method for many years. Many errors stick out right away. E.g. if you have 1,000,000 files checksummed in your source but only 250,000 in your destination, then you quickly know that you screwed up.
  - crottypeter 10y ago
    Buy a new hard drive and restore to that?

by Twisell 10y ago

laumars 10y ago

I appreciate this was a long time ago and you've learned a lot since, but your story perfectly illustrates the importance of not just performing backups, but periodically testing them as well. That way you eliminate your blind faith in a black box by ensuring you actually have a working solution to fallback on.

Twisell 10y ago
I my case (and I can't emphasis more on my teenager status back then) it was a permission issue, everything looked fine from my perspective, but CCC somewhat didn't have the right permissions to sync folder of others users. So their home folder where actually empty. From my perspective the backup was a success... This traumatizing error is when I first learned about superuser and permissions. I guess CCC has gone a long way since then and added more safeguard. But I must confess that I've sticked to a "use the goddam standard tools" since then. Sure there must be a lot of more customizable or faster tools out there. But Time Machine stay "the backup tool my mum can't use wrong" I can't give enough to credit to Apple for that. And putting aside NSA/FBI problem, iCloud backup for iPhone is exactly as smooth. Literally every time I go to a Genius Bar, someone is about to hug a genius because of a successful restore from the iCloud of their broken/stolen iPhone. This is how you build a "faithful" customer base and no other tech company understand that better than Apple so far.
- laumars 10y ago
  Those points don't change the fact that you (and everyone else) should test back ups periodically. If it's not file system permissions nor other configuration problems then it will be a failing storage medium or just dumb user error. So the only way to be sure you have a working back up is to test that back up - ideally before you actually need to use it.
  - Twisell 10y ago
    To be clear I can't agree more! I got a little lost while writing. But indeed my point was double check! Because in my case everything looked fine at first sight. This is even why I don't use the same process for my two backup (daily/monthly). Because to be fair you can't constantly check that your daily/hourly backup isn't corrupted, you rely on its build-in safeguard and occasionally you check it. But by using two different methods for daily vs monthly(offsite) backup, you significantly reduce the odds that the two different methods failed at the same time while you need them. (Also a monthly full clone is way easy to check than a Time Machine Disk)
  - PhantomGremlin 10y ago
    Yeah, once upon a time at a startup, I wanted a file restored. We were small; the CEO was actually the one doing the backups (to QIC tape cartridges). Of course, the recovery failed. In fact, the backups hadn't been working properly for months. I lost a few hours of work; if our file server had actually failed, not having a good backup could have literally been catastrophic. As in would the company have survived?! I recall an ancient quip, more or less: "if you don't test your backups, you don't have backups, you have dreams".
bemmu 10y ago
I keep hearing this, but as a Mac user who uses Time Machine and CCC, what's a good way to test the backups? After all deleting my only system to attempt a restore seems even more dangerous.
- PhantomGremlin 10y ago
  what's a good way to test the backups? Here's a way to do it. You need to feel comfortable around the OS X terminal command line. You only rely on OS X's built in programs, so it's an independent way to check if your backup program is doing the right thing. As root, I used to do something like this: cd / find -x \ . \ -type f -print0 \ | xargs -0 -n 100 -x md5 \ | sort > /tmp/src.md5 Then I would run the same type of script in the destination directory and diff the results. There are some annoyances. E.g. (from memory) the ~/Library/Caches files aren't backed up, so they will be missing at the destination. I wrote some sed commands to first remove some of these from the result checksums to keep my diff's more manageable. Instead of the above, I currently use a Python script that I wrote, that let me fine tune things. At the heart of it is (as root, of course) using Python's os.walk to traverse a directory tree. For each file I use hashlib.sha256 to generate a checksum. I also don't descend into certain directories. Etc. Using a few Python scripts to generate and process the source and destination checksums allows me to, at the end, use a simple vimdiff source.checksums destination.checksums to quickly see the few differences that remain. Keep in mind that the perfect is the enemy of the "good enough". Just using the basic 'find' (and not a custom Python script) is plenty good. I used that method for many years. Many errors stick out right away. E.g. if you have 1,000,000 files checksummed in your source but only 250,000 in your destination, then you quickly know that you screwed up.
- crottypeter 10y ago
  Buy a new hard drive and restore to that?