11 ms·

I lost my data trying to back it up

185 points by h4myio 8y ago

jeffrallen 8y ago
Bummer, dude.
kkarakk 8y ago
why would you try this out with "live" data? why not incrementally connect drives and do it that way? i'm basically asking what the impetus was for "just going for it" which is probably going to be a reason that other people share...
- jacobush 8y ago
  Incrementally?
  - tomschwiha 8y ago
    I assume by incrementally is meant to have a base backup and store the difference in between. Personally I prefer full backups - they feel more complete.
    - mnw21cam 8y ago
      I have a server with 300TB of data. Incremental is the only way that my backups are going to be able to run every night. But also, there are different styles of incremental. Some are dodgier than others.
      - tomschwiha 8y ago
        I agree for this volume of data incremental is the best choice. My projects so far ranged from 1-100GB so a full daily backup is quite quick done and its more easy to recover compared to applying incremental backups individually. Well, always choose the right tool for the task.
        Izkata 8y ago
        > compared to applying incremental backups individually. At least one style of incremental backups can be restored in a single copy - I use rsync to create hardlinks to the previous day if the file is unchanged (based on modified time). This results in around 400M of additional disk usage per day due to changes, for around 90G of data in my home directory.
- evanweaver 8y ago
  This is a good point; he could have physically removed one pair of drives, at least, at which point he would have had an actual backup sitting on the shelf.
mobilemidget 8y ago
(many others including myself, have been there, different way though for me) The bright side is, you'll never make this mistake again. The downside is, for me at least, I end up with triple+ backups of everything..
- h4myio 8y ago
  I can defiantly see myself ending up with triple+ backups of everything...
  - tubbs 8y ago
    Not to be a grammar Nazi, but I believe the word you're looking for is definitely. I wouldn't have normally picked it out, but I did see you use it like that in the article as well. It was a good read! Thanks for taking your time to write it up.
cm2187 8y ago
That sounds like a very complicated way to do a full OS backup. Dead easy, 10 min, can't fail alternative for home projects: use a VM instead, switch off the VM, copy the disk file, switch the VM back on. Use NVMe SSDs and depending on the size, you can probably do an automated daily backup with minimal downtime.
- sokoloff 8y ago
  Anyone have a solution that can snapshot a running VM that has a PCI or PCIe card passed through to the guest OS? I use snapshots on ESXi, but they don't work for guests that have a card passed-through.
  - sn 8y ago
    Are you asking for a recommendation on how to do this with ESXi or a recommendation for an alternative to ESXi?
    - sokoloff 8y ago
      Either is fine. If it's within-ESXi, that's a little easier, but I'm not tightly/inextricably wed to that hypervisor.
bklaasen 8y ago
Did you log a bug against `dmraid`? You're right, it shouldn't make changes to the file system without user intervention.
- h4myio 8y ago
  Not yet. I suppose I should do so via Debian bug report? Cause the last update for the upstream source seems to be around 8 years ago.
arendtio 8y ago
Well, it is not like I didn't do any stupid things in the past, but booting into to a different OS (live system) with such a setup is obviously risky. I am a bit conservative when it comes to backups and I do not trust myself very much. So I am doing backups on the block device level to avoid not copying certain file types, file attributes or whatever feature the filesystem in question posses. In order to avoid, risky OS reboots, I am using lvm snapshots like this: # sync the filesystem to disk sync # create a snapshot with a 10GB buffer lvcreate -L10G -s -n "$snapshot" "/dev/vg/$lv" # write the backup to a backup disk dd if="/dev/vg/$snapshot" of="/mnt/backups/lvm-vg-${lv}_backup-$(date +%Y%m%d).dd" bs=64k # remove the snapshot lvremove -f "/dev/vg/$snapshot" That method is certainly not perfect, as you are still taking the backup with a mounted FS in place and the snapshot size must be adapted to the use-case, but in the end, this is a method to create backups which are complete (independent of FS), easily accessible (e.g. mount via loop device) and do not interrupt the service.
simonh 8y ago
There's now way he could have possible known simply booting the Ubuntu live disk would do this. He could have done that without any consideration to doing a backup. However regular tried and true backups are the way to go, preferably automated. My worst horror story wasn't really mine. I did desktop support in my first job. The library had their own system and their own support arrangements but I had a good relationship with them and would advise from time to time. One day they said the PC running their archive management system wouldn't boot. Could I take a quick look before they called the company on their support contract? Sure enough the PC wouldn't boot, the hard drive had died. I asked if they had recent backups? Oh yes the backups were fine, in fact it had got much better recently. Before the backups used to take an hour, but for for the last few weeks it only took 2 minutes. They only had 2 weeks of tapes on rotation. Needless to say, the backups hadn't been running at all and failed silently, probably due to early symptoms of the hard drive failure. They lost everything.
- Filligree 8y ago
  Did it end up going to data recovery? Most of the time it should be possible to retrieve most of the data, at least.
  - simonh 8y ago
    I don't actually know, I didn't directly support them and they were so embarrassed about the whole thing they clammed up.
- krylon 8y ago
  I have seen one case like this, too. Every week, the customer put a tape into the drive, the machine did something with the tape, but apparently not what it was supposed to do. So when the application running on that machine managed to scramble its (FoxPro) database, they were quite surprised to find their tapes empty. I was really happy I did not have to work on that case. ;-)
- rlpb 8y ago
  There's a collision here between Ubuntu trying to do the right thing automatically (that's part of Ubuntu's design philosophy AIUI) and the author's need to have nothing done at all. I think the Ubuntu live image is therefore unsuitable for this kind of work. What was needed here is a more specialist "rescue" system image that has a "forensic" mode. For example, Finnix has this: https://www.finnix.org/Forensics https://www.finnix.org/Forensics
olavgg 8y ago
Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with a log about how the backup/snapshot/replication went. Even if it went fine. So if I don't get mail, I know something is up and can fix it asap. I have never slept so well after doing this with ZFS, even backup for Windows SQL Server with iSCSI volumes works great! More people should do backup with ZFS!
- fyfy18 8y ago
  Do you use ZFS on the desktop too, or something different? I've used btrfs raid on a server before, and it worked apart one incident which took it two weeks (for 4TB of data) to recover from, so I'm probably going to move to ZFS on the next upgrade. On the desktop there seem to be many more tools for btrfs though, like timeshift [0] - effectively Time Machine for Linux. [0] https://github.com/teejee2008/timeshift https://github.com/teejee2008/timeshift
  - manjana 8y ago
    Just stay away from raid 5/6 with Btrfs "The parity RAID code has multiple serious data-loss bugs in it. It should not be used for anything other than testing purposes" [1] [1] https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
  - olavgg 8y ago
    I use NFS, which works really great and is faster than what most people believe. With the exception of no local page cache which impact latency for some applications. iSCSI is another great alternative, and with iSCSI you will also benefit from your local page cache. Though only one machine can use one iSCSI volume at the time. NFS can be shared between different machines. In my setup there is no special network tuning either, it is completely basic MTU is 1500, NO VLAN's, both iSCSI and regular network traffic works great without impacting for each other. Performance is excellent with the cheap Mellanox ConnextX-2 ethernet cards you find on Ebay for 20 USD. There are some really excellent, cheap, silent and with 10G SFP+ ports network switches available too for just 100USD
diggan 8y ago
Similar story but way shorter. Was working on the beginning of Typeform years ago and we had some issues in production, wanted to take a look at the logs. The logs were in a tar archive, so just needed to untar it. Mixed up the arguments in the tar command and instead overwrote the logs with a empty archive. Doh. In the end, lesson learned: be 100% sure arguments are correct if you're stupid enough to run terminal commands in production environments. Obviously, duplicate data (pull down logs locally) if you're planning to read something that can possibly write instead of just reading is another lesson.
- alexpetralia 8y ago
  Linux commands: with great power comes great responsibility. We're all seduced by the power, but quickly learn the responsibility part!
  - tyingq 8y ago
    There was a time when tar did not remove a leading / in a tar archive when unrolling it. That required some great caution.
- funkymike 8y ago
  If you tar up backups, it is useful to change the permissions on the archive to read-only. It will prevent accidents like this.
aritmo 8y ago
There was a recent example with Asrock motherboards that are supposed to have compliant RAID support https://news.ycombinator.com/item?id=18541493 https://news.ycombinator.com/item?id=18541493 Should open source software fill up with quirks for the mistakes of other software? This looks like a bug report to other RAID software that should clear up any obsolete flags and enforce a sane condition of the RAID.
z3t4 8y ago
These commands look a bit dangerous: # dmsetup remove_all # dmraid --activate y --format isw Is that y a "yes" flag !? eg answer Yes on any question. In my experience, working with anything beside simple mirrors in a RAID config will eventually result in loss of data, and that most data loss is because of human failure rather then disk failure.
- zaarn 8y ago
  No, IIRC the --activate flag accepts either 'y' or 'n' so you can active as well as inactive the array (LVM has similar flags).
anc84 8y ago
RAID, not even once. At least for me.
zaarn 8y ago
I think the moral of the story is; always use software RAID. Hardware RAID (even fake ones like Intel) are time bombs, either you have fun configurations like this or the controller dies and no replacement exists. Unless you'd absolutely need a HW RAID, I'd go for a Software RAID that mounts and works natively in Linux. I can put the harddrives into a fully foreign computer and it would work.
- ghaff 8y ago
  The other moral is that RAID is not a backup. I’d also argue that for most individuals who don’t really need to worry about an hour or two of downtime, it’s a lot more useful to have good (multiple) current backups than potentially complex RAID configurations.
  - anilakar 8y ago
    RAID is the zeroth-tier backup. At least it will prevent most of the horror stories where people accidentally destroy their backups when trying to restore them.
    - arghwhat 8y ago
      No. RAID allows you to combine make many highly fallible disks into one big less fallible disk. Making independent devices into a unified device is a concept that is the polar opposite to a backup. Your example is also why people say that two backups are needed. Once your live system fails, if you only had one backup, you are now left with none. What was a backup is now your only copy, leaving you very vulnerable.
- anilakar 8y ago
  I had this happen eleven years ago, when an integrated hardware RAID controller in a Dell server decided to literally blow up. Oddly enough, despite all the black burn marks on the mainboard, rest of the server seemed to work just okay. Another identical (hardware-wise) cold spare was already in the rack right below the unlucky one, but it wouldn't accept the transplant license dongle because of service tag mismatch. First I had to install Windows 95 just to run the service tag changer tool and flash the firmware on the spare controller to match the original before it recognized the old disks as an existing array. In around one week we were back up and running (this was a non-workplace environment so there were no concerned execs bugging us for status updates every fifteen minutes). Had we been using mdraid, we could just have inserted the disks in just about any commodity PC hardware. In hindsight, just restoring the week-old backups would have been easier. Lesson learned: even if you are using hardware RAID and have an ostensibly identical hardware spare available, it won't necessarily save you.
NKosmatos 8y ago
Sorry for your loss. Here are my two cents... For complete system backup on Windows use Acronis [0], on MacOS use Carbon Copy Cloner [1] and on Linux Clonezilla [2]. Also never forget the 3-2-1 rule of backups [3], 3 copies, 2 local copies on different mediums and 1 copy offsite (cloud, remote) :-) [0] https://www.acronis.com/en-us/personal/computer-backup/ https://www.acronis.com/en-us/personal/computer-backup/ [1] https://bombich.com/ https://bombich.com/ [2] https://clonezilla.org/ https://clonezilla.org/ [3] https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
- rikkus 8y ago
  I use Code42 CrashPlan[0]. This has clients for MacOS, Linux and Windows and it's been flawless since I started using it years ago for personal and business use. Going by the names and the descriptions I've read, the overloaded term 'backup' only applies to Carbon Copy Cloner and Clonezilla in the sense that they make a copy of your storage, so don't protect against corruption (such as ransomware, or you accidentally corrupting files, or deleting files...) Please let me know if I'm wrong on this. Of course you can do real backups with a clone-only tool if you rotate the backup media yourself, but I prefer this aspect to be handled automatically. [0] https://www.crashplan.com https://www.crashplan.com
  - AdmiralAsshat 8y ago
    CrashPlan for personal use no longer exists. Only the business license remains.
    - rikkus 8y ago
      Yes, that's what I'm using. You don't have to be a business to use it.
- guitarbill 8y ago
  For macOS, Time Machine can do automatic backups to a mounted NAS volume over WiFi. So my Macbook backs up to a ZFS 3-way mirror. Important files are also stored in DropBox. That's how I get relatively hassle free 3-2-1 backups. For Windows, a similar strategy is possible, but I consider my Windows box disposable (only gaming), so don't have much recent experience.
alkonaut 8y ago
Takeaways - Your data is at most risk from yourself. Other risks such as hackers, burlgars, fire or hardware failures aren't nearly as dangerous to your data as you are. - Backup is HARD. Have someone else do it. - Mirroring is fine for one first copy. Whether that copy is raid or just another NAS or something, but there should always be a proper BACKUP too. - To be a proper backup it should be 1) Write only/snapshotting, so even if you remove all your local files, the deltions aren't mirrored. 2) Have long retention. When you mess up a document you want to be able to notice 2 years later and just fetch that version 3) Be off site so theft and fire doesn't risk the primary copy and the backups.
- wetpaste 8y ago
  - Get backups sorted out before you consider using it as a production system. If OP had iterated on having a backup system BEFORE adding important data, they wouldn't have consequences to trying to implement it. At the very least, do an online, filesystem level backup of the things that are important before trying to block level disk copy things. Run a database dump, rsync it to an external location, put important things in version control and push to a remote origin. And rsync or tarball your home directory and any other important directories(whole system if possible) and push it out somewhere, then and ONLY then should you feel semi-comfortable to start messing around with RAID settings/LVM/fdisk/ etc.
zeckalpha 8y ago
I think this is clear to the author, but readers should note that RAID isn’t backup. Separately, having at least a partial online backup would have minimized the issues caused by the offline backup.
madshiva 8y ago
This is the wrong way of doing backup. Raid for backup is not so good, specially if you have only 2 drive... Using software (all provided is too a not good example.) Just copy to another hare drive (network, other machine) and mirroir rsync that damn folder, the cost is low and easy to setup and recover.
jondubois 8y ago
The last time I lost all most of my data was when I tried upgrading Ubuntu to the next major version using the upgrade manager in Ubuntu. I thought that since the UI was so friendly looking, it must be the safest route. WRONG. It was the second time that I trusted Ubuntu's UI for doing something critical. Never again. Now I always back up all the files I need onto an external drive and then I install the OS from scratch. Only trust the command line.
- EForEndeavour 8y ago
  Definitely don't even trust the command line. I'm as big of a threat to my own data as a poorly designed upgrade manager.
21 8y ago
No mention of offsite backup. Article could have stopped there. More subtly, he is the epitome of what I call "cowboy programmer/sysadmin", who doesn't really know what he's doing, just always improvising the next step based on minimal research. Which is fine and productive, until you are dealing with critical data/systems. I've witnessed so many times during my career this kind of people royally fuck things up with this "I'm a terminal God, of course I'm doing it live on production" attitude. Everybody I know who used hardware RAID for their personal setup ended up losing their data EVERY SINGLE TIME. Hardware RAID has zero tolerance for human error, it's like the Suicide Linux distribution.
- guitarbill 8y ago
  Bit harsh dude. He admitted he was wrong and even wrote a post about it. That's very honest, and something we should commend.
- dang 8y ago
  Personal attacks are not ok here, regardless of how bad you feel someone is at backups. Please review https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and follow the rules when posting here. We've already had to ask you many times. When accounts continue to violate the guidelines, we end up banning them, and I don't want to do that.
giomasce 8y ago
To me the takeaway is that backup system should be designed with the storage system, when the storage system is still empty of valuable data. If you change the backup system, you should treat that as a change of the whole storage system, so first you would run the former backup, archive that securely, set up the new system, check it and only then release the former backup as not critical anymore.
tracker1 8y ago
Raid is not backup... Your data isn't backed up unless there are at least 3 full copies in 2 physical locations. I have to admit I'm slightly paranoid about it... 20-something years ago I was working in tech support for iomega (mostly os/2 and jazz calls). Nothing taught me the importance of true backups more than that job. I do leverage a few things though... for the most part, I keep stuff on my nas. Projects are in git remotes as well as several local copies. Serial numbers, access keys etc are encrypted and stored in dropbox and copied to google drive. These days since most of my work is in source control, I'm less worried about that aspect. That said, however, RAID is not backup.
- arghwhat 8y ago
  I agree with your idea, but your definition is overkill. A backup is a copy of data on hardware separate from the live system. A good backup is a backup in in a remote location. A backup strategy is to maintain a good backup. A good backup strategy is to have routine restore tests, to make sure that the backups serve their purpose. A great backup strategy requires at least 2 good backups.
  - tracker1 8y ago
    I understand... by 3 full copies, that can include the active in-use copy. I just don't consider something backed up short of having the data in a separate device and location.
    - arghwhat 8y ago
      It has to be on a separate device to be a backup indeed, just like a spare wheel isn't a spare if you're driving on it. The location thing is trickier, and while you can't have a any backup on the same device, you can have a bad backup on the same location. A bad backup is better than no backup by a long shot.
      - tracker1 8y ago
        I'm fine with a backup on-site... I just feel that data isn't safely backed up unless it's off-site too. Not that everything needs to go that far. But it's usually useful to think in those terms imho.
cik 8y ago
I back up all my personal data from each associated desktop via SpiderOak. Personal server data I rely on a combination of rsnapshot, rsync, and SpiderOak. I also rsync to an external drive bay that I swap out weekly. At the end of the day, every single machine in my network of things, including my desktop can light itself on fire - and I don't care. If my office or house is bombed, I don't care. Now - if my city were bombed.. I'd argue that I probably have other fish to fry.
moviuro 8y ago
Redundancy I don't have backups myself, but I have 5 copies of my data at 3 different locations + snapshots everywhere: desktop, laptop (linux, BTRFS), NAS at home (FreeBSD, ZFS); off-site dedicated server at OVH (FreeBSD, ZFS); at school (FreeBSD, ZFS/netapp, managed by school IT). Syncthing [0] + whatever handles local snapshots [1,2] work wonders [3] [0] https://syncthing.net https://syncthing.net [1] https://github.com/zfsnap/zfsnap https://github.com/zfsnap/zfsnap [2] https://gitlab.com/moviuro/butter https://gitlab.com/moviuro/butter [3] https://try.popho.be/securing-home.html https://try.popho.be/securing-home.html
evanweaver 8y ago
> By doing so, I effectively set off a time bomb. That single sector stayed around, lurking, waiting for the right moment. Technically this is a booby trap, not a time bomb. ;-)
saint_abroad 8y ago
It's commendable that the OP was looking to backup but has the reliability of RAID against disk failure contributed to this being a "black swan" event? Do we reduce failure rates to "acceptable" at the expense of neglecting recovery?
sn 8y ago
The recommendation to only wipe the first and last part of the drive before reusing them is inadequate. You're best off wiping the entire drive with either zeros or hdparm security-erase-enhanced. The reason is that not all metadata is kept just at the beginning or end of the drive, especially if you have multiple partitions in the drive. If that is untenable, mdadm has an option --zero-superblock, lvm has pvremove, and wipefs will remove file system signatures, which you can run before changing the storage layout.
- h4myio 8y ago
  Thanks for pointing this out. Do you have an example (or rather source) of a RAID metadata outside of the first/last 1MiB of the drive?
hjek 8y ago
> All it took for my RAID set to become invalid and me losing my data, was booting up Ubuntu Desktop 18.04 ISO image. That's quite a claim. Was there really no way to recover anything, even with testdisk or photorec?
tbyehl 8y ago
If you have the resources to run your own backup server, I heartily recommend UrBackup[1]. It does Windows right by default. It can handle Linux ARM devices. Macs, too. Setup of the server and clients is super simple. 10/10, would install again. [1] https://www.urbackup.org https://www.urbackup.org
sbov 8y ago
I'm always paranoid about this stuff. There's lots of fancy tech words in that article. Before you use ANY piece of backup tech you do not 100.0000000000% understand, back up your data with tech you COMPLETELY and UTTERLY understand how to use. This likely means a simple copy to an external drive. After that, set up your first automated backup solution. Test that it works. After that, set up your second automated backup solution. Test that it works. For the especially paranoid (me), you can still do manual backups using tech you completely and utterly understand.
asaph 8y ago
And... I'm backing up my data right now...
RcouF1uZ4gsC 8y ago
After reading several of these I lost my data stories, one thing that seems to popup over and over is RAID. For me, it is not worth using RAID other than RAID 1 (Mirroring). My data and my time are far more valuable than the extra storage gained. With RAID 1, I can take any working drive, mount it and instantly access it. In addition, there are fewer moving parts and fewer areas for screwups. Just say no to fancy RAID.
herpderperator 8y ago
You should always favor mdadm (software raid) over fake RAID solutions. Sometimes it's better to use this even over real RAID cards (which are proprietary) because you don't have to find an exact clone of the card if it fails; mdadm is all software. The only time it would be appropriate to use a real RAID card is when the performance gain is an absolutely necessary requirement, or if you need to expose a logical device to the BIOS or OS that maps to the RAID array without additional setup.
- bashinator 8y ago
  I once had a particularly bad time where the 3ware RAID card had a bad interaction with the controller on our hard drives, preventing a rebuild after a disk failure. Thank goodness I had marked that system as best-effort, no-guarantees when asked for cheap local mass storage. This wasn’t even fake motherboard RAID - but it was a screwdriver DIY box. Mdadm is far more resilient, and I would never go with hardware RAID again without full vendor support.
rootusrootus 8y ago
I will say, when I was reading that, I got nervous as it went along. You are (were!) less paranoid than I am. Welcome to the club ;-). This is how you get more backups...
coldtea 8y ago
>When it comes to backing up the full OS, I don’t believe in online backup solutions. And how did that work out for you? If you did believe in online backup solutions, in addition to whatever else, you'd have your data now. (I understand worrying about quick recovery, booting, etc. But, as an additional method besides hard local copies, it makes sense, as having an extra off site backup of the data is important. Better to have to spend 3 days to manually download and restore a bootable image, than to not have the data at all. And online backup services also allow for automated backups on your backup -- backup-ception).
- simcop2387 8y ago
  This is how I treat it too. Always have a local backup, but an online or offsite one is also important. The mantra I've often heard repeated is "Two is one, and one is none".
  - et-al 8y ago
    Two is one; one is none is more of a military slogan. Backups should follow the 3-2-1 mantra, fullstop: 3 copies, on 2 different media, and 1 offsite.
    - jrockway 8y ago
      I generalize this to saying "have N+1 replicas, where N is the number of simultaneous failures you want to survive". I'm perfectly happy having my .emacs backed up in 1 place, GitHub. If all my computers and GitHub lose data at the exact same time, I will just make new key bindings. The same might not hold true for your family heirlooms or company's invoices or whatever. Just make the right decision for your application, and don't trust any one system too much. I also feel like adding, replication is less likely to work correctly the more abstract it tries to be. Compare hardware RAID (pretends to the rest of the computer that there is one disk instead of two) to application-level replication (don't return success until the document has been written successfully to 3 nodes). That first one never works correctly, because there's so much that can go wrong and so little you can do to fix it. The on-disk format depends on some random firmware that is treated as disposable by the manufacturer of your motherboard; with no documentation or source code to be had. The second one pretty much always works because it's simple, and when it breaks, you can just examine the content of each node and figure out what went wrong, because you designed it and it's as simple as it can possibly be. (If you want to not store 3 copies of your data, you add your own Reed-Solomon coding to the data you write, and pick the ratio of data size to lost chunks to suit your needs. That's what RAID-5 is, just abstracted over an entire POSIX filesystem and hundreds of thousands of lines of code with no unit tests.) I would never use RAID again, as I consider the complexity too high for the benefits. Treat filesystems and disks as disposable and transient; don't try to build a filesystem that's durable.
HenryBemis 8y ago
Many years ago I was the go-to guy for all friends and family for recovery. I would try my best to recover as much as I could and then I would charge them €100 and I was going and buying them 100 blank DVDs and external DVD writers so they burn the "My Documents" every week. It was a time that CrashPlan and Carbonite were not around.
cellularmitosis 8y ago
I think the key mistake here was allowing the complexity of his higher-level filesystem arrangements (raid, etc) to leak into his lower-level backup routine. Making a system backup like this should just be about imaging the raw disks, regardless of OS, filesystem, raid arrangement, etc. sda is sda is sda, period. Keep it simple and you drastically reduce the likelihood of making a mistake like this. A copy of Knoppix and dd will do the job just fine. Paragon Software, Clonezilla, ntfs-3g, mdadm -- distractions. Even better, if you keep the fundamental units of your system simple (a raw image of sda), you maximize tool compatibility / availability and your chances of recovering when things aren't going to plan.
bndw 8y ago
"Digital data doesn't exist unless it's backed up in at least 3 locations". I had a professor back in 2008 that hammered this into my brain and I share it every chance I get.
fipple 8y ago
Damn, this kind of screwing around with RAID is not what you do with your actual personal data. It’s like building your own car for your daily commute.
- porknubbins 8y ago
  Yeah, when I first got into building PCs it was fun to do the most cutting edge, overclocked, using all possible features setups I could and then 3 years later have no memory of how things worked when I had to replace or fix something. Now I do absolute absolute bone stock simple idiot proof builds and get 90%+ of the performance.
empath75 8y ago
I guess since I do devops sorts of things for a living where we treat servers as ephemeral, I don’t really even consider full drive backups any more. Generally when I set up a computer, I use automatic configuration tools, and any sort of data store will have its own automatic backup and restore solution — or even better, won’t be on that server at all. In the case of a home computer, I’ll install everything using ansible and homebrew (or the App Store or steam) and backup documents and photos to Dropbox and/or iCloud or s3, and all my code and dotfiles of course are on github. I’ve spilled a soda on my laptop and been back up and running with a replacement in 10 or 15 minutes. My whole house could burn down and in terms of data, I’d be at best mildly inconvenienced. For people that don’t trust cloud, it’s way more likely that your physical copy is going to be lost or ruined than that multiple cloud services are going to lose your data.
vagab0nd 8y ago
Slightly off topic, but I've learned (the hard way) that the most important thing in a backup/restore process is the restore part. It doesn't matter how reliable your backup is. If you can't restore it you've done nothing. So far I've managed to: - Backup encrypted content without saving the decryption key - Backup everything to a remote server without backing up the ssh key to that server So do yourself a favor, don't just backup. Try restoring some of your content from time to time (or at least imagine how you would do it). Another scenario I've been recently thinking is, if the house burns down tonight, and you lose everything, phones/computers/etc., would you be able to get your data back? EDIT: formatting
erichocean 8y ago
The only time I've ever lost any data is when trying to back it up.
anon4lol 8y ago
RAID is not a backup strategy. I've lost data with the Intel onboard raid, several times, so I refused to use the Intel raid implementations; if you are going to utilize RAID, I've learned the hard way to pay for a proper RAID card and cage (or use the tried and true software raid setups). If you loose the RAID array, take a break before you start fixing it. A client had their subversion repositories with over a decade of work on a Windows server, depending on a RAID setup. It fell down and turns out they didn't have any good backups. The IT guy tried to fix it and nuked the super block and did who knows what else. They ended up packing up and sending the entire server to some firm that specialized in data recovery and spending a small fortune.

185 points by h4myio 8y ago

Bummer, dude.

why would you try this out with "live" data? why not incrementally connect drives and do it that way? i'm basically asking what the impetus was for "just going for it" which is probably going to be a reason that other people share...

jacobush 8y ago
Incrementally?
- tomschwiha 8y ago
  I assume by incrementally is meant to have a base backup and store the difference in between. Personally I prefer full backups - they feel more complete.
  - mnw21cam 8y ago
    I have a server with 300TB of data. Incremental is the only way that my backups are going to be able to run every night. But also, there are different styles of incremental. Some are dodgier than others.
    - tomschwiha 8y ago
      I agree for this volume of data incremental is the best choice. My projects so far ranged from 1-100GB so a full daily backup is quite quick done and its more easy to recover compared to applying incremental backups individually. Well, always choose the right tool for the task.
      - Izkata 8y ago
        > compared to applying incremental backups individually. At least one style of incremental backups can be restored in a single copy - I use rsync to create hardlinks to the previous day if the file is unchanged (based on modified time). This results in around 400M of additional disk usage per day due to changes, for around 90G of data in my home directory.
evanweaver 8y ago
This is a good point; he could have physically removed one pair of drives, at least, at which point he would have had an actual backup sitting on the shelf.

mobilemidget 8y ago

(many others including myself, have been there, different way though for me) The bright side is, you'll never make this mistake again. The downside is, for me at least, I end up with triple+ backups of everything..

h4myio 8y ago
I can defiantly see myself ending up with triple+ backups of everything...
- tubbs 8y ago
  Not to be a grammar Nazi, but I believe the word you're looking for is definitely. I wouldn't have normally picked it out, but I did see you use it like that in the article as well. It was a good read! Thanks for taking your time to write it up.

cm2187 8y ago

That sounds like a very complicated way to do a full OS backup. Dead easy, 10 min, can't fail alternative for home projects: use a VM instead, switch off the VM, copy the disk file, switch the VM back on. Use NVMe SSDs and depending on the size, you can probably do an automated daily backup with minimal downtime.

sokoloff 8y ago
Anyone have a solution that can snapshot a running VM that has a PCI or PCIe card passed through to the guest OS? I use snapshots on ESXi, but they don't work for guests that have a card passed-through.
- sn 8y ago
  Are you asking for a recommendation on how to do this with ESXi or a recommendation for an alternative to ESXi?
  - sokoloff 8y ago
    Either is fine. If it's within-ESXi, that's a little easier, but I'm not tightly/inextricably wed to that hypervisor.

bklaasen 8y ago

Did you log a bug against `dmraid`? You're right, it shouldn't make changes to the file system without user intervention.

h4myio 8y ago
Not yet. I suppose I should do so via Debian bug report? Cause the last update for the upstream source seems to be around 8 years ago.

arendtio 8y ago

Well, it is not like I didn't do any stupid things in the past, but booting into to a different OS (live system) with such a setup is obviously risky. I am a bit conservative when it comes to backups and I do not trust myself very much. So I am doing backups on the block device level to avoid not copying certain file types, file attributes or whatever feature the filesystem in question posses. In order to avoid, risky OS reboots, I am using lvm snapshots like this: # sync the filesystem to disk sync # create a snapshot with a 10GB buffer lvcreate -L10G -s -n "$snapshot" "/dev/vg/$lv" # write the backup to a backup disk dd if="/dev/vg/$snapshot" of="/mnt/backups/lvm-vg-${lv}_backup-$(date +%Y%m%d).dd" bs=64k # remove the snapshot lvremove -f "/dev/vg/$snapshot" That method is certainly not perfect, as you are still taking the backup with a mounted FS in place and the snapshot size must be adapted to the use-case, but in the end, this is a method to create backups which are complete (independent of FS), easily accessible (e.g. mount via loop device) and do not interrupt the service.

simonh 8y ago

There's now way he could have possible known simply booting the Ubuntu live disk would do this. He could have done that without any consideration to doing a backup. However regular tried and true backups are the way to go, preferably automated. My worst horror story wasn't really mine. I did desktop support in my first job. The library had their own system and their own support arrangements but I had a good relationship with them and would advise from time to time. One day they said the PC running their archive management system wouldn't boot. Could I take a quick look before they called the company on their support contract? Sure enough the PC wouldn't boot, the hard drive had died. I asked if they had recent backups? Oh yes the backups were fine, in fact it had got much better recently. Before the backups used to take an hour, but for for the last few weeks it only took 2 minutes. They only had 2 weeks of tapes on rotation. Needless to say, the backups hadn't been running at all and failed silently, probably due to early symptoms of the hard drive failure. They lost everything.

Filligree 8y ago
Did it end up going to data recovery? Most of the time it should be possible to retrieve most of the data, at least.
- simonh 8y ago
  I don't actually know, I didn't directly support them and they were so embarrassed about the whole thing they clammed up.
krylon 8y ago
I have seen one case like this, too. Every week, the customer put a tape into the drive, the machine did something with the tape, but apparently not what it was supposed to do. So when the application running on that machine managed to scramble its (FoxPro) database, they were quite surprised to find their tapes empty. I was really happy I did not have to work on that case. ;-)
rlpb 8y ago
There's a collision here between Ubuntu trying to do the right thing automatically (that's part of Ubuntu's design philosophy AIUI) and the author's need to have nothing done at all. I think the Ubuntu live image is therefore unsuitable for this kind of work. What was needed here is a more specialist "rescue" system image that has a "forensic" mode. For example, Finnix has this: https://www.finnix.org/Forensics https://www.finnix.org/Forensics

olavgg 8y ago

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with a log about how the backup/snapshot/replication went. Even if it went fine. So if I don't get mail, I know something is up and can fix it asap. I have never slept so well after doing this with ZFS, even backup for Windows SQL Server with iSCSI volumes works great! More people should do backup with ZFS!

fyfy18 8y ago
Do you use ZFS on the desktop too, or something different? I've used btrfs raid on a server before, and it worked apart one incident which took it two weeks (for 4TB of data) to recover from, so I'm probably going to move to ZFS on the next upgrade. On the desktop there seem to be many more tools for btrfs though, like timeshift [0] - effectively Time Machine for Linux. [0] https://github.com/teejee2008/timeshift https://github.com/teejee2008/timeshift
- manjana 8y ago
  Just stay away from raid 5/6 with Btrfs "The parity RAID code has multiple serious data-loss bugs in it. It should not be used for anything other than testing purposes" [1] [1] https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
- olavgg 8y ago
  I use NFS, which works really great and is faster than what most people believe. With the exception of no local page cache which impact latency for some applications. iSCSI is another great alternative, and with iSCSI you will also benefit from your local page cache. Though only one machine can use one iSCSI volume at the time. NFS can be shared between different machines. In my setup there is no special network tuning either, it is completely basic MTU is 1500, NO VLAN's, both iSCSI and regular network traffic works great without impacting for each other. Performance is excellent with the cheap Mellanox ConnextX-2 ethernet cards you find on Ebay for 20 USD. There are some really excellent, cheap, silent and with 10G SFP+ ports network switches available too for just 100USD

diggan 8y ago

Similar story but way shorter. Was working on the beginning of Typeform years ago and we had some issues in production, wanted to take a look at the logs. The logs were in a tar archive, so just needed to untar it. Mixed up the arguments in the tar command and instead overwrote the logs with a empty archive. Doh. In the end, lesson learned: be 100% sure arguments are correct if you're stupid enough to run terminal commands in production environments. Obviously, duplicate data (pull down logs locally) if you're planning to read something that can possibly write instead of just reading is another lesson.

alexpetralia 8y ago
Linux commands: with great power comes great responsibility. We're all seduced by the power, but quickly learn the responsibility part!
- tyingq 8y ago
  There was a time when tar did not remove a leading / in a tar archive when unrolling it. That required some great caution.
funkymike 8y ago
If you tar up backups, it is useful to change the permissions on the archive to read-only. It will prevent accidents like this.

aritmo 8y ago

There was a recent example with Asrock motherboards that are supposed to have compliant RAID support https://news.ycombinator.com/item?id=18541493 https://news.ycombinator.com/item?id=18541493 Should open source software fill up with quirks for the mistakes of other software? This looks like a bug report to other RAID software that should clear up any obsolete flags and enforce a sane condition of the RAID.

z3t4 8y ago

These commands look a bit dangerous: # dmsetup remove_all # dmraid --activate y --format isw Is that y a "yes" flag !? eg answer Yes on any question. In my experience, working with anything beside simple mirrors in a RAID config will eventually result in loss of data, and that most data loss is because of human failure rather then disk failure.

zaarn 8y ago
No, IIRC the --activate flag accepts either 'y' or 'n' so you can active as well as inactive the array (LVM has similar flags).

anc84 8y ago

RAID, not even once. At least for me.

zaarn 8y ago

I think the moral of the story is; always use software RAID. Hardware RAID (even fake ones like Intel) are time bombs, either you have fun configurations like this or the controller dies and no replacement exists. Unless you'd absolutely need a HW RAID, I'd go for a Software RAID that mounts and works natively in Linux. I can put the harddrives into a fully foreign computer and it would work.

ghaff 8y ago
The other moral is that RAID is not a backup. I’d also argue that for most individuals who don’t really need to worry about an hour or two of downtime, it’s a lot more useful to have good (multiple) current backups than potentially complex RAID configurations.
- anilakar 8y ago
  RAID is the zeroth-tier backup. At least it will prevent most of the horror stories where people accidentally destroy their backups when trying to restore them.
  - arghwhat 8y ago
    No. RAID allows you to combine make many highly fallible disks into one big less fallible disk. Making independent devices into a unified device is a concept that is the polar opposite to a backup. Your example is also why people say that two backups are needed. Once your live system fails, if you only had one backup, you are now left with none. What was a backup is now your only copy, leaving you very vulnerable.
anilakar 8y ago
I had this happen eleven years ago, when an integrated hardware RAID controller in a Dell server decided to literally blow up. Oddly enough, despite all the black burn marks on the mainboard, rest of the server seemed to work just okay. Another identical (hardware-wise) cold spare was already in the rack right below the unlucky one, but it wouldn't accept the transplant license dongle because of service tag mismatch. First I had to install Windows 95 just to run the service tag changer tool and flash the firmware on the spare controller to match the original before it recognized the old disks as an existing array. In around one week we were back up and running (this was a non-workplace environment so there were no concerned execs bugging us for status updates every fifteen minutes). Had we been using mdraid, we could just have inserted the disks in just about any commodity PC hardware. In hindsight, just restoring the week-old backups would have been easier. Lesson learned: even if you are using hardware RAID and have an ostensibly identical hardware spare available, it won't necessarily save you.

NKosmatos 8y ago

Sorry for your loss. Here are my two cents... For complete system backup on Windows use Acronis [0], on MacOS use Carbon Copy Cloner [1] and on Linux Clonezilla [2]. Also never forget the 3-2-1 rule of backups [3], 3 copies, 2 local copies on different mediums and 1 copy offsite (cloud, remote) :-) [0] https://www.acronis.com/en-us/personal/computer-backup/ https://www.acronis.com/en-us/personal/computer-backup/ [1] https://bombich.com/ https://bombich.com/ [2] https://clonezilla.org/ https://clonezilla.org/ [3] https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/

rikkus 8y ago
I use Code42 CrashPlan[0]. This has clients for MacOS, Linux and Windows and it's been flawless since I started using it years ago for personal and business use. Going by the names and the descriptions I've read, the overloaded term 'backup' only applies to Carbon Copy Cloner and Clonezilla in the sense that they make a copy of your storage, so don't protect against corruption (such as ransomware, or you accidentally corrupting files, or deleting files...) Please let me know if I'm wrong on this. Of course you can do real backups with a clone-only tool if you rotate the backup media yourself, but I prefer this aspect to be handled automatically. [0] https://www.crashplan.com https://www.crashplan.com
- AdmiralAsshat 8y ago
  CrashPlan for personal use no longer exists. Only the business license remains.
  - rikkus 8y ago
    Yes, that's what I'm using. You don't have to be a business to use it.
guitarbill 8y ago
For macOS, Time Machine can do automatic backups to a mounted NAS volume over WiFi. So my Macbook backs up to a ZFS 3-way mirror. Important files are also stored in DropBox. That's how I get relatively hassle free 3-2-1 backups. For Windows, a similar strategy is possible, but I consider my Windows box disposable (only gaming), so don't have much recent experience.

alkonaut 8y ago

Takeaways - Your data is at most risk from yourself. Other risks such as hackers, burlgars, fire or hardware failures aren't nearly as dangerous to your data as you are. - Backup is HARD. Have someone else do it. - Mirroring is fine for one first copy. Whether that copy is raid or just another NAS or something, but there should always be a proper BACKUP too. - To be a proper backup it should be 1) Write only/snapshotting, so even if you remove all your local files, the deltions aren't mirrored. 2) Have long retention. When you mess up a document you want to be able to notice 2 years later and just fetch that version 3) Be off site so theft and fire doesn't risk the primary copy and the backups.

wetpaste 8y ago
- Get backups sorted out before you consider using it as a production system. If OP had iterated on having a backup system BEFORE adding important data, they wouldn't have consequences to trying to implement it. At the very least, do an online, filesystem level backup of the things that are important before trying to block level disk copy things. Run a database dump, rsync it to an external location, put important things in version control and push to a remote origin. And rsync or tarball your home directory and any other important directories(whole system if possible) and push it out somewhere, then and ONLY then should you feel semi-comfortable to start messing around with RAID settings/LVM/fdisk/ etc.

zeckalpha 8y ago

I think this is clear to the author, but readers should note that RAID isn’t backup. Separately, having at least a partial online backup would have minimized the issues caused by the offline backup.

madshiva 8y ago

This is the wrong way of doing backup. Raid for backup is not so good, specially if you have only 2 drive... Using software (all provided is too a not good example.) Just copy to another hare drive (network, other machine) and mirroir rsync that damn folder, the cost is low and easy to setup and recover.

jondubois 8y ago

The last time I lost all most of my data was when I tried upgrading Ubuntu to the next major version using the upgrade manager in Ubuntu. I thought that since the UI was so friendly looking, it must be the safest route. WRONG. It was the second time that I trusted Ubuntu's UI for doing something critical. Never again. Now I always back up all the files I need onto an external drive and then I install the OS from scratch. Only trust the command line.

EForEndeavour 8y ago
Definitely don't even trust the command line. I'm as big of a threat to my own data as a poorly designed upgrade manager.

21 8y ago

No mention of offsite backup. Article could have stopped there. More subtly, he is the epitome of what I call "cowboy programmer/sysadmin", who doesn't really know what he's doing, just always improvising the next step based on minimal research. Which is fine and productive, until you are dealing with critical data/systems. I've witnessed so many times during my career this kind of people royally fuck things up with this "I'm a terminal God, of course I'm doing it live on production" attitude. Everybody I know who used hardware RAID for their personal setup ended up losing their data EVERY SINGLE TIME. Hardware RAID has zero tolerance for human error, it's like the Suicide Linux distribution.

guitarbill 8y ago
Bit harsh dude. He admitted he was wrong and even wrote a post about it. That's very honest, and something we should commend.
dang 8y ago
Personal attacks are not ok here, regardless of how bad you feel someone is at backups. Please review https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and follow the rules when posting here. We've already had to ask you many times. When accounts continue to violate the guidelines, we end up banning them, and I don't want to do that.

giomasce 8y ago

To me the takeaway is that backup system should be designed with the storage system, when the storage system is still empty of valuable data. If you change the backup system, you should treat that as a change of the whole storage system, so first you would run the former backup, archive that securely, set up the new system, check it and only then release the former backup as not critical anymore.

tracker1 8y ago

Raid is not backup... Your data isn't backed up unless there are at least 3 full copies in 2 physical locations. I have to admit I'm slightly paranoid about it... 20-something years ago I was working in tech support for iomega (mostly os/2 and jazz calls). Nothing taught me the importance of true backups more than that job. I do leverage a few things though... for the most part, I keep stuff on my nas. Projects are in git remotes as well as several local copies. Serial numbers, access keys etc are encrypted and stored in dropbox and copied to google drive. These days since most of my work is in source control, I'm less worried about that aspect. That said, however, RAID is not backup.

arghwhat 8y ago
I agree with your idea, but your definition is overkill. A backup is a copy of data on hardware separate from the live system. A good backup is a backup in in a remote location. A backup strategy is to maintain a good backup. A good backup strategy is to have routine restore tests, to make sure that the backups serve their purpose. A great backup strategy requires at least 2 good backups.
- tracker1 8y ago
  I understand... by 3 full copies, that can include the active in-use copy. I just don't consider something backed up short of having the data in a separate device and location.
  - arghwhat 8y ago
    It has to be on a separate device to be a backup indeed, just like a spare wheel isn't a spare if you're driving on it. The location thing is trickier, and while you can't have a any backup on the same device, you can have a bad backup on the same location. A bad backup is better than no backup by a long shot.
    - tracker1 8y ago
      I'm fine with a backup on-site... I just feel that data isn't safely backed up unless it's off-site too. Not that everything needs to go that far. But it's usually useful to think in those terms imho.

cik 8y ago

I back up all my personal data from each associated desktop via SpiderOak. Personal server data I rely on a combination of rsnapshot, rsync, and SpiderOak. I also rsync to an external drive bay that I swap out weekly. At the end of the day, every single machine in my network of things, including my desktop can light itself on fire - and I don't care. If my office or house is bombed, I don't care. Now - if my city were bombed.. I'd argue that I probably have other fish to fry.

moviuro 8y ago

Redundancy I don't have backups myself, but I have 5 copies of my data at 3 different locations + snapshots everywhere: desktop, laptop (linux, BTRFS), NAS at home (FreeBSD, ZFS); off-site dedicated server at OVH (FreeBSD, ZFS); at school (FreeBSD, ZFS/netapp, managed by school IT). Syncthing [0] + whatever handles local snapshots [1,2] work wonders [3] [0] https://syncthing.net https://syncthing.net [1] https://github.com/zfsnap/zfsnap https://github.com/zfsnap/zfsnap [2] https://gitlab.com/moviuro/butter https://gitlab.com/moviuro/butter [3] https://try.popho.be/securing-home.html https://try.popho.be/securing-home.html

evanweaver 8y ago

> By doing so, I effectively set off a time bomb. That single sector stayed around, lurking, waiting for the right moment. Technically this is a booby trap, not a time bomb. ;-)

saint_abroad 8y ago

It's commendable that the OP was looking to backup but has the reliability of RAID against disk failure contributed to this being a "black swan" event? Do we reduce failure rates to "acceptable" at the expense of neglecting recovery?

sn 8y ago

The recommendation to only wipe the first and last part of the drive before reusing them is inadequate. You're best off wiping the entire drive with either zeros or hdparm security-erase-enhanced. The reason is that not all metadata is kept just at the beginning or end of the drive, especially if you have multiple partitions in the drive. If that is untenable, mdadm has an option --zero-superblock, lvm has pvremove, and wipefs will remove file system signatures, which you can run before changing the storage layout.

h4myio 8y ago
Thanks for pointing this out. Do you have an example (or rather source) of a RAID metadata outside of the first/last 1MiB of the drive?

hjek 8y ago

> All it took for my RAID set to become invalid and me losing my data, was booting up Ubuntu Desktop 18.04 ISO image. That's quite a claim. Was there really no way to recover anything, even with testdisk or photorec?

tbyehl 8y ago

If you have the resources to run your own backup server, I heartily recommend UrBackup[1]. It does Windows right by default. It can handle Linux ARM devices. Macs, too. Setup of the server and clients is super simple. 10/10, would install again. [1] https://www.urbackup.org https://www.urbackup.org

sbov 8y ago

I'm always paranoid about this stuff. There's lots of fancy tech words in that article. Before you use ANY piece of backup tech you do not 100.0000000000% understand, back up your data with tech you COMPLETELY and UTTERLY understand how to use. This likely means a simple copy to an external drive. After that, set up your first automated backup solution. Test that it works. After that, set up your second automated backup solution. Test that it works. For the especially paranoid (me), you can still do manual backups using tech you completely and utterly understand.

asaph 8y ago

And... I'm backing up my data right now...

RcouF1uZ4gsC 8y ago

After reading several of these I lost my data stories, one thing that seems to popup over and over is RAID. For me, it is not worth using RAID other than RAID 1 (Mirroring). My data and my time are far more valuable than the extra storage gained. With RAID 1, I can take any working drive, mount it and instantly access it. In addition, there are fewer moving parts and fewer areas for screwups. Just say no to fancy RAID.

herpderperator 8y ago

You should always favor mdadm (software raid) over fake RAID solutions. Sometimes it's better to use this even over real RAID cards (which are proprietary) because you don't have to find an exact clone of the card if it fails; mdadm is all software. The only time it would be appropriate to use a real RAID card is when the performance gain is an absolutely necessary requirement, or if you need to expose a logical device to the BIOS or OS that maps to the RAID array without additional setup.

bashinator 8y ago
I once had a particularly bad time where the 3ware RAID card had a bad interaction with the controller on our hard drives, preventing a rebuild after a disk failure. Thank goodness I had marked that system as best-effort, no-guarantees when asked for cheap local mass storage. This wasn’t even fake motherboard RAID - but it was a screwdriver DIY box. Mdadm is far more resilient, and I would never go with hardware RAID again without full vendor support.

rootusrootus 8y ago

I will say, when I was reading that, I got nervous as it went along. You are (were!) less paranoid than I am. Welcome to the club ;-). This is how you get more backups...

coldtea 8y ago

>When it comes to backing up the full OS, I don’t believe in online backup solutions. And how did that work out for you? If you did believe in online backup solutions, in addition to whatever else, you'd have your data now. (I understand worrying about quick recovery, booting, etc. But, as an additional method besides hard local copies, it makes sense, as having an extra off site backup of the data is important. Better to have to spend 3 days to manually download and restore a bootable image, than to not have the data at all. And online backup services also allow for automated backups on your backup -- backup-ception).

simcop2387 8y ago
This is how I treat it too. Always have a local backup, but an online or offsite one is also important. The mantra I've often heard repeated is "Two is one, and one is none".
- et-al 8y ago
  Two is one; one is none is more of a military slogan. Backups should follow the 3-2-1 mantra, fullstop: 3 copies, on 2 different media, and 1 offsite.
  - jrockway 8y ago
    I generalize this to saying "have N+1 replicas, where N is the number of simultaneous failures you want to survive". I'm perfectly happy having my .emacs backed up in 1 place, GitHub. If all my computers and GitHub lose data at the exact same time, I will just make new key bindings. The same might not hold true for your family heirlooms or company's invoices or whatever. Just make the right decision for your application, and don't trust any one system too much. I also feel like adding, replication is less likely to work correctly the more abstract it tries to be. Compare hardware RAID (pretends to the rest of the computer that there is one disk instead of two) to application-level replication (don't return success until the document has been written successfully to 3 nodes). That first one never works correctly, because there's so much that can go wrong and so little you can do to fix it. The on-disk format depends on some random firmware that is treated as disposable by the manufacturer of your motherboard; with no documentation or source code to be had. The second one pretty much always works because it's simple, and when it breaks, you can just examine the content of each node and figure out what went wrong, because you designed it and it's as simple as it can possibly be. (If you want to not store 3 copies of your data, you add your own Reed-Solomon coding to the data you write, and pick the ratio of data size to lost chunks to suit your needs. That's what RAID-5 is, just abstracted over an entire POSIX filesystem and hundreds of thousands of lines of code with no unit tests.) I would never use RAID again, as I consider the complexity too high for the benefits. Treat filesystems and disks as disposable and transient; don't try to build a filesystem that's durable.

HenryBemis 8y ago

Many years ago I was the go-to guy for all friends and family for recovery. I would try my best to recover as much as I could and then I would charge them €100 and I was going and buying them 100 blank DVDs and external DVD writers so they burn the "My Documents" every week. It was a time that CrashPlan and Carbonite were not around.

cellularmitosis 8y ago

I think the key mistake here was allowing the complexity of his higher-level filesystem arrangements (raid, etc) to leak into his lower-level backup routine. Making a system backup like this should just be about imaging the raw disks, regardless of OS, filesystem, raid arrangement, etc. sda is sda is sda, period. Keep it simple and you drastically reduce the likelihood of making a mistake like this. A copy of Knoppix and dd will do the job just fine. Paragon Software, Clonezilla, ntfs-3g, mdadm -- distractions. Even better, if you keep the fundamental units of your system simple (a raw image of sda), you maximize tool compatibility / availability and your chances of recovering when things aren't going to plan.

bndw 8y ago

"Digital data doesn't exist unless it's backed up in at least 3 locations". I had a professor back in 2008 that hammered this into my brain and I share it every chance I get.

fipple 8y ago

Damn, this kind of screwing around with RAID is not what you do with your actual personal data. It’s like building your own car for your daily commute.

porknubbins 8y ago
Yeah, when I first got into building PCs it was fun to do the most cutting edge, overclocked, using all possible features setups I could and then 3 years later have no memory of how things worked when I had to replace or fix something. Now I do absolute absolute bone stock simple idiot proof builds and get 90%+ of the performance.

empath75 8y ago

I guess since I do devops sorts of things for a living where we treat servers as ephemeral, I don’t really even consider full drive backups any more. Generally when I set up a computer, I use automatic configuration tools, and any sort of data store will have its own automatic backup and restore solution — or even better, won’t be on that server at all. In the case of a home computer, I’ll install everything using ansible and homebrew (or the App Store or steam) and backup documents and photos to Dropbox and/or iCloud or s3, and all my code and dotfiles of course are on github. I’ve spilled a soda on my laptop and been back up and running with a replacement in 10 or 15 minutes. My whole house could burn down and in terms of data, I’d be at best mildly inconvenienced. For people that don’t trust cloud, it’s way more likely that your physical copy is going to be lost or ruined than that multiple cloud services are going to lose your data.

vagab0nd 8y ago

Slightly off topic, but I've learned (the hard way) that the most important thing in a backup/restore process is the restore part. It doesn't matter how reliable your backup is. If you can't restore it you've done nothing. So far I've managed to: - Backup encrypted content without saving the decryption key - Backup everything to a remote server without backing up the ssh key to that server So do yourself a favor, don't just backup. Try restoring some of your content from time to time (or at least imagine how you would do it). Another scenario I've been recently thinking is, if the house burns down tonight, and you lose everything, phones/computers/etc., would you be able to get your data back? EDIT: formatting

erichocean 8y ago

The only time I've ever lost any data is when trying to back it up.

anon4lol 8y ago

RAID is not a backup strategy. I've lost data with the Intel onboard raid, several times, so I refused to use the Intel raid implementations; if you are going to utilize RAID, I've learned the hard way to pay for a proper RAID card and cage (or use the tried and true software raid setups). If you loose the RAID array, take a break before you start fixing it. A client had their subversion repositories with over a decade of work on a Windows server, depending on a RAID setup. It fell down and turns out they didn't have any good backups. The IT guy tried to fix it and nuked the super block and did who knows what else. They ended up packing up and sending the entire server to some firm that specialized in data recovery and spending a small fortune.