6 ms·

Periodic reminder to check if your backups are working, and if you can also restore them. It doesn't matter which file system or operating system you use, make

Periodic reminder to check if your backups are working, and if you can also restore them. It doesn't matter which file system or operating system you use, make sure to backup your stuff. In a way that's immune to ransomware as well, so not just a RAID-1/5/Z or another form of hot/warm storage (RAID is not a backup, it's an uptime/availability mechanism) but cold storage. (I snapshot and tar that snapshot every night, then back it up both on tape and in the cloud.)

bgro 3y ago
It’s always amazing to me how frequently backups silently fail. Every backup software or general common tool to back things up that I’ve seen has many points of silent failure where it just gives up copying at some point in the process or skips over files for some reason without indicating what or why. If you don’t delete files as you go, now you have an unknown partial backup state that basically doubles your needed space. If you delete as you go, sometimes something happens and the process stops or corrupts so your data is now split and you may have lost something. Even trying to log all the failures during the process is amazingly difficult and solutions to work around that specific problem, themselves, somehow introduce more and new types silent failure in some type of irony.
- MenhirMike 3y ago
  Yes! The worst is that even if you set up all kinds of reports etc. on what you expect, if the backup runs for weeks/months successfully, you just stop paying attention and then when something fails, you won't notice it. I do think that file systems that support snapshots - like ZFS, but I think LVM can be used for stuff like ext4, and Apple APFS does too - is the way to go. Not sure how well NTFS's Shadow Copies/Volume Shadow Service work, I heard horror stories, but not sure if those are one-off freak accidents. Probably worth considering ReFS anyway these days on a Windows Server. But with a Snapshot, you're at least insulating yourself mostly from changes to the data you're backing up. At the expensive of managing snapshots, that is, getting rid of old ones after a while because they keep taking up space.
  - MenhirMike 3y ago
    (Edit: Though a snapshot of the file system isn't enough if you need to back up services that are currently running. E.g., a database server might have stuff uncommitted in memory that wouldn't be captured by a file system snapshot. But database backups are their own beast to wrangle.)
tetha 3y ago
It is also a good idea to test the restore procedures and documentation as well. Don't have the grizzled old storage admin / DBA test the backup. They know a million and one weird necessary workarounds and just execute them. However, if you need a restore and they are currently exploring caves or something, things turn dire. Have a chipper junior restore something based off of the documentation (and prepare to spend a few days updating documentation...) And make sure to test backup you don't regularly touch. And very much test those backups you really don't want to test.
- ericbarrett 3y ago
  As a grizzled old storage admin who somehow made a career out of database backups, I wholeheartedly agree with all of this. Especially having someone else do a test restore. They don't have to be junior, just not intimately familiar with the systems involved.
  - FridgeSeal 3y ago
    I’ll go a step further: Have a different person do it each time, having them add and refine the documentation and any tooling once they’ve done it. Keep any tools and scripts used fastidiously current-few things are worse than “to fix this issue, run the repair.sh script” only to find it stopped working 6 months ago because it relied on some extremely specific lib somewhere.
  - MenhirMike 3y ago
    Oh, database backups are fun, especially if the database server is still running! You want multiple databases, all at a consistent point in time, without taking the system offline? SUFFER, YOU FOOL! The joys of realizing that file system snapshots won't help with data that's still to be committed and that taking a backup database-by-database means that two databases whose data relies on each other are no longer properly in sync really warms my heart. Oh wait, it's the whiskey that runs through my veins that does that, being a backup operator is a fantastic pathway into alcoholism. Especially once the databases become so large, that the time it takes to take the backup become a performance concern of the running system. I think Postgres did it right by abbreviating their "Continuous Archiving and Point-in-Time Recovery" as PITR because it's very close to PITA. But PITR and CHECKPOINT actually make Postgres probably one of the better database systems to backup (and restore!), so yet another reason why I think it's a fantastic database.
stevenAthompson 3y ago
I see this advice repeated frequently, but it's always very general. Do you have any advice as to HOW the average home NAS user can affordably backup modern NAS devices? The last time I looked it could easily cost hundreds of dollars per month to back up as little as 40TB to the cloud.
- dewey 3y ago
  “Average home NAS user” doesn’t have 40TB of data. With a subset of data that’s important like photos it’s not that expensive and with Backblaze and other services that are directly integrated in operating systems like Synology also not that hard to do.
  - samastur 3y ago
    I agree with the advice which is what we do. Average home user (with emphasis on average) doesn't have 40TB, but a "normal" non-professional one might. We have about 9TB of photos. I can easily imagine someone like us, who is into video, of having more than 40TB of videos.
    - feanaro 3y ago
      When will you ever be able to appreciate and look at 9T of photos?
      - doublepg23 3y ago
        AI tools analyze photos pretty well now. It’s very common they bubble up old photos I had forgotten about.
        fomine3 3y ago
        Good point, now AI is a real good excuse for thoughtless data hoarding.
      - pferde 3y ago
andyjohnson0 3y ago
I'd be interested to know what tape setup you use? I occasionally look into using LTO tapes for home backup, but the media and hardware always seems a bit too expensive compared to something like Backblaze (which I currently use). Also afaik tapes need a stable storage environment: how do you manage that?
- grepfru_it 3y ago
  LTO5. The cost of my LTO5 system is the cost of downloading all of my data once from a remote cloud provider. It's a nobrainer
  - linuxdude314 3y ago
    As someone who used to admin a 30PB+ LTO library I love me some tape, but unfortunately it’s not that simple of a value proposition. Bit rot is less of a thing with LTO, but still a thing.. I.e. you will at some point need to update your LTO system and it’s storage media. The robot I owned was the library storage for movie frames at a major motion picture studio. We would upgrade every other release, so while I was there we were upgrading from LTO-5 to LTO—7. The robot was big, and would write data to two redundant tapes. One copy would be sent to Iron Mountain, the other stayed in the robot. Creating a backup like you are isn’t really protecting much if you don’t have a good facility to store the tapes in. Part of the point of paying for a service like AWS Deep Glacier is that it’s an offsite backup. An LTO backup has no advantage over a hard disk if your home catches on fire.
    - grepfru_it 3y ago
      Well I store two sets of offsite tapes. One at my colo site and one in a climate controlled storage facility. I rotate my tapes weekly and the inbound tapes get restored and compared against the disk backup. I also ran IT departments for the last 30 years, so you probably shouldn’t use me as the scapegoat :)
- MenhirMike 3y ago
  For tapes, LTO is really the only game in town, every other tape format is dead. You can get LTO-4 tape drives for dirt cheap because companies have been upgrading them. Yeah, they'll be used, but those drives are meant for heavy duty, and you can just pick up some spares. I found that IBM Fibre-Channel drives are available aplenty, cheap, and they usually come with a front bezel for installation into either two 5.25" slots or something like a Dell PowerVault 114X. (Unlike Library Drives that usually come naked and in non-standard form factors). A FibreChannel host adapter, some cables and transceivers, for probably less than $20 combined, and you're good to go. LTO-4 tapes hold 800 GB and are readily available new for affordable prices as well. I did upgrade to an LTO-5 drive last year so so, after finding a new-in-box from a liquidation sale for something like $450. The nice thing about LTO is that it's 2 Generations R/W and 3 Generations Read - so the LTO-5 drive will Read/Write LTO-5 (1.5 TB) and LTO-4 tapes, and read LTO-3 tapes. I think with one of the new standards (LTO-8?) it's a bit more muddy, so check compatibility. I think that LTO-4 and LTO-5 is the sweet spot for hobbyists: You still need to spend some money on a drive or two and buy brand new tapes, but it's reasonably affordable. That said, for a business, I'd just bite the bullet and buy a new drive. Dell sells an external SAS LTO-7 drive brand new for $3700 list price, but I think there might be cheaper options. Together with some tapes and a SAS Controller, I'd say that for $5000 you can get a decent, brand new setup. I put the tapes in Turtle LTO Cases (https://turtlecase.com/products/lto-20-black https://turtlecase.com/products/lto-20-black), and they sit in a closet. It's not climate controlled or anything, but the place is roughly at a similar temperature year round. The tapes aren't THAT sensitive, but I'd definitely not store them in the garage where I might get a 50+ degree temperature difference throughout the year. That said, there are companies that offer off-site storage options with climate controlled environments. I haven't looked into their pricing since I didn't need it, but the nice thing about tapes is that you can just backup to two tapes and send the second tape off-site. LTO has built-in encryption support, so that's an option. Twice a year or so, I run a restore of the tape and compare it to the SHA256 that I took while backing up the file (I did build myself some rudimentary cataloging system to SHA256 hash every file, then back it up to tape with tar, and make a record of what file with what SHA256 got backed up when on what tape). I've yet to encounter any bit rot/defective tape issues, but YMMV. I do use Backblaze's B2 service as well for cold-ish storage. Though I only back up truly irreplaceable or inconvenient to recreate data into B2. That way, I have multiple copies of truly important stuff, I have stuff readily available where I am, and I have terabytes of stuff that isn't worth the expense for the cloud since I can re-create it, but nice to have a copy of. Tape Drives may be overkill for many and external hard drives (plural!) might be a better option for many. What I like about tape drives is that the media isn't "hot". If I have ransomware running wild, connecting an external hard drive puts everything on it at risk (hence the need for multiple drives), whereas with a tape, it would have to specifically try to rewind the tape and start overwriting, and I would notice it. But YMMV, I never had a ransomware problem myself, but I do have stuff I really don't want to lose, so multiple backups of it in multiple ways (Daily .tar archive on a hard drive, backed up to tape, and backed up to the cloud) should hopefully give defense in depth and the ability to at least recover some older state.
KingOfCoders 3y ago
After every backup it needs to automatically be checked if it isn't corrupted. At minimun check file size and see if it can be decrypted / untarred. Best check the latest data. Backup that isn't checked isn't done.
amluto 3y ago
This particular bug won’t be easily caught just by testing backups, as the bytes in the filesystem never actually change. So you can diff the bytes on disk between the live system and the backup, and they’ll match. I like to keep a separate database of what files I expect to have along with their hashes. The off-the-shelf tooling for this is weak, to say the least. Even S3’s integrity checking support is desultory at best, and a bunch of S3 clones don’t implement it at all (cough minio cough).
deleted 3y ago
[deleted]
mekster 3y ago
The periodic reminder is to at least use 2 different implementations to make backups than rely on 1 such as Borg.

bgro 3y ago

It’s always amazing to me how frequently backups silently fail. Every backup software or general common tool to back things up that I’ve seen has many points of silent failure where it just gives up copying at some point in the process or skips over files for some reason without indicating what or why. If you don’t delete files as you go, now you have an unknown partial backup state that basically doubles your needed space. If you delete as you go, sometimes something happens and the process stops or corrupts so your data is now split and you may have lost something. Even trying to log all the failures during the process is amazingly difficult and solutions to work around that specific problem, themselves, somehow introduce more and new types silent failure in some type of irony.

MenhirMike 3y ago
Yes! The worst is that even if you set up all kinds of reports etc. on what you expect, if the backup runs for weeks/months successfully, you just stop paying attention and then when something fails, you won't notice it. I do think that file systems that support snapshots - like ZFS, but I think LVM can be used for stuff like ext4, and Apple APFS does too - is the way to go. Not sure how well NTFS's Shadow Copies/Volume Shadow Service work, I heard horror stories, but not sure if those are one-off freak accidents. Probably worth considering ReFS anyway these days on a Windows Server. But with a Snapshot, you're at least insulating yourself mostly from changes to the data you're backing up. At the expensive of managing snapshots, that is, getting rid of old ones after a while because they keep taking up space.
- MenhirMike 3y ago
  (Edit: Though a snapshot of the file system isn't enough if you need to back up services that are currently running. E.g., a database server might have stuff uncommitted in memory that wouldn't be captured by a file system snapshot. But database backups are their own beast to wrangle.)

tetha 3y ago

It is also a good idea to test the restore procedures and documentation as well. Don't have the grizzled old storage admin / DBA test the backup. They know a million and one weird necessary workarounds and just execute them. However, if you need a restore and they are currently exploring caves or something, things turn dire. Have a chipper junior restore something based off of the documentation (and prepare to spend a few days updating documentation...) And make sure to test backup you don't regularly touch. And very much test those backups you really don't want to test.

ericbarrett 3y ago
As a grizzled old storage admin who somehow made a career out of database backups, I wholeheartedly agree with all of this. Especially having someone else do a test restore. They don't have to be junior, just not intimately familiar with the systems involved.
- FridgeSeal 3y ago
  I’ll go a step further: Have a different person do it each time, having them add and refine the documentation and any tooling once they’ve done it. Keep any tools and scripts used fastidiously current-few things are worse than “to fix this issue, run the repair.sh script” only to find it stopped working 6 months ago because it relied on some extremely specific lib somewhere.
- MenhirMike 3y ago
  Oh, database backups are fun, especially if the database server is still running! You want multiple databases, all at a consistent point in time, without taking the system offline? SUFFER, YOU FOOL! The joys of realizing that file system snapshots won't help with data that's still to be committed and that taking a backup database-by-database means that two databases whose data relies on each other are no longer properly in sync really warms my heart. Oh wait, it's the whiskey that runs through my veins that does that, being a backup operator is a fantastic pathway into alcoholism. Especially once the databases become so large, that the time it takes to take the backup become a performance concern of the running system. I think Postgres did it right by abbreviating their "Continuous Archiving and Point-in-Time Recovery" as PITR because it's very close to PITA. But PITR and CHECKPOINT actually make Postgres probably one of the better database systems to backup (and restore!), so yet another reason why I think it's a fantastic database.

stevenAthompson 3y ago

I see this advice repeated frequently, but it's always very general. Do you have any advice as to HOW the average home NAS user can affordably backup modern NAS devices? The last time I looked it could easily cost hundreds of dollars per month to back up as little as 40TB to the cloud.

dewey 3y ago
“Average home NAS user” doesn’t have 40TB of data. With a subset of data that’s important like photos it’s not that expensive and with Backblaze and other services that are directly integrated in operating systems like Synology also not that hard to do.
- samastur 3y ago
  I agree with the advice which is what we do. Average home user (with emphasis on average) doesn't have 40TB, but a "normal" non-professional one might. We have about 9TB of photos. I can easily imagine someone like us, who is into video, of having more than 40TB of videos.
  - feanaro 3y ago
    When will you ever be able to appreciate and look at 9T of photos?
    - doublepg23 3y ago
      AI tools analyze photos pretty well now. It’s very common they bubble up old photos I had forgotten about.
      - fomine3 3y ago
        Good point, now AI is a real good excuse for thoughtless data hoarding.
    - pferde 3y ago

andyjohnson0 3y ago

I'd be interested to know what tape setup you use? I occasionally look into using LTO tapes for home backup, but the media and hardware always seems a bit too expensive compared to something like Backblaze (which I currently use). Also afaik tapes need a stable storage environment: how do you manage that?

grepfru_it 3y ago
LTO5. The cost of my LTO5 system is the cost of downloading all of my data once from a remote cloud provider. It's a nobrainer
- linuxdude314 3y ago
  As someone who used to admin a 30PB+ LTO library I love me some tape, but unfortunately it’s not that simple of a value proposition. Bit rot is less of a thing with LTO, but still a thing.. I.e. you will at some point need to update your LTO system and it’s storage media. The robot I owned was the library storage for movie frames at a major motion picture studio. We would upgrade every other release, so while I was there we were upgrading from LTO-5 to LTO—7. The robot was big, and would write data to two redundant tapes. One copy would be sent to Iron Mountain, the other stayed in the robot. Creating a backup like you are isn’t really protecting much if you don’t have a good facility to store the tapes in. Part of the point of paying for a service like AWS Deep Glacier is that it’s an offsite backup. An LTO backup has no advantage over a hard disk if your home catches on fire.
  - grepfru_it 3y ago
    Well I store two sets of offsite tapes. One at my colo site and one in a climate controlled storage facility. I rotate my tapes weekly and the inbound tapes get restored and compared against the disk backup. I also ran IT departments for the last 30 years, so you probably shouldn’t use me as the scapegoat :)
MenhirMike 3y ago
For tapes, LTO is really the only game in town, every other tape format is dead. You can get LTO-4 tape drives for dirt cheap because companies have been upgrading them. Yeah, they'll be used, but those drives are meant for heavy duty, and you can just pick up some spares. I found that IBM Fibre-Channel drives are available aplenty, cheap, and they usually come with a front bezel for installation into either two 5.25" slots or something like a Dell PowerVault 114X. (Unlike Library Drives that usually come naked and in non-standard form factors). A FibreChannel host adapter, some cables and transceivers, for probably less than $20 combined, and you're good to go. LTO-4 tapes hold 800 GB and are readily available new for affordable prices as well. I did upgrade to an LTO-5 drive last year so so, after finding a new-in-box from a liquidation sale for something like $450. The nice thing about LTO is that it's 2 Generations R/W and 3 Generations Read - so the LTO-5 drive will Read/Write LTO-5 (1.5 TB) and LTO-4 tapes, and read LTO-3 tapes. I think with one of the new standards (LTO-8?) it's a bit more muddy, so check compatibility. I think that LTO-4 and LTO-5 is the sweet spot for hobbyists: You still need to spend some money on a drive or two and buy brand new tapes, but it's reasonably affordable. That said, for a business, I'd just bite the bullet and buy a new drive. Dell sells an external SAS LTO-7 drive brand new for $3700 list price, but I think there might be cheaper options. Together with some tapes and a SAS Controller, I'd say that for $5000 you can get a decent, brand new setup. I put the tapes in Turtle LTO Cases (https://turtlecase.com/products/lto-20-black https://turtlecase.com/products/lto-20-black), and they sit in a closet. It's not climate controlled or anything, but the place is roughly at a similar temperature year round. The tapes aren't THAT sensitive, but I'd definitely not store them in the garage where I might get a 50+ degree temperature difference throughout the year. That said, there are companies that offer off-site storage options with climate controlled environments. I haven't looked into their pricing since I didn't need it, but the nice thing about tapes is that you can just backup to two tapes and send the second tape off-site. LTO has built-in encryption support, so that's an option. Twice a year or so, I run a restore of the tape and compare it to the SHA256 that I took while backing up the file (I did build myself some rudimentary cataloging system to SHA256 hash every file, then back it up to tape with tar, and make a record of what file with what SHA256 got backed up when on what tape). I've yet to encounter any bit rot/defective tape issues, but YMMV. I do use Backblaze's B2 service as well for cold-ish storage. Though I only back up truly irreplaceable or inconvenient to recreate data into B2. That way, I have multiple copies of truly important stuff, I have stuff readily available where I am, and I have terabytes of stuff that isn't worth the expense for the cloud since I can re-create it, but nice to have a copy of. Tape Drives may be overkill for many and external hard drives (plural!) might be a better option for many. What I like about tape drives is that the media isn't "hot". If I have ransomware running wild, connecting an external hard drive puts everything on it at risk (hence the need for multiple drives), whereas with a tape, it would have to specifically try to rewind the tape and start overwriting, and I would notice it. But YMMV, I never had a ransomware problem myself, but I do have stuff I really don't want to lose, so multiple backups of it in multiple ways (Daily .tar archive on a hard drive, backed up to tape, and backed up to the cloud) should hopefully give defense in depth and the ability to at least recover some older state.

KingOfCoders 3y ago

After every backup it needs to automatically be checked if it isn't corrupted. At minimun check file size and see if it can be decrypted / untarred. Best check the latest data. Backup that isn't checked isn't done.

amluto 3y ago

This particular bug won’t be easily caught just by testing backups, as the bytes in the filesystem never actually change. So you can diff the bytes on disk between the live system and the backup, and they’ll match. I like to keep a separate database of what files I expect to have along with their hashes. The off-the-shelf tooling for this is weak, to say the least. Even S3’s integrity checking support is desultory at best, and a bunch of S3 clones don’t implement it at all (cough minio cough).

deleted 3y ago

[deleted]

mekster 3y ago

The periodic reminder is to at least use 2 different implementations to make backups than rely on 1 such as Borg.