5 ms·

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 18

by olavgg 8y ago

Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local. I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos. I've also configured a daily mail with a log about how the backup/snapshot/replication went. Even if it went fine. So if I don't get mail, I know something is up and can fix it asap. I have never slept so well after doing this with ZFS, even backup for Windows SQL Server with iSCSI volumes works great! More people should do backup with ZFS!

fyfy18 8y ago
Do you use ZFS on the desktop too, or something different? I've used btrfs raid on a server before, and it worked apart one incident which took it two weeks (for 4TB of data) to recover from, so I'm probably going to move to ZFS on the next upgrade. On the desktop there seem to be many more tools for btrfs though, like timeshift [0] - effectively Time Machine for Linux. [0] https://github.com/teejee2008/timeshift https://github.com/teejee2008/timeshift
- manjana 8y ago
  Just stay away from raid 5/6 with Btrfs "The parity RAID code has multiple serious data-loss bugs in it. It should not be used for anything other than testing purposes" [1] [1] https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
- olavgg 8y ago
  I use NFS, which works really great and is faster than what most people believe. With the exception of no local page cache which impact latency for some applications. iSCSI is another great alternative, and with iSCSI you will also benefit from your local page cache. Though only one machine can use one iSCSI volume at the time. NFS can be shared between different machines. In my setup there is no special network tuning either, it is completely basic MTU is 1500, NO VLAN's, both iSCSI and regular network traffic works great without impacting for each other. Performance is excellent with the cheap Mellanox ConnextX-2 ethernet cards you find on Ebay for 20 USD. There are some really excellent, cheap, silent and with 10G SFP+ ports network switches available too for just 100USD
chatmasta 8y ago
Is it safe to backup a database by just straight copying its files? Wouldn't it be better to use a specific backup tool, e.g. pg_dump for Postgres or mongodump for Mongo? (Hopefully you've tested recovery procedures, because as the saying goes, "your backups are only as good as your last restore")
- pmlnr 8y ago
  ZFS is copy-on-write, so in most cases, it is safe.
- lmz 8y ago
  If you use a consistent snapshot of the volume it should be ok. The database would perform crash recovery on startup but data should still be consistent.
- chousuke 8y ago
  With Postgres, it's perfectly safe and supported if you just do it according to the documentation using pg_start/stop_backup and also archive your WALs properly. Most databases have something similar.
  - lettergram 8y ago
    Note, you do need to track the version of Postgres. So a backup from a PG 8 may need to be reloaded into PG 8 before upgrading to PG 9, 10, 11, etc
- mbell 8y ago
  Depends on the database / disk setup. For Mongo you should probably do an fsync/lock prior to taking the snapshot then release it, this will ensure the data on disk is in a consistent state. I don't think this is technically required with a single disk setup (at least with LVM, I've not used ZFS with mongo) but it's probably a good idea anyway and only causes a few ms hiccup.
- wang_li 8y ago
  > Is it safe to backup a database by just straight copying its files? ZFS snapshots are atomic. Recovering from a snapshot is the same as recovering from a loss of power. All serious databases don't acknowledge commits until either the transaction log or the container is synced to disk and can handle an unexpected loss of power/reboot/system crash without significant data loss. However, it is absolutely better to put the database into some kind of backup/write-suspend mode when taking the snapshot.
rikkus 8y ago
If you check that you can restore from backup regularly, then whatever the system is underneath that matters much less. Even an email saying 'things went well' doesn't tell you that you can restore your files.
pmlnr 8y ago
ZFS is great, but it has it's own quirks sometimes. I had an extremely hard time getting rid of a broken zpool a while ago.
- arghwhat 8y ago
  ZFS has no room for configuration errors. A common critical mistake that required making a new pool was to add a non-redundant vdev to a redundant pool, as vdevs could not be removed from pools, permanently rendering this pool non-redundant due to its weakest link. IIRC, the ability to remove a vdev has just recently been added as a feature for ZFS-on-Linux on master, but I do not believe it is released yet.
newscracker 8y ago
What exactly do you do to verify that whatever you backed up was indeed backed up bit-for-bit and that the backup medium didn't just report success even though it didn't get some data stored correctly? Do you verify every backup using some tool? If yes, how long does that take? The reason I ask is because so many sites talk about the benefits of backing up regularly, using backup rotation and retention policies, etc., but they usually don't talk about verification of the backups themselves. Even many popular backup tools don't seem to handle this part (sometimes due to complexity introduced by buffer caches).
- olavgg 8y ago
  ZFS has checksums, so you get a message if some bits are off. Backups has to be verified regulary, there is no exception even with ZFS. For example once a week I fire up a PostgreSQL instance on my remote backup server by cloning one of the snapshots and then I use pg_dump so I also have a "traditional" backup of the database. I do not spend time on verifying this dump though, as the ZFS snapshot already worked fine :-) But I sometimes use the dump to test new hardware or a new PostgreSQL version.
- eslaught 8y ago
  It's somewhat basic right now, but I developed a tool that you can use as an additional level of verification that your backups are working. https://github.com/elliottslaughter/integrity-checker https://github.com/elliottslaughter/integrity-checker
- empath75 8y ago
  If you haven’t restored your data you aren’t doing backups. At work once I set up data backups, the first thing I do is try and replace the backed up server with a restored server. And you should regularly do that if the data is at all important.
pnutjam 8y ago
If the article writer didn't have 2 copies of his data, he didn't have a backup. I keep a running linux server that all my desktops, phones, and other devices back up to, it's a sort of home NAS. I have a 2nd internal drive that gets mounted and all the important photes and less personal stuff gets rsynced to it. My main drive is XFS, but the backup drive is btrfs. I use btrfs to maintain snapshots and unmount the drive when it's not in use. I also take that same set of important stuff and rsync it to a cloud server. I have a TB "storage server" from Time4vps (affiliate link: https://billing.time4vps.eu/?affid=1881 https://billing.time4vps.eu/?affid=1881). I then use borg to backup my more sensitive personal data to my cloud server and 2nd drive.
- pault 8y ago
  I had several copies of my digital archive going back to 1996, including all of the art and music I made as a kid, as well as the first programs I wrote. A few weeks ago I reformatted one of my backup drives so I could use it with my roku TV to watch The Crimes of Grindlewald. Shortly after formatting the disk I realized that I had been mistaken and it was actually the only copy. The worst part is, the movie wasn't even that good.
  - pnutjam 8y ago
    You might be able to recover some of your info with testdisk, a linux utility. Download the system rescue cd (www.system-rescue-cd.org) if you need a rescue disk.
arghwhat 8y ago
It is very important to elaborate that a snapshot is not a backup of anything. If you snapshot a backup to provide a history of backups, then the resulting snapshot is also a backup (albeit not a new one, merely part of the original). However, if what you snapshot is not a backup, then neither is the snapshot. As a good rule of thumb: If you lose data when the machine suddenly bursts into flames in the most literal sense, then that machine cannot even in the slightest be considered a backup. Not even for home use. While the thought of spontaneous combustion seems drastic, total death situations are very real and not rare at all. Lightning, power surge, power supply failure, drive controller failure, simultaneous disk failure, filesystem bugs, and the list goes. A good backup for things that truly matter is also resistant to the entire building being lost in the flames. For ZFS, the obvious choice would be to `zfs send` to replicate to a machine in a remote location location, but beware of the incremental send hole birth issues. If you do not use ZFS, or do not have multiple ZFS machines for replication, look at restic (https://restic.net/ https://restic.net/) and rclone (https://rclone.org/ https://rclone.org/). Restic can through rclone do nice, incremental (snapshot based), deduplicated and encrypted backup to anything from google drive to your toaster. For small payloads (<5TB), consumer cloud solutions such as OneDrive ends up being quite affordable.
- arantius 8y ago
  GP's third sentence: > I keep 180 days with snapshots on the remote backup and 30 days for the local.
  - arghwhat 8y ago
    Yes? I am merely elaborating for clarity, as many have before drawn the conclusion that zfs snapshots are a backup tool on its own.

by olavgg 8y ago

fyfy18 8y ago

Do you use ZFS on the desktop too, or something different? I've used btrfs raid on a server before, and it worked apart one incident which took it two weeks (for 4TB of data) to recover from, so I'm probably going to move to ZFS on the next upgrade. On the desktop there seem to be many more tools for btrfs though, like timeshift [0] - effectively Time Machine for Linux. [0] https://github.com/teejee2008/timeshift https://github.com/teejee2008/timeshift

manjana 8y ago
Just stay away from raid 5/6 with Btrfs "The parity RAID code has multiple serious data-loss bugs in it. It should not be used for anything other than testing purposes" [1] [1] https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
olavgg 8y ago
I use NFS, which works really great and is faster than what most people believe. With the exception of no local page cache which impact latency for some applications. iSCSI is another great alternative, and with iSCSI you will also benefit from your local page cache. Though only one machine can use one iSCSI volume at the time. NFS can be shared between different machines. In my setup there is no special network tuning either, it is completely basic MTU is 1500, NO VLAN's, both iSCSI and regular network traffic works great without impacting for each other. Performance is excellent with the cheap Mellanox ConnextX-2 ethernet cards you find on Ebay for 20 USD. There are some really excellent, cheap, silent and with 10G SFP+ ports network switches available too for just 100USD

chatmasta 8y ago

Is it safe to backup a database by just straight copying its files? Wouldn't it be better to use a specific backup tool, e.g. pg_dump for Postgres or mongodump for Mongo? (Hopefully you've tested recovery procedures, because as the saying goes, "your backups are only as good as your last restore")

pmlnr 8y ago
ZFS is copy-on-write, so in most cases, it is safe.
lmz 8y ago
If you use a consistent snapshot of the volume it should be ok. The database would perform crash recovery on startup but data should still be consistent.
chousuke 8y ago
With Postgres, it's perfectly safe and supported if you just do it according to the documentation using pg_start/stop_backup and also archive your WALs properly. Most databases have something similar.
- lettergram 8y ago
  Note, you do need to track the version of Postgres. So a backup from a PG 8 may need to be reloaded into PG 8 before upgrading to PG 9, 10, 11, etc
mbell 8y ago
Depends on the database / disk setup. For Mongo you should probably do an fsync/lock prior to taking the snapshot then release it, this will ensure the data on disk is in a consistent state. I don't think this is technically required with a single disk setup (at least with LVM, I've not used ZFS with mongo) but it's probably a good idea anyway and only causes a few ms hiccup.
wang_li 8y ago
> Is it safe to backup a database by just straight copying its files? ZFS snapshots are atomic. Recovering from a snapshot is the same as recovering from a loss of power. All serious databases don't acknowledge commits until either the transaction log or the container is synced to disk and can handle an unexpected loss of power/reboot/system crash without significant data loss. However, it is absolutely better to put the database into some kind of backup/write-suspend mode when taking the snapshot.

rikkus 8y ago

If you check that you can restore from backup regularly, then whatever the system is underneath that matters much less. Even an email saying 'things went well' doesn't tell you that you can restore your files.

pmlnr 8y ago

ZFS is great, but it has it's own quirks sometimes. I had an extremely hard time getting rid of a broken zpool a while ago.

arghwhat 8y ago
ZFS has no room for configuration errors. A common critical mistake that required making a new pool was to add a non-redundant vdev to a redundant pool, as vdevs could not be removed from pools, permanently rendering this pool non-redundant due to its weakest link. IIRC, the ability to remove a vdev has just recently been added as a feature for ZFS-on-Linux on master, but I do not believe it is released yet.

newscracker 8y ago

What exactly do you do to verify that whatever you backed up was indeed backed up bit-for-bit and that the backup medium didn't just report success even though it didn't get some data stored correctly? Do you verify every backup using some tool? If yes, how long does that take? The reason I ask is because so many sites talk about the benefits of backing up regularly, using backup rotation and retention policies, etc., but they usually don't talk about verification of the backups themselves. Even many popular backup tools don't seem to handle this part (sometimes due to complexity introduced by buffer caches).

olavgg 8y ago
ZFS has checksums, so you get a message if some bits are off. Backups has to be verified regulary, there is no exception even with ZFS. For example once a week I fire up a PostgreSQL instance on my remote backup server by cloning one of the snapshots and then I use pg_dump so I also have a "traditional" backup of the database. I do not spend time on verifying this dump though, as the ZFS snapshot already worked fine :-) But I sometimes use the dump to test new hardware or a new PostgreSQL version.
eslaught 8y ago
It's somewhat basic right now, but I developed a tool that you can use as an additional level of verification that your backups are working. https://github.com/elliottslaughter/integrity-checker https://github.com/elliottslaughter/integrity-checker
empath75 8y ago
If you haven’t restored your data you aren’t doing backups. At work once I set up data backups, the first thing I do is try and replace the backed up server with a restored server. And you should regularly do that if the data is at all important.

pnutjam 8y ago

If the article writer didn't have 2 copies of his data, he didn't have a backup. I keep a running linux server that all my desktops, phones, and other devices back up to, it's a sort of home NAS. I have a 2nd internal drive that gets mounted and all the important photes and less personal stuff gets rsynced to it. My main drive is XFS, but the backup drive is btrfs. I use btrfs to maintain snapshots and unmount the drive when it's not in use. I also take that same set of important stuff and rsync it to a cloud server. I have a TB "storage server" from Time4vps (affiliate link: https://billing.time4vps.eu/?affid=1881 https://billing.time4vps.eu/?affid=1881). I then use borg to backup my more sensitive personal data to my cloud server and 2nd drive.

pault 8y ago
I had several copies of my digital archive going back to 1996, including all of the art and music I made as a kid, as well as the first programs I wrote. A few weeks ago I reformatted one of my backup drives so I could use it with my roku TV to watch The Crimes of Grindlewald. Shortly after formatting the disk I realized that I had been mistaken and it was actually the only copy. The worst part is, the movie wasn't even that good.
- pnutjam 8y ago
  You might be able to recover some of your info with testdisk, a linux utility. Download the system rescue cd (www.system-rescue-cd.org) if you need a rescue disk.

arghwhat 8y ago

It is very important to elaborate that a snapshot is not a backup of anything. If you snapshot a backup to provide a history of backups, then the resulting snapshot is also a backup (albeit not a new one, merely part of the original). However, if what you snapshot is not a backup, then neither is the snapshot. As a good rule of thumb: If you lose data when the machine suddenly bursts into flames in the most literal sense, then that machine cannot even in the slightest be considered a backup. Not even for home use. While the thought of spontaneous combustion seems drastic, total death situations are very real and not rare at all. Lightning, power surge, power supply failure, drive controller failure, simultaneous disk failure, filesystem bugs, and the list goes. A good backup for things that truly matter is also resistant to the entire building being lost in the flames. For ZFS, the obvious choice would be to `zfs send` to replicate to a machine in a remote location location, but beware of the incremental send hole birth issues. If you do not use ZFS, or do not have multiple ZFS machines for replication, look at restic (https://restic.net/ https://restic.net/) and rclone (https://rclone.org/ https://rclone.org/). Restic can through rclone do nice, incremental (snapshot based), deduplicated and encrypted backup to anything from google drive to your toaster. For small payloads (<5TB), consumer cloud solutions such as OneDrive ends up being quite affordable.

arantius 8y ago
GP's third sentence: > I keep 180 days with snapshots on the remote backup and 30 days for the local.
- arghwhat 8y ago
  Yes? I am merely elaborating for clarity, as many have before drawn the conclusion that zfs snapshots are a backup tool on its own.