8 ms·

Let’s Talk OpenZFS Snapshots

119 points by rodrigo975 5y ago

anotherhue 5y ago
If you're using a recent Ubuntu, docker/zfs can lead to a large amount of snapshots. 'docker system prune' may help but has had zfs issues for me.
- yjftsjthsd-h 5y ago
  > but has had zfs issues for me. Can you give more details so I know what to watch out for? (Recently started running Docker+ZFS in prod, haven't seen it break yet but appreciate heads up)
  - flemhans 5y ago
    The zfs storage driver isn't production ready yet.
    - r1ch 5y ago
      I love that they say this, then default to using it when ZFS is detected.
    - cosmin800 5y ago
      I am pretty sure Canonical will soon release a stable,production ready snap package for zfs.
      - alyandon 5y ago
        That gets updated arbitrarily at random intervals (because who doesn't want filesystem modules updated frequently!?!?!) and then purges logs every 72 hours so when you have a problem you'll never even know what was updated! I have to run snapd on one of my servers and I hate everything about it.
        flemhans 5y ago
        I'm considering to run lxd as a snap because I want Debian 11 (for openzfs 2.0) and lxd at the same time. Am I in for a world of hurt?
        alyandon 5y ago
        I wouldn't say you are in for a world of hurt but you'll have to accept that the package will periodically update and in the process kill your containers/vms. My primary objections to snapd are: 1) updates are pushed/applied and services restarted on an arbitrary schedule which means you can't test updates prior to rolling them out or otherwise control when a service is taken offline by an update 2) snapd doesn't log to standard OS logging facilities and the logs it does have are cleared after 72 hours 3) snapd pollutes home directories with a directory called "snap" snapd is a poorly conceived project written by people that are profoundly ignorant of how real services and servers are actually operated. Despite having the above flaws repeatedly pointed out, the developers have shown zero interest in taking actual concrete actions to address them.
        flemhans 5y ago
        Thank you for the reply. It seems snapd is indeed a poorly conceived project. How would you work around it in a production setting if you HAD to use it? Auto-updating sounds like the biggest issue of those.
        zekrioca 5y ago
        Build LXD yourself. It is not that complicated if one updates go to something beyond 1.14. Only downside is that you may need to create your own init scripts and a lxd group, but that's not complicated either.
  - anotherhue 5y ago
    there were a number of containers immune to 'docker rm', seemed it was trying to invoke 'zfs delete' on something that had already been removed (presumably moments ago as part of a previous step). Nasty workaround was to manually recreate the dataset (and children) so that it could remove them. The issue was common enough that I could google it so don't stress. More worryingly, 'prune' locked up for about 10 minutes before getting to do it's thing. pinning the cpu the whole time. This was on my devbox, so maybe it's my doing but did not inspire confidence. I've run FreeBSD ZFS for about a decade and openzfs/ubuntu in prod for kafka clusters so I'm no stranger - this docker usage was just particularly strange.
  - e12e 5y ago
    (not op, but) I don't recall exactly what I experienced (in development- I think it was just terribly slow?) - I ended up carving out a volume (zvol) and formatting it with ext4 (or xfs?) for docker. It might actually have been this ms sql issue that led me to do that, actually: https://github.com/moby/moby/issues/33191 https://github.com/moby/moby/issues/33191
- hddherman 5y ago
  I've had bad experiences with Docker+Debian as well when the storage was on ZFS. I've had a much better experience by just keeping /var/log/containers on something simple, like ext4. Valuable data is still on ZFS and mounted inside the containers, no issues there.
mvanbaak 5y ago
'In this article, we’ll learn how ZFS boot environments (BEs) are used to boot the operating system itself into a previous point in time.' The whole article does not touch the subject of boot environments ... Maybe the author can update the page and state that the article is about managing snapshots instead of boot environments?
- beermonster 5y ago
  I read the entire article as I was interested in that aspect too. Misleading title.
  - vermaden 5y ago
    Here: https://is.gd/BECTL https://is.gd/BECTL
  - 1MachineElf 5y ago
    A common pattern with beadm on FreeBSD is for the root filesystem dataset to be called "ROOT", and this mirrors what's in the main article. It appears this person has boot environments enabled, but didn't actually explain anything about them. ZFS boot environments are awesome. You can get them on both FreeBSD and Illumos with beadm. On Linux there are a number of utilities, but I've only used ZSys. It comes working out of the box if you do a ZFS-on-root install with Ubuntu.
- 5wp3 5y ago
  Boot environment is just a fancy term for root filesystem clones created from snapshots. Fyi: snapshots are read-only. Normal, read-writable filesystem can be created by `zfs clone root@snap newroot`.
- vermaden 5y ago
  Here: https://is.gd/BECTL https://is.gd/BECTL
barrkel 5y ago
I make snapshots every hour for 24 hours, every day for 7 days, every week for 4 weeks and every month for three months. This keeps the number of snapshots reasonable while keeping most stuff around should you accidentally delete or overwrite it. I can see a good argument for more fine grained snapshots (e.g. every few minutes) but I think I'd limit that to directories (and thus FSes) known to have frequent updates. I sometimes blow away all snapshots if I have a big clear out or rewrite of data, or else that stuff sticks around for months.
- throw0101a 5y ago
  > I can see a good argument for more fine grained snapshots (e.g. every few minutes) but I think I'd limit that to directories (and thus FSes) known to have frequent updates. Fun fact, the ZFS send-receive functionality was born with snapshots being created every few minutes in mind: > ZFS send and receive wasn’t considered until late in the development cycle. The idea came to me in 2005. ZFS was nearing integration, and I was spending a few months working in Sun’s new office in Beijing, China. The network link between Beijing and Menlo Park was low-bandwidth and high-latency, and our NFS-based source code manager was painful to use. I needed a way to quickly ship incremental changes to a workspace across the Pacific. A POSIX-based utility (like rsync) would at best have to traverse all the files and directories to find the few that were modified since a specific date, and at worst it would compare the files on each side, incurring many high-latency round trips. I realized that the block pointers in ZFS already have all the information we need: the birth time allows us to quickly and precisely find the blocks that are changed since a given snapshot. It was easiest to implement ZFS send at the DMU layer, just below the ZPL. This allows the semantically-important changes to be transferred exactly, without any special code to handle features like NFSv4 style ACLs, case-insensitivity, and extended attributes. Storage-specific settings, like compression and RAID type, can be different on the sending and receiving sides. What began as a workaround for a crappy network link has become one of the pillars of ZFS, and the foundation of several remote replication products, including the one at Delphix. * https://www.delphix.com/blog/delphix-engineering/zfs-10-year-anniversary https://www.delphix.com/blog/delphix-engineering/zfs-10-year...
  - 5y ago
kissgyorgy 5y ago
ZFS snapshots saved my ass multiple times after a bad Ubuntu upgrade. It's pretty easy to roll back.
- unixhero 5y ago
  How do you roll back after a botched OS update then? Boot into a rescue cd?
  - vermaden 5y ago
    Like that: https://is.gd/BECTL https://is.gd/BECTL By using ZFS Boot Environments.
    - unixhero 5y ago
      Woot, thanks a lot.
eptcyka 5y ago
Is there any good tooling on managing automatic snapshotting and sending backups?
- ptomato 5y ago
  sanoid (https://github.com/jimsalterjrs/sanoid/ https://github.com/jimsalterjrs/sanoid/) is probably the most featureful, but there's quite a few I'm aware of - zfsnap, zfs-auto-snapshot, zfs_autobackup, znapzend...
- indiv0 5y ago
  Personally I just setup sanoid and syncoid on my NAS. They were dead simple to get up and running with. Sanoid makes snapshots according to a schedule you define (e.g. 1 monthly, 31 daily, 24 hourly). It also prunes old snapshots to ensure that you always have the exact snapshots you need available and no more. Syncoid uses zfs send/recv to copy over snapshots from one ZFS dataset to another, typically between machines. You basically have two options: either you can: 1) copy over ALL of the snapshots from the source in the order they were created (like cherry picking one git commit after another onto another branch), or you can: 2) configure it to make a new snapshot every time it syncs and just copy over the diff since the last time you synced (a bit like squashing all the commits into one and just cherry-picking that). The former is good if you want to maintain the same snapshots on the source and the destination (you can also use sanoid on the destination to prune the snapshots further). The latter is good if you want to minimize the amount of churn sent over the network. For example if you're snapshotting VM images you'd be interested in sending over the state of the VM as it appears at the end of the day, rather than all the intermediate state.
- waz0wski 5y ago
  sanoid is a good one. I've been on a quest to run mostly golang-based utilities and switched to https://github.com/zrepl/zrepl https://github.com/zrepl/zrepl which has been running/syncing without issue for a couple years now
- unixhero 5y ago
  Proxmox backup node https://www.proxmox.com/en/proxmox-backup-server https://www.proxmox.com/en/proxmox-backup-server
pawelduda 5y ago
Does it make sense to snapshot the entire root dir or just the home directory? Asking about desktop PC.
- tiernano 5y ago
  Probably home did, maybe etc and var, maybe other important dirs... full root would include temp and logs that may not be needed and folders that don't change often
- solarengineer 5y ago
  For Solaris 11 and for Illumos based distros, snapshotting the entire root is normal. That way, with an OS update and application update, the logs are associated with that particular root file system. You can always look into earlier snapshots to learn of what the logs contained for that earlier root file system. With Boot Environments (I don't know if Linux based Distros let one do this) on Solaris and Illumos, we have multiple active root file systems to switch between and can mark snapshots for each of them. Certainly, for Belenix (KDE based opensolaris distro), we snapshotted the entire root file system.
  - throw0101a 5y ago
    > For Solaris 11 and for Illumos based distros, snapshotting the entire root is normal. FreeBSD stole this feature from Solaris as well: * https://www.freebsd.org/cgi/man.cgi?beadm https://www.freebsd.org/cgi/man.cgi?beadm * https://wiki.freebsd.org/BootEnvironments https://wiki.freebsd.org/BootEnvironments
    - vermaden 5y ago
      ... and Illumos stole Bhyve and Bootloader from FreeBSD? :) IMHO implemented or ported is better word here.
  - 5wp3 5y ago
    This is implemented for Arch Linux with bieaz or zbm; and Ubuntu with zsys. bieaz: https://openzfs.github.io/openzfs-docs/Getting%20Started/Arch%20Linux/Root%20on%20ZFS/4-optional-configuration.html#boot-environment-manager https://openzfs.github.io/openzfs-docs/Getting%20Started/Arc... zbm: https://github.com/zbm-dev/zfsbootmenu https://github.com/zbm-dev/zfsbootmenu zsys: https://github.com/ubuntu/zsys https://github.com/ubuntu/zsys
infogulch 5y ago
Can you somehow associate snapshots from separate machines? Lets say I have two servers that provide services for an app, one runs a postgresql db and the other is just a front for a directory that stores attachments (it's an example ok). Both servers are running ZFS and each service is running on its own volume which has regular snapshots taken. If I'm not too worried about the snapshots being in perfect sync (its ok if I have an extra attachment or have to pull an attachment from the next snapshot; I'm not asking for a solution to the distributed consensus problem here), is there some particular way I should store or manage the snapshots? Or do I need to write some custom scripts to associate the snapshots and restore them together in case of a recovery situation?
- justinsaccount 5y ago
  snapshots are usually just named based on the date, like 2021-07-29-10-hour 2021-07-29-10-53-minute or similar. Then it's just a matter of ensuring that you have NTP working and your clocks are in sync.
joshspankit 5y ago
I tuned out of ZFS when it was being designed, ducked my head in a few years later to read about 8GB+ RAM requirements, and have not tuned back in properly (don’t kick me off HN lol) My question is on snapshots vs file-specific backups: If a folder is set up for non-destructive writes, does it then need snapshots? Is there a benefit to using both snapshots and NDR in the field?
- unixhero 5y ago
  You need a lot of ram for de-duplication. It is not needed for general use. 128 bits man, you're missing out!!!
ewwhite 5y ago
I orchestrate ZFS snapshots using Monit and Syncoid. See: https://serverfault.com/a/842740/13325 https://serverfault.com/a/842740/13325

119 points by rodrigo975 5y ago

If you're using a recent Ubuntu, docker/zfs can lead to a large amount of snapshots. 'docker system prune' may help but has had zfs issues for me.

yjftsjthsd-h 5y ago
> but has had zfs issues for me. Can you give more details so I know what to watch out for? (Recently started running Docker+ZFS in prod, haven't seen it break yet but appreciate heads up)
- flemhans 5y ago
  The zfs storage driver isn't production ready yet.
  - r1ch 5y ago
    I love that they say this, then default to using it when ZFS is detected.
  - cosmin800 5y ago
    I am pretty sure Canonical will soon release a stable,production ready snap package for zfs.
    - alyandon 5y ago
      That gets updated arbitrarily at random intervals (because who doesn't want filesystem modules updated frequently!?!?!) and then purges logs every 72 hours so when you have a problem you'll never even know what was updated! I have to run snapd on one of my servers and I hate everything about it.
      - flemhans 5y ago
        I'm considering to run lxd as a snap because I want Debian 11 (for openzfs 2.0) and lxd at the same time. Am I in for a world of hurt?
        alyandon 5y ago
        I wouldn't say you are in for a world of hurt but you'll have to accept that the package will periodically update and in the process kill your containers/vms. My primary objections to snapd are: 1) updates are pushed/applied and services restarted on an arbitrary schedule which means you can't test updates prior to rolling them out or otherwise control when a service is taken offline by an update 2) snapd doesn't log to standard OS logging facilities and the logs it does have are cleared after 72 hours 3) snapd pollutes home directories with a directory called "snap" snapd is a poorly conceived project written by people that are profoundly ignorant of how real services and servers are actually operated. Despite having the above flaws repeatedly pointed out, the developers have shown zero interest in taking actual concrete actions to address them.
        flemhans 5y ago
        Thank you for the reply. It seems snapd is indeed a poorly conceived project. How would you work around it in a production setting if you HAD to use it? Auto-updating sounds like the biggest issue of those.
        zekrioca 5y ago
        Build LXD yourself. It is not that complicated if one updates go to something beyond 1.14. Only downside is that you may need to create your own init scripts and a lxd group, but that's not complicated either.
- anotherhue 5y ago
  there were a number of containers immune to 'docker rm', seemed it was trying to invoke 'zfs delete' on something that had already been removed (presumably moments ago as part of a previous step). Nasty workaround was to manually recreate the dataset (and children) so that it could remove them. The issue was common enough that I could google it so don't stress. More worryingly, 'prune' locked up for about 10 minutes before getting to do it's thing. pinning the cpu the whole time. This was on my devbox, so maybe it's my doing but did not inspire confidence. I've run FreeBSD ZFS for about a decade and openzfs/ubuntu in prod for kafka clusters so I'm no stranger - this docker usage was just particularly strange.
- e12e 5y ago
  (not op, but) I don't recall exactly what I experienced (in development- I think it was just terribly slow?) - I ended up carving out a volume (zvol) and formatting it with ext4 (or xfs?) for docker. It might actually have been this ms sql issue that led me to do that, actually: https://github.com/moby/moby/issues/33191 https://github.com/moby/moby/issues/33191
hddherman 5y ago
I've had bad experiences with Docker+Debian as well when the storage was on ZFS. I've had a much better experience by just keeping /var/log/containers on something simple, like ext4. Valuable data is still on ZFS and mounted inside the containers, no issues there.

mvanbaak 5y ago

'In this article, we’ll learn how ZFS boot environments (BEs) are used to boot the operating system itself into a previous point in time.' The whole article does not touch the subject of boot environments ... Maybe the author can update the page and state that the article is about managing snapshots instead of boot environments?

beermonster 5y ago
I read the entire article as I was interested in that aspect too. Misleading title.
- vermaden 5y ago
  Here: https://is.gd/BECTL https://is.gd/BECTL
- 1MachineElf 5y ago
  A common pattern with beadm on FreeBSD is for the root filesystem dataset to be called "ROOT", and this mirrors what's in the main article. It appears this person has boot environments enabled, but didn't actually explain anything about them. ZFS boot environments are awesome. You can get them on both FreeBSD and Illumos with beadm. On Linux there are a number of utilities, but I've only used ZSys. It comes working out of the box if you do a ZFS-on-root install with Ubuntu.
5wp3 5y ago
Boot environment is just a fancy term for root filesystem clones created from snapshots. Fyi: snapshots are read-only. Normal, read-writable filesystem can be created by `zfs clone root@snap newroot`.
vermaden 5y ago
Here: https://is.gd/BECTL https://is.gd/BECTL

barrkel 5y ago

I make snapshots every hour for 24 hours, every day for 7 days, every week for 4 weeks and every month for three months. This keeps the number of snapshots reasonable while keeping most stuff around should you accidentally delete or overwrite it. I can see a good argument for more fine grained snapshots (e.g. every few minutes) but I think I'd limit that to directories (and thus FSes) known to have frequent updates. I sometimes blow away all snapshots if I have a big clear out or rewrite of data, or else that stuff sticks around for months.

throw0101a 5y ago
> I can see a good argument for more fine grained snapshots (e.g. every few minutes) but I think I'd limit that to directories (and thus FSes) known to have frequent updates. Fun fact, the ZFS send-receive functionality was born with snapshots being created every few minutes in mind: > ZFS send and receive wasn’t considered until late in the development cycle. The idea came to me in 2005. ZFS was nearing integration, and I was spending a few months working in Sun’s new office in Beijing, China. The network link between Beijing and Menlo Park was low-bandwidth and high-latency, and our NFS-based source code manager was painful to use. I needed a way to quickly ship incremental changes to a workspace across the Pacific. A POSIX-based utility (like rsync) would at best have to traverse all the files and directories to find the few that were modified since a specific date, and at worst it would compare the files on each side, incurring many high-latency round trips. I realized that the block pointers in ZFS already have all the information we need: the birth time allows us to quickly and precisely find the blocks that are changed since a given snapshot. It was easiest to implement ZFS send at the DMU layer, just below the ZPL. This allows the semantically-important changes to be transferred exactly, without any special code to handle features like NFSv4 style ACLs, case-insensitivity, and extended attributes. Storage-specific settings, like compression and RAID type, can be different on the sending and receiving sides. What began as a workaround for a crappy network link has become one of the pillars of ZFS, and the foundation of several remote replication products, including the one at Delphix. * https://www.delphix.com/blog/delphix-engineering/zfs-10-year-anniversary https://www.delphix.com/blog/delphix-engineering/zfs-10-year...
- 5y ago

kissgyorgy 5y ago

ZFS snapshots saved my ass multiple times after a bad Ubuntu upgrade. It's pretty easy to roll back.

unixhero 5y ago
How do you roll back after a botched OS update then? Boot into a rescue cd?
- vermaden 5y ago
  Like that: https://is.gd/BECTL https://is.gd/BECTL By using ZFS Boot Environments.
  - unixhero 5y ago
    Woot, thanks a lot.

eptcyka 5y ago

Is there any good tooling on managing automatic snapshotting and sending backups?

ptomato 5y ago
sanoid (https://github.com/jimsalterjrs/sanoid/ https://github.com/jimsalterjrs/sanoid/) is probably the most featureful, but there's quite a few I'm aware of - zfsnap, zfs-auto-snapshot, zfs_autobackup, znapzend...
indiv0 5y ago
Personally I just setup sanoid and syncoid on my NAS. They were dead simple to get up and running with. Sanoid makes snapshots according to a schedule you define (e.g. 1 monthly, 31 daily, 24 hourly). It also prunes old snapshots to ensure that you always have the exact snapshots you need available and no more. Syncoid uses zfs send/recv to copy over snapshots from one ZFS dataset to another, typically between machines. You basically have two options: either you can: 1) copy over ALL of the snapshots from the source in the order they were created (like cherry picking one git commit after another onto another branch), or you can: 2) configure it to make a new snapshot every time it syncs and just copy over the diff since the last time you synced (a bit like squashing all the commits into one and just cherry-picking that). The former is good if you want to maintain the same snapshots on the source and the destination (you can also use sanoid on the destination to prune the snapshots further). The latter is good if you want to minimize the amount of churn sent over the network. For example if you're snapshotting VM images you'd be interested in sending over the state of the VM as it appears at the end of the day, rather than all the intermediate state.
waz0wski 5y ago
sanoid is a good one. I've been on a quest to run mostly golang-based utilities and switched to https://github.com/zrepl/zrepl https://github.com/zrepl/zrepl which has been running/syncing without issue for a couple years now
unixhero 5y ago
Proxmox backup node https://www.proxmox.com/en/proxmox-backup-server https://www.proxmox.com/en/proxmox-backup-server

pawelduda 5y ago

Does it make sense to snapshot the entire root dir or just the home directory? Asking about desktop PC.

tiernano 5y ago
Probably home did, maybe etc and var, maybe other important dirs... full root would include temp and logs that may not be needed and folders that don't change often
solarengineer 5y ago
For Solaris 11 and for Illumos based distros, snapshotting the entire root is normal. That way, with an OS update and application update, the logs are associated with that particular root file system. You can always look into earlier snapshots to learn of what the logs contained for that earlier root file system. With Boot Environments (I don't know if Linux based Distros let one do this) on Solaris and Illumos, we have multiple active root file systems to switch between and can mark snapshots for each of them. Certainly, for Belenix (KDE based opensolaris distro), we snapshotted the entire root file system.
- throw0101a 5y ago
  > For Solaris 11 and for Illumos based distros, snapshotting the entire root is normal. FreeBSD stole this feature from Solaris as well: * https://www.freebsd.org/cgi/man.cgi?beadm https://www.freebsd.org/cgi/man.cgi?beadm * https://wiki.freebsd.org/BootEnvironments https://wiki.freebsd.org/BootEnvironments
  - vermaden 5y ago
    ... and Illumos stole Bhyve and Bootloader from FreeBSD? :) IMHO implemented or ported is better word here.
- 5wp3 5y ago
  This is implemented for Arch Linux with bieaz or zbm; and Ubuntu with zsys. bieaz: https://openzfs.github.io/openzfs-docs/Getting%20Started/Arch%20Linux/Root%20on%20ZFS/4-optional-configuration.html#boot-environment-manager https://openzfs.github.io/openzfs-docs/Getting%20Started/Arc... zbm: https://github.com/zbm-dev/zfsbootmenu https://github.com/zbm-dev/zfsbootmenu zsys: https://github.com/ubuntu/zsys https://github.com/ubuntu/zsys

infogulch 5y ago

Can you somehow associate snapshots from separate machines? Lets say I have two servers that provide services for an app, one runs a postgresql db and the other is just a front for a directory that stores attachments (it's an example ok). Both servers are running ZFS and each service is running on its own volume which has regular snapshots taken. If I'm not too worried about the snapshots being in perfect sync (its ok if I have an extra attachment or have to pull an attachment from the next snapshot; I'm not asking for a solution to the distributed consensus problem here), is there some particular way I should store or manage the snapshots? Or do I need to write some custom scripts to associate the snapshots and restore them together in case of a recovery situation?

justinsaccount 5y ago
snapshots are usually just named based on the date, like 2021-07-29-10-hour 2021-07-29-10-53-minute or similar. Then it's just a matter of ensuring that you have NTP working and your clocks are in sync.

joshspankit 5y ago

I tuned out of ZFS when it was being designed, ducked my head in a few years later to read about 8GB+ RAM requirements, and have not tuned back in properly (don’t kick me off HN lol) My question is on snapshots vs file-specific backups: If a folder is set up for non-destructive writes, does it then need snapshots? Is there a benefit to using both snapshots and NDR in the field?

unixhero 5y ago
You need a lot of ram for de-duplication. It is not needed for general use. 128 bits man, you're missing out!!!

ewwhite 5y ago

I orchestrate ZFS snapshots using Monit and Syncoid. See: https://serverfault.com/a/842740/13325 https://serverfault.com/a/842740/13325