12 ms·

Everyone tries to reinvent the wheel with zfs mirroring and snapshot retention tools.

by feld 11y ago

binwiederhier 11y ago
Any suggestions? I looked for a while but couldn't find anything so I adapted this script which a guy on IRC linked me to.
- mbreese 11y ago
  I don't have any suggestions, and I don't suspect that are anyone else will either. Why? Because ZFS backup and mirroring is going to be so system specific, that it is better left to one-off scripts written by the admin. For example, your zfsu tx script - if I'm reading it right, you assumes that root can ssh from the backup server to the target server without a password. This is rarely the case (and I'd argue that it should never be the case). But, this is something that works on your systems... The last time I had to setup ZFS snapshot mirroring, I did something similar, but it was so specific to my setup that it wasn't worth making it into a package that others would use. Each of the backup targets had a "backup" user with a password-less SSH key that could execute exactly one command with sudo (zfs send). That's not exactly something that lends itself to a simple install script. Instead, it probably would be more beneficial to write a blog post somewhere that detailed why you needed these scripts, and the one or two magic lines in them that got the job done. The most I remember finding were the original Sun/Oracle ZFS docs.
  - feld 11y ago
    I disagree completely. And I really don't trust someone to write zfs replication scripts on their own that can properly handle differentials automatically or the new resumable-send feature which this guy's tool doesn't even mention because he probably doesn't even know it exists. Listen, this is hard and it sucks that there isn't a tool that comes with zfs to do this for you. But there are scripts that have been in production for years that have never failed. Scripts that are currently pushing hundreds of GBs per hour across the internet and recovering gracefully from failures. Use them. They work. They're even authored by zfs experts.
    - mbreese 11y ago
      Yes, there are best-practices that should be used and advertised. But there is no way to have a generic tool that will work in some reasonable percentage of environments. Yes, it's hard, but it's better to teach people how to manage it instead of giving them an unmodifiable tool and just say "here - use this". Telling them "here - do it like this", is much better.
dang 11y ago
Please don't be dismissive of new work on HN. It would be fine to mention prior work and ask how this relates, or to ask what prompted the author to build something new. But a generic dismissal like "everyone tries to reinvent the wheel" is just a slap, and that goes against the culture we want here, especially in Show HN threads: https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html
kondro 11y ago
What are the established alternatives? ZFS has tools do execute the actual snapshots and to do the actual mirroring, but nothing to manage which snapshots to keep or how to actually reliably get a snapshot from one place to another.
- mbreese 11y ago
  Just use SSH with custom scripts. It's pretty much part of the original ZFS documentation: http://docs.oracle.com/cd/E19253-01/819-5461/gbinw/ http://docs.oracle.com/cd/E19253-01/819-5461/gbinw/
- feld 11y ago
  zxfer https://github.com/allanjude/zxfer https://github.com/allanjude/zxfer zfstools https://github.com/bdrewery/zfstools https://github.com/bdrewery/zfstools zfsnap https://github.com/zfsnap/zfsnap https://github.com/zfsnap/zfsnap
  - waz0wski 11y ago
    sanoid https://github.com/jimsalterjrs/sanoid https://github.com/jimsalterjrs/sanoid

Everyone tries to reinvent the wheel with zfs mirroring and snapshot retention tools.

by feld 11y ago

Everyone tries to reinvent the wheel with zfs mirroring and snapshot retention tools.

Any suggestions? I looked for a while but couldn't find anything so I adapted this script which a guy on IRC linked me to.

mbreese 11y ago
I don't have any suggestions, and I don't suspect that are anyone else will either. Why? Because ZFS backup and mirroring is going to be so system specific, that it is better left to one-off scripts written by the admin. For example, your zfsu tx script - if I'm reading it right, you assumes that root can ssh from the backup server to the target server without a password. This is rarely the case (and I'd argue that it should never be the case). But, this is something that works on your systems... The last time I had to setup ZFS snapshot mirroring, I did something similar, but it was so specific to my setup that it wasn't worth making it into a package that others would use. Each of the backup targets had a "backup" user with a password-less SSH key that could execute exactly one command with sudo (zfs send). That's not exactly something that lends itself to a simple install script. Instead, it probably would be more beneficial to write a blog post somewhere that detailed why you needed these scripts, and the one or two magic lines in them that got the job done. The most I remember finding were the original Sun/Oracle ZFS docs.
- feld 11y ago
  I disagree completely. And I really don't trust someone to write zfs replication scripts on their own that can properly handle differentials automatically or the new resumable-send feature which this guy's tool doesn't even mention because he probably doesn't even know it exists. Listen, this is hard and it sucks that there isn't a tool that comes with zfs to do this for you. But there are scripts that have been in production for years that have never failed. Scripts that are currently pushing hundreds of GBs per hour across the internet and recovering gracefully from failures. Use them. They work. They're even authored by zfs experts.
  - mbreese 11y ago
    Yes, there are best-practices that should be used and advertised. But there is no way to have a generic tool that will work in some reasonable percentage of environments. Yes, it's hard, but it's better to teach people how to manage it instead of giving them an unmodifiable tool and just say "here - use this". Telling them "here - do it like this", is much better.

dang 11y ago

Please don't be dismissive of new work on HN. It would be fine to mention prior work and ask how this relates, or to ask what prompted the author to build something new. But a generic dismissal like "everyone tries to reinvent the wheel" is just a slap, and that goes against the culture we want here, especially in Show HN threads: https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html

kondro 11y ago

What are the established alternatives? ZFS has tools do execute the actual snapshots and to do the actual mirroring, but nothing to manage which snapshots to keep or how to actually reliably get a snapshot from one place to another.

mbreese 11y ago
Just use SSH with custom scripts. It's pretty much part of the original ZFS documentation: http://docs.oracle.com/cd/E19253-01/819-5461/gbinw/ http://docs.oracle.com/cd/E19253-01/819-5461/gbinw/
feld 11y ago
zxfer https://github.com/allanjude/zxfer https://github.com/allanjude/zxfer zfstools https://github.com/bdrewery/zfstools https://github.com/bdrewery/zfstools zfsnap https://github.com/zfsnap/zfsnap https://github.com/zfsnap/zfsnap
- waz0wski 11y ago
  sanoid https://github.com/jimsalterjrs/sanoid https://github.com/jimsalterjrs/sanoid