3 ms·

For anyone who might be interested in doing this with fewer dependencies/tooling I have written a bash script that pushes ZFS snapshots to S3: https://github.c

by agurk 6y ago

For anyone who might be interested in doing this with fewer dependencies/tooling I have written a bash script that pushes ZFS snapshots to S3: https://github.com/agurk/zfs-to-aws https://github.com/agurk/zfs-to-aws I have it running in a cronjob on my NAS nightly, and a retention policy on aws to push it straight into glacier. There's no restore script as this is my last line of backups, so I thought if I was restoring from here I'd want to do it manually. I might get around to writing one now however.

geek_at 6y ago
how do you handle encryption? can you add it before storing it to AWS or do you trust your unencrypted data to amazon?
- agurk 6y ago
  Native (per-dataset) encryption. This has been a possibility since ZoL 0.8.0 [0] released two years ago. This means that the stream created by zfs send is seamlessly encrypted. I waited until this feature was released/stable before I started uploading to AWS. Practically you need to add the -w or --raw flag to zfs send to make it do this. [0] https://github.com/openzfs/zfs/releases/tag/zfs-0.8.0 https://github.com/openzfs/zfs/releases/tag/zfs-0.8.0
  - nickik 6y ago
    But if my dataset is not encrypted already (legacy dataset), it will not be encrypted?
    - agurk 6y ago
      That is correct, this script will not add any encryption. It would be trivial to add however by adding gpg into the send command pipeline: /sbin/zfs send --raw -cp $snapshot | pv -s $snapshot_size | aws s3 cp - <args> would become: /sbin/zfs send --raw -cp $snapshot | gpg --encrypt <args> | pv -s $snapshot_size | aws s3 cp - <args>
  - charcircuit 6y ago
    Doesn't that only encrypt the data, leaking stuff like filenames, or has that been mitigated?
    - agurk 6y ago
      From the zfs man page: > ZFS will encrypt file and zvol data, file attributes, ACLs, permission bits, directory listings, FUID mappings, and userused / groupused data. I don't believe that the native encryption ever leaked file listings. That could have been something that the older/non-native techniques did.
      - charcircuit 6y ago
        I was probably thinking of file sizes not being encrypted. Thanks for correcting me.

by agurk 6y ago

geek_at 6y ago

how do you handle encryption? can you add it before storing it to AWS or do you trust your unencrypted data to amazon?

agurk 6y ago
Native (per-dataset) encryption. This has been a possibility since ZoL 0.8.0 [0] released two years ago. This means that the stream created by zfs send is seamlessly encrypted. I waited until this feature was released/stable before I started uploading to AWS. Practically you need to add the -w or --raw flag to zfs send to make it do this. [0] https://github.com/openzfs/zfs/releases/tag/zfs-0.8.0 https://github.com/openzfs/zfs/releases/tag/zfs-0.8.0
- nickik 6y ago
  But if my dataset is not encrypted already (legacy dataset), it will not be encrypted?
  - agurk 6y ago
    That is correct, this script will not add any encryption. It would be trivial to add however by adding gpg into the send command pipeline: /sbin/zfs send --raw -cp $snapshot | pv -s $snapshot_size | aws s3 cp - <args> would become: /sbin/zfs send --raw -cp $snapshot | gpg --encrypt <args> | pv -s $snapshot_size | aws s3 cp - <args>
- charcircuit 6y ago
  Doesn't that only encrypt the data, leaking stuff like filenames, or has that been mitigated?
  - agurk 6y ago
    From the zfs man page: > ZFS will encrypt file and zvol data, file attributes, ACLs, permission bits, directory listings, FUID mappings, and userused / groupused data. I don't believe that the native encryption ever leaked file listings. That could have been something that the older/non-native techniques did.
    - charcircuit 6y ago
      I was probably thinking of file sizes not being encrypted. Thanks for correcting me.