6 ms·

Just a reminder, as this is sometimes confusing - a plain old rsync.net account is running on a ZFS filesystem and has all of the benefits that that entails (po

by rsync 4y ago

Just a reminder, as this is sometimes confusing - a plain old rsync.net account is running on a ZFS filesystem and has all of the benefits that that entails (point in time snapshots, checksummed integrity, CoW efficiency, etc.) but they do not have their own zpool to themselves. If you want to zfs send to an rsync.net account you need to have an account enabled for that which is the same price, but has a higher (4TB) minimum account size.

LiamMcCalloway 4y ago
I just set up zrepl yesterday and this was going to be my next task. Super glad to find this here. I already have a borg-enabled account with you: Is it possible for a single account to do both borg and zfs send?
ssl232 4y ago
Out of curiosity, why do you insist on a minimum of 4TB for zfs send enabled accounts? (Happy rsync.net customer btw)
- mekster 4y ago
  That's because their zfs account requires a dedicated IP and a VM. It used to be 1TB minimum until pretty recently but they went with 4TB now. If you want a less space solution, just spin up a t3.small (I suppose at least 2GB memory would be better for zfs) on AWS as Ubuntu with Cold HDD (minimum 125GB) as extra storage which is quite cheap at about 0.015/GB and install "zfsutils-linux" and you're good to go to use zfs.
  - Nextgrid 4y ago
    Keep in mind that AWS's outbound bandwidth charges will bite you if you need to retrieve that data later. It's probably fine for a true "last resort" backup, but if you're frequently restoring it's worth keeping in mind.
waynesonfire 4y ago
4TB? That's $60 per month. Mmkay. Just use duplicati + cloud cold storage. Way cheaper. Cold storage has a complicated pricing model because it's al-a-carte, but offsite backups tend to be write-once-read-never. $0.0036 per GB / Month for aws $0.007 per GB-month for storage for gcs
- easton 4y ago
  > write-once-read-never Well, if you're being smart, it's going to be write-once-read-sometimes-once because you should be testing your backups every so often. You'd probably want to backup to some kind of service in the middle, then at the end of the month/quarter ship a backup to cold storage.
- kqr 4y ago
  Way cheaper if all you want is vendor-specific cloud cold storage, yes. If you want a generic hot remote zfs filesystem it's not even a competition.
  - waynesonfire 4y ago
    The abstraction is duplicati. It's trivial to change the back-end so I'm not sure I'd be too concerned about vendor-lock. If you want a generic remote zfs fs it's going to cost $$$. I have that but it's not offsite.
- mekster 4y ago
  If you think duplicati (or pretty much most maybe except Borg) is same as zfs, you're badly wrong. If you don't care about stability or reliability, choose whatever the landing page sells you.
- rsync 4y ago
  "4TB? That's $60 per month. Mmkay. Just use duplicati + cloud cold storage. Way cheaper." This is random access, live storage - which is not the case with cold storage options such as Glacier. A more fitting comparison would be S3 ... I think you are, in many cases, correct - offsite backups can, indeed, be write-once ... but if you're looking for the very specific efficiencies and features that zfs-send affords you, that will not be the case.
infogulch 4y ago
You've discussed why a zpool account has a relatively high minimum in the past [1]. Basically because you need a persistent isolated vm/bhyve with sshd listening on its own ip and separate zfs and zpool for each customer. It's still fuzzy to me specifically why a dedicated 'vm' is necessary. You don't need a dedicated ssh ip for normal 'shell' accounts, and certainly you still provide security and privacy for these customers. Is it because zfs doesn't have fine-grained permissions to allow doing zfs send/recv actions on one's own datasets/volumes without also giving access to other customers' data? Or is it because send/recv workloads just consume more compute resources in general? [1]: https://news.ycombinator.com/item?id=27994369#28000100 https://news.ycombinator.com/item?id=27994369#28000100
- rsync 4y ago
  I myself am still fuzzy as to why we need a VM but it is, indeed, due to the permissions that ZFS requires to do a 'zfs destroy' ... which you need in order to manage your own snapshots. So, basically, you need to be root to fully manage your own zpool ... and if you need to be root, you need to be in a VM. In our case, we use bhyve and we have had very good success. We have this on very good authority - Allan Jude of Klara Systems has helped us audit the entire setup and there is not, unfortunately, a lighter way to do it.
  - infogulch 4y ago
    Thank you for clarifying! From zfs-allow.8 docs Example 2 [1] it looks like it should be possible, but I guess there are important details omitted from that document that make such a design nonviable in your case. I'm glad to hear that you are using system designs vetted by zfs experts. [1]: https://openzfs.github.io/openzfs-docs/man/8/zfs-allow.8.html#EXAMPLES https://openzfs.github.io/openzfs-docs/man/8/zfs-allow.8.htm...

by rsync 4y ago

LiamMcCalloway 4y ago

I just set up zrepl yesterday and this was going to be my next task. Super glad to find this here. I already have a borg-enabled account with you: Is it possible for a single account to do both borg and zfs send?

ssl232 4y ago

Out of curiosity, why do you insist on a minimum of 4TB for zfs send enabled accounts? (Happy rsync.net customer btw)

mekster 4y ago
That's because their zfs account requires a dedicated IP and a VM. It used to be 1TB minimum until pretty recently but they went with 4TB now. If you want a less space solution, just spin up a t3.small (I suppose at least 2GB memory would be better for zfs) on AWS as Ubuntu with Cold HDD (minimum 125GB) as extra storage which is quite cheap at about 0.015/GB and install "zfsutils-linux" and you're good to go to use zfs.
- Nextgrid 4y ago
  Keep in mind that AWS's outbound bandwidth charges will bite you if you need to retrieve that data later. It's probably fine for a true "last resort" backup, but if you're frequently restoring it's worth keeping in mind.

waynesonfire 4y ago

4TB? That's $60 per month. Mmkay. Just use duplicati + cloud cold storage. Way cheaper. Cold storage has a complicated pricing model because it's al-a-carte, but offsite backups tend to be write-once-read-never. $0.0036 per GB / Month for aws $0.007 per GB-month for storage for gcs

easton 4y ago
> write-once-read-never Well, if you're being smart, it's going to be write-once-read-sometimes-once because you should be testing your backups every so often. You'd probably want to backup to some kind of service in the middle, then at the end of the month/quarter ship a backup to cold storage.
kqr 4y ago
Way cheaper if all you want is vendor-specific cloud cold storage, yes. If you want a generic hot remote zfs filesystem it's not even a competition.
- waynesonfire 4y ago
  The abstraction is duplicati. It's trivial to change the back-end so I'm not sure I'd be too concerned about vendor-lock. If you want a generic remote zfs fs it's going to cost $$$. I have that but it's not offsite.
mekster 4y ago
If you think duplicati (or pretty much most maybe except Borg) is same as zfs, you're badly wrong. If you don't care about stability or reliability, choose whatever the landing page sells you.
rsync 4y ago
"4TB? That's $60 per month. Mmkay. Just use duplicati + cloud cold storage. Way cheaper." This is random access, live storage - which is not the case with cold storage options such as Glacier. A more fitting comparison would be S3 ... I think you are, in many cases, correct - offsite backups can, indeed, be write-once ... but if you're looking for the very specific efficiencies and features that zfs-send affords you, that will not be the case.

infogulch 4y ago

You've discussed why a zpool account has a relatively high minimum in the past [1]. Basically because you need a persistent isolated vm/bhyve with sshd listening on its own ip and separate zfs and zpool for each customer. It's still fuzzy to me specifically why a dedicated 'vm' is necessary. You don't need a dedicated ssh ip for normal 'shell' accounts, and certainly you still provide security and privacy for these customers. Is it because zfs doesn't have fine-grained permissions to allow doing zfs send/recv actions on one's own datasets/volumes without also giving access to other customers' data? Or is it because send/recv workloads just consume more compute resources in general? [1]: https://news.ycombinator.com/item?id=27994369#28000100 https://news.ycombinator.com/item?id=27994369#28000100

rsync 4y ago
I myself am still fuzzy as to why we need a VM but it is, indeed, due to the permissions that ZFS requires to do a 'zfs destroy' ... which you need in order to manage your own snapshots. So, basically, you need to be root to fully manage your own zpool ... and if you need to be root, you need to be in a VM. In our case, we use bhyve and we have had very good success. We have this on very good authority - Allan Jude of Klara Systems has helped us audit the entire setup and there is not, unfortunately, a lighter way to do it.
- infogulch 4y ago
  Thank you for clarifying! From zfs-allow.8 docs Example 2 [1] it looks like it should be possible, but I guess there are important details omitted from that document that make such a design nonviable in your case. I'm glad to hear that you are using system designs vetted by zfs experts. [1]: https://openzfs.github.io/openzfs-docs/man/8/zfs-allow.8.html#EXAMPLES https://openzfs.github.io/openzfs-docs/man/8/zfs-allow.8.htm...