5 ms·

This is the piece I was looking for: "Which would push-replicate the specified ZFS filesystem from the local host to remotehost over an SSH tunnel ..." sy

by rsync 7y ago

This is the piece I was looking for: "Which would push-replicate the specified ZFS filesystem from the local host to remotehost over an SSH tunnel ..." syncoid root@remotehost:data/images/vm backup/images/vm Now, if only there were a cloud storage provider that you could directly zfs-send to ... If only ...

dangerboysteve 7y ago
like rsync.net ? this is just storage.
- icebraining 7y ago
  The previous user runs rsync.net, he's pushing his service.
  - dangerboysteve 7y ago
    Totally missed that.
dmd 7y ago
If only there were one that were a reasonable price increment on top of raw storage, like 1.5x or even 2x, instead of 5x what, say, S3 or B2 costs.
- cpach 7y ago
  Feel free to start a competitor…
  - papyruskaleido 7y ago
    It's probably easier and cheaper to just use Sanoid though.
- blattimwind 7y ago
  Remote access to a ZFS gives you much different guarantees from your average S3-lookalike, so an integer price difference does not seem unreasonable to me.
  - dmd 7y ago
    2 is an integer! 3 is too...
- rsync 7y ago
  S3 currently[1] costs 2.3c per GB, per month and there are additional costs for various transfer and usage behaviors. rsync.net pricing is 2.5c per GB, per month with no other charges possible, regardless of usage. I feel like we're in the ballpark there ... As a bonus, you can do things like this: ssh user@rsync.net rclone s3:/some/bucket rsync/home/dir ssh user@rsync.net rclone gdrive:/blah s3:/some/bucket ... which is detailed here[2]. [1] https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/ [2] https://www.rsync.net/products/universal.html https://www.rsync.net/products/universal.html
  - dmd 7y ago
    Huh. I only use B2 (which is 0.5c per GB), and I had simply assumed that S3 was in roughly the same ballpark. Ok then!
kuon 7y ago
I've been interested into rsync.net. How does it work if I want my remote ZFS dataset to be encrypted without leaving the key on the remote server?
- e12e 7y ago
  Which remote server? You would only need the key at the remote server, if you wanted to mount the snapshot?
- RJIb8RBYxzAMX9u 7y ago
  https://github.com/openzfs/zfs/commit/b52563034230b35f0562b6f40ad1a00f02bd9a05 https://github.com/openzfs/zfs/commit/b52563034230b35f0562b6... "The last addition is the ability to do raw, encrypted sends and receives. The idea here is to send raw encrypted and compressed data and receive it exactly as is on a backup system. This means that the dataset on the receiving system is protected using the same user key that is in use on the sending side. By doing so, datasets can be efficiently backed up to an untrusted system without fear of data being compromised." Unfortunately it's taking a long time for the feature to propagate through the ZFS implementations. I don't know if rsync.net's servers support it.
- rsync 7y ago
  "I've been interested into rsync.net. How does it work if I want my remote ZFS dataset to be encrypted without leaving the key on the remote server?" If you have multiple terabytes of data, zfs-send is a very efficient (and elegant) way to transfer them to rsync.net. We can give you your own zpool, with root, running the latest production/stable ZoL code - so you will have encryption and raw send, etc. However ... If you have hundreds of GB up to ... one or two TB ... we paste this suggestion to you instead: I think you should look into the 'borg' backup tool - it has become the de facto standard for remote backups because it does everything that rsync does (efficient, changes only backups) but also produces strongly encrypted remote backup sets that only you have a key to ... rsync.net has no access to the data. The borg website is here: https://borgbackup.readthedocs.io/en/stable/ https://borgbackup.readthedocs.io/en/stable/ and a good description of how it works and why you should use it is here: https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/ ... and our own page with details of our attic/borg support, along with a deeply discounted signup link, is here: http://rsync.net/products/borg.html http://rsync.net/products/borg.html
jimmcslim 7y ago
I have an Ubuntu server with a ZFS pool that I want to rebuild as a FreeNAS/TrueNAS server. Can I use something like rsync.net to push my datasets up as a backup, rebuild my server, and then pull them back down into a reorganised/renamed tree of datasets? Are there compatibility issues between ZFS on Linux and ZFS on FreeNAS that I need to be concerned about? (that come into play when pushing/pulling datasets)
- ses1984 7y ago
  Just out of curiosity why do you want to rebuild this as a FreeNAS server?
  - jimmcslim 7y ago
    I should also mention that this is actually an Ubuntu VM running in Hyper-V on a Windows Server 2012 host... don’t ask! It needs to be rebuilt one way or another (and it is running on an HP Microserver N54L!) As for FreeNAS, I feel it gives the closest thing to a Synology NAS in terms of manageability through a GUI, etc, while also supporting ZFS.
    - ses1984 7y ago
      Ahh, I don't have the hardware to run zfs in a VM, for home I run a big zfs array on bare metal and also host some vms next to it. If I were running zfs in a VM itself I would be more open to freenas. Maintaining freenas and trying to run some other virtualized apps next to it was not good for me.
- e12e 7y ago
  > Can I use something like rsync.net to > push my datasets up as a backup Yes, but depending on the size/value of the dataset, rsync might not be the best option. Even after recent price reduction (Yay!), and with a hn discount (Yay!) - I still consider rsync.net as a personal backup service a little overkill/expensive past 100 GB or so. And likely you could just handle anything on the order of 1 to 2 tb locally anyway (spare disks). You technically could use it for 10+tb of data... But uploading and downloading is going to be slow unless you have a real uplink (ie at least 1gbps). And if you did, unless you need filesystem level Metadata - rsync might be a better option than zfs send/receive (even though zfs is cool). >, rebuild my server, and then pull them back down into a reorganised/renamed tree of datasets? Again, if you just need the data back - you might be well served with rsync or restic (both choices should be orthogonal to using rsync). > Are there compatibility issues between ZFS on Linux and ZFS on FreeNAS that I need to be concerned about? The world is divided in Oracle and not-Oracle - AFAIK there shouldn't be major differences between those two - that I believe now both follow upstream zfsonlinux. However - I believe rsync.net still doesn't support zfsonlinux/openzfs encryption (zfsonlinux 0.8+). While locally it's possible to encrypt the disk "under" zfs... Native encryption is tempting. And you would of course never send anyone data without encrypting it?
deleted 7y ago
[deleted]
hartzell 7y ago
[edit, of course moments after posting this I found this link https://www.rsync.net/products/zfsintro.html https://www.rsync.net/products/zfsintro.html in the Arstechnica article. Still can't find it linked from the site though...] > Now, if only there were a cloud storage provider that you could directly zfs-send to ... Is there something on rsync.net's site that describes its support for zfs send? I've found this old thread on HN referring to an Arstechnica article: https://news.ycombinator.com/item?id=10751269 https://news.ycombinator.com/item?id=10751269 and a reference to it on IXSystem's site: https://www.ixsystems.com/community/threads/rsync-net-now-supports-zfs-send-and-receive-on-their-cloud-storage-platform.35759/ https://www.ixsystems.com/community/threads/rsync-net-now-su... but they're both back in 2015. I can't find anything on the site, https://www.rsync.net/index.html https://www.rsync.net/index.html, but I may be reading right past it.
- rodgerd 7y ago
  (As an aside, I'm grateful they've finally gotten over their absurd pricing notation)

by rsync 7y ago

dangerboysteve 7y ago

like rsync.net ? this is just storage.

icebraining 7y ago
The previous user runs rsync.net, he's pushing his service.
- dangerboysteve 7y ago
  Totally missed that.

dmd 7y ago

If only there were one that were a reasonable price increment on top of raw storage, like 1.5x or even 2x, instead of 5x what, say, S3 or B2 costs.

cpach 7y ago
Feel free to start a competitor…
- papyruskaleido 7y ago
  It's probably easier and cheaper to just use Sanoid though.
blattimwind 7y ago
Remote access to a ZFS gives you much different guarantees from your average S3-lookalike, so an integer price difference does not seem unreasonable to me.
- dmd 7y ago
  2 is an integer! 3 is too...
rsync 7y ago
S3 currently[1] costs 2.3c per GB, per month and there are additional costs for various transfer and usage behaviors. rsync.net pricing is 2.5c per GB, per month with no other charges possible, regardless of usage. I feel like we're in the ballpark there ... As a bonus, you can do things like this: ssh user@rsync.net rclone s3:/some/bucket rsync/home/dir ssh user@rsync.net rclone gdrive:/blah s3:/some/bucket ... which is detailed here[2]. [1] https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/ [2] https://www.rsync.net/products/universal.html https://www.rsync.net/products/universal.html
- dmd 7y ago
  Huh. I only use B2 (which is 0.5c per GB), and I had simply assumed that S3 was in roughly the same ballpark. Ok then!

kuon 7y ago

I've been interested into rsync.net. How does it work if I want my remote ZFS dataset to be encrypted without leaving the key on the remote server?

e12e 7y ago
Which remote server? You would only need the key at the remote server, if you wanted to mount the snapshot?
RJIb8RBYxzAMX9u 7y ago
https://github.com/openzfs/zfs/commit/b52563034230b35f0562b6f40ad1a00f02bd9a05 https://github.com/openzfs/zfs/commit/b52563034230b35f0562b6... "The last addition is the ability to do raw, encrypted sends and receives. The idea here is to send raw encrypted and compressed data and receive it exactly as is on a backup system. This means that the dataset on the receiving system is protected using the same user key that is in use on the sending side. By doing so, datasets can be efficiently backed up to an untrusted system without fear of data being compromised." Unfortunately it's taking a long time for the feature to propagate through the ZFS implementations. I don't know if rsync.net's servers support it.
rsync 7y ago
"I've been interested into rsync.net. How does it work if I want my remote ZFS dataset to be encrypted without leaving the key on the remote server?" If you have multiple terabytes of data, zfs-send is a very efficient (and elegant) way to transfer them to rsync.net. We can give you your own zpool, with root, running the latest production/stable ZoL code - so you will have encryption and raw send, etc. However ... If you have hundreds of GB up to ... one or two TB ... we paste this suggestion to you instead: I think you should look into the 'borg' backup tool - it has become the de facto standard for remote backups because it does everything that rsync does (efficient, changes only backups) but also produces strongly encrypted remote backup sets that only you have a key to ... rsync.net has no access to the data. The borg website is here: https://borgbackup.readthedocs.io/en/stable/ https://borgbackup.readthedocs.io/en/stable/ and a good description of how it works and why you should use it is here: https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/ ... and our own page with details of our attic/borg support, along with a deeply discounted signup link, is here: http://rsync.net/products/borg.html http://rsync.net/products/borg.html

jimmcslim 7y ago

I have an Ubuntu server with a ZFS pool that I want to rebuild as a FreeNAS/TrueNAS server. Can I use something like rsync.net to push my datasets up as a backup, rebuild my server, and then pull them back down into a reorganised/renamed tree of datasets? Are there compatibility issues between ZFS on Linux and ZFS on FreeNAS that I need to be concerned about? (that come into play when pushing/pulling datasets)

ses1984 7y ago
Just out of curiosity why do you want to rebuild this as a FreeNAS server?
- jimmcslim 7y ago
  I should also mention that this is actually an Ubuntu VM running in Hyper-V on a Windows Server 2012 host... don’t ask! It needs to be rebuilt one way or another (and it is running on an HP Microserver N54L!) As for FreeNAS, I feel it gives the closest thing to a Synology NAS in terms of manageability through a GUI, etc, while also supporting ZFS.
  - ses1984 7y ago
    Ahh, I don't have the hardware to run zfs in a VM, for home I run a big zfs array on bare metal and also host some vms next to it. If I were running zfs in a VM itself I would be more open to freenas. Maintaining freenas and trying to run some other virtualized apps next to it was not good for me.
e12e 7y ago
> Can I use something like rsync.net to > push my datasets up as a backup Yes, but depending on the size/value of the dataset, rsync might not be the best option. Even after recent price reduction (Yay!), and with a hn discount (Yay!) - I still consider rsync.net as a personal backup service a little overkill/expensive past 100 GB or so. And likely you could just handle anything on the order of 1 to 2 tb locally anyway (spare disks). You technically could use it for 10+tb of data... But uploading and downloading is going to be slow unless you have a real uplink (ie at least 1gbps). And if you did, unless you need filesystem level Metadata - rsync might be a better option than zfs send/receive (even though zfs is cool). >, rebuild my server, and then pull them back down into a reorganised/renamed tree of datasets? Again, if you just need the data back - you might be well served with rsync or restic (both choices should be orthogonal to using rsync). > Are there compatibility issues between ZFS on Linux and ZFS on FreeNAS that I need to be concerned about? The world is divided in Oracle and not-Oracle - AFAIK there shouldn't be major differences between those two - that I believe now both follow upstream zfsonlinux. However - I believe rsync.net still doesn't support zfsonlinux/openzfs encryption (zfsonlinux 0.8+). While locally it's possible to encrypt the disk "under" zfs... Native encryption is tempting. And you would of course never send anyone data without encrypting it?

deleted 7y ago

[deleted]

hartzell 7y ago

[edit, of course moments after posting this I found this link https://www.rsync.net/products/zfsintro.html https://www.rsync.net/products/zfsintro.html in the Arstechnica article. Still can't find it linked from the site though...] > Now, if only there were a cloud storage provider that you could directly zfs-send to ... Is there something on rsync.net's site that describes its support for zfs send? I've found this old thread on HN referring to an Arstechnica article: https://news.ycombinator.com/item?id=10751269 https://news.ycombinator.com/item?id=10751269 and a reference to it on IXSystem's site: https://www.ixsystems.com/community/threads/rsync-net-now-supports-zfs-send-and-receive-on-their-cloud-storage-platform.35759/ https://www.ixsystems.com/community/threads/rsync-net-now-su... but they're both back in 2015. I can't find anything on the site, https://www.rsync.net/index.html https://www.rsync.net/index.html, but I may be reading right past it.

rodgerd 7y ago
(As an aside, I'm grateful they've finally gotten over their absurd pricing notation)