7 ms·

Rsync.net: ZFS Replication to the cloud is here and fast

125 points by wstrange 11y ago

coldpie 11y ago
I loved reading this article because it's just pure nerd joy. The guy is so enthusiastic about rsync, measurement, and unix, just reading the article is fun, even if I don't have a use for this particular tool.
matt2000 11y ago
Really fun to read article, but could someone give me a couple examples of what kind of system you'd use this with? I guess I'm so used to S3 now that I've forgotten what it's like to manage your own files.
- seiji 11y ago
  ZFS supports (and excels at) two necessary things for a good file system: copy on write snapshots and filesystem serialization. ZFS is designed to run one filesystem per user, so ZFS is designed to scale crazy high with little management overhead. It also makes it easy for snapshots to equal isolated per-user or even per-appication backups. You can do neat things like: create a base filesystem, create a snapshot of it (i.e. "freeze" the filesystem in time), send the snapshot to another system (or over the network for S3 storage, etc), continue modifying files, create another snapshot on top of your first one, now you can send/backup the new snapshot which is only a diff of the original snapshot. Many other systems besides ZFS support snapshots/copy-on-write layers, but they are mostly hacks and your performance will drop considerably during the "snapshot existing" time. ZFS is different because the snapshot mechanism doesn't kill your performance (unlike the 18 layer Linux file system organization). "Sending" a snapshot can either be directly over the network to another system to ingest the snapshot (enabling you to stream a new filesystem/snapshot into existence on a remote machine), or you can gzip the snapshot stream and send it off to backup storage. You can layer your snapshot diffs as much as you like. As long as you have all the diffs back to the original file system, you can re-create any point-in-time snapshot layer from your backup archive. Longer details with examples are all over, but a good overview point is: https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.htm... Also, it's basically the only sane way to run containers in production since all the copy-on-write and snapshot sending isn't an 8 layer deep hack cake. In Solaris, containers are aware of filesystems at the deepest levels and the networking is aware of containers too. All Linux-based container fads over the past 5 years still haven't caught up to Solaris+ZFS+Zones+Crossbow from 6 to 10 years ago.
- barkingcat 11y ago
  This type of ZFS replication / synchronization and rsync as well isn't at all similar to S3. As seiji posted, this is a fast diff'ing mechanism - identifying and trasmitting only what changed from one version of a file to the next, excelling at when files are system images or very large multi-gig or multi-terabyte files. With S3, I don't believe it has any ability to diff and/or store and transmit diffs - S3 is just a data store. ZFS replication / rsync is more for the use case where you want to make sure files on multiple sides of the data transfer equation are the same, and can pass diffs back and forth between them. Basically, if you need and only use S3 and file storage, this isn't want you want at all. The closest analogue is a binary git merge for very very very large files.
- takeda 11y ago
  Well if you use aws s3 sync, then know that this in fact relies only on timestamp and file size to compare files. If there's a difference it'll send the whole thing. rsync is smarter it actually can compare and send parts of modified files. This article talks about new functionality which utilizes zfs snapshots. Now zfs knows better what has changed (for example whether file was renamed or moved) so it can be even faster when replicating the changes.
- jlgaddis 11y ago
  I work for an ISP and wear a bunch of different hats but one of them is my "mail server admin hat". I hated our (when I started) Windows-based piece of crap "solution" we were using and have been migrating all of our users and domains to a "proper" Postfix+Dovecot on FreeBSD mail system. All hosted e-mail lives under /var/vmail on its own ZFS dataset ("filesystem"). I snapshot this dataset (via cron) several times per hour and ship those snapshots off to another machine in a different datacenter for safekeeping. If anything tragic were to happen to the mail server itself, I've got a pristine copy of all my users' mailboxes I can turn to that will be, at most, roughly 15 minutes out of date.
swiley 11y ago
Just a note to anyone new to rsync and friends: Rsync is a really fast version of cp that also works over networks. Rsync does not go both ways /and/ handle file deletion (you have to choose one or the other), so it can't replace dropbox and syncthing for example.
- LeoPanthera 11y ago
  If you do want something that goes both ways, unison is essentially a bidirectional rsync.
  - moviuro 11y ago
    An syncthing (http://syncthing.net http://syncthing.net) is a working unison :) (See http://popho.be/#gift http://popho.be/#gift for some more details, search unison)
    - rsync 11y ago
      Also note - rsync.net has supported Unison since 2007, although these days we would recommend using attic or borg, which we now support.[1] [1] As in, full, native support - not just using an sshfs mount...
      - LeoPanthera 11y ago
        Interesting. Both both and attic are new to me. What are the differences between them? They look very similar. Edit: Never mind, I found a handy list of differences here: https://borgbackup.readthedocs.org/en/stable/#differences-between-attic-and-borg https://borgbackup.readthedocs.org/en/stable/#differences-be...
      - darklajid 11y ago
        Not sure if you'll get back to this thread, but .. is something like syncthing impossible to imagine for rsync.net? I guess that's a 'daemon' vs. 'executed binary on the remote side', but I think that'd be awesome.
gjem97 11y ago
What other cloud providers do people use that support ZFS? Is ZFS on Linux a viable option these days?
- benjohnson 11y ago
  Hetzner.de works fine for my needs - they have a ~220 Euro storage server with 10 x 6TB drives that you can install FreeBSD 10.2 on. With ZFS you can have about 35TG usable (ZFS filesystems degrade in speed at 80% in my experience) My cost is $.007 USD per GB.
  - toomuchtodo 11y ago
    You may want to look into Backblaze's new storage offering B2. It's $0.005 USD/GB. https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html
antman 11y ago
OK so how do we setup a fresh ubuntu all with zfs?
- moviuro 11y ago
  You use FreeBSD + an hypervisor on top? Or you try ZOL (ZFS on Linux): good luck!
  - brunoqc 11y ago
    https://wiki.ubuntu.com/Kernel/Reference/ZFS https://wiki.ubuntu.com/Kernel/Reference/ZFS I'm not sure if it's usable now and if it will be good with 16.04.
    - rincebrain 11y ago
      I've been extremely astonished with how stable ZFS on Linux has been for me - I remember when it was unusable.
- barkingcat 11y ago
  ZFS on linux is not an undertaking that can be easily explained in a hn commment. People have posted links to alternatives below.
- aeorgnoieang 11y ago
  I don't remember the exact sequence of commands I entered to setup ZFS on several of my Ubuntu (14.04) boxes, but it certainly wasn't hard.
- jlgaddis 11y ago
  Someone will correct me if I'm wrong because I haven't used ZFS-on-Linux (my remaining Linux boxes are slowly being converted to all FreeBSD) but I don't think you can do "/ on ZFS" on Ubuntu since ZFS support is via a loadable kernel module (i.e., it's not in the mainline kernel -- and apparently never will be). You can install on, e.g., ext4, and then create /home as ZFS and then take advantage of all that ZFS offers (such as sending full/incremental snapshots like this article talks about). Or you could put your databases on ZFS and use snapshots to back them up, and so on.
ClashTheBunny 11y ago
Combine this with https://github.com/jollyjinx/ZFS-TimeMachine https://github.com/jollyjinx/ZFS-TimeMachine and you are in a much better place than Apple's terrible network Time Machine. It's cross-platform (except Windows) and it's much more reliable than SMB or AppleTalk.
- rsync 11y ago
  Just a note ... ALL rsync.net accounts run on a ZFS infrastructure. The 1TB+ accounts that you can send/recv to do not have snapshots enabled, since it's your zpool and you can set up whatever snapshots you want. But, our normal accounts[1] that you cannot ZFS send/recv to do have ZFS snapshots set up that we create and maintain for you, and they are immutable[2]. So this means you can just do a plain old 1:1 rsync mirror to your rsync.net account, forget all about incrementals or versions, and we will do whatever snapshot schedule you like.[3] So it's just like time machine, but more efficient (block level vs. hard links) and you don't have to set it up or worry about it and an attacker that gets your rsync.net credentials cannot wipe out your historical backups, since they are immutable here. [1] As always, HN-readers discount - just email info@rsync.net [2] even for our local root ... [3] default, for free, is 7 daily, or 7 daily + 4 weekly for TB+ customers, but you can set up any schedule you want (21 days, 8 weeks, 6 months, etc.)
  - rincebrain 11y ago
    How do you enforce immutable snapshots for local root?
  - tekacs 11y ago
    Any comment on why normal accounts can't ZFS send/recv? Is it because there isn't one filesystem per customer? Is that something that could be done, feasibly? Would be curious to hear. :)
brunoqc 11y ago
Is the data encrypted?
- LeoPanthera 11y ago
  It runs over SSH, so yes.
  - brunoqc 11y ago
    What about on the server side? For the storage.
    - M4v3R 11y ago
      If you want server-side encryption, then Tarsnap is probably better for you.
      - brunoqc 11y ago
        Yes I use Tarsnap but an encrypted backup using file system snapshots would be the best thing ever.
        barkingcat 11y ago
        One way that you can do this is just to zfs replicate a volume to another system that has an encrypted zfs mount. Probably not what you are looking for, but voila - end to end snapshotted encryption - except it's not a "backup", more like a hot failover volume.
        rlpb 11y ago
        So just take a ZFS snapshot and give Tarsnap the snapshot to back it up?
- barkingcat 11y ago
  I don't think this is what you need. This is a base file diff and transfer system - you have to add encryption on top of it yourself. Or you use a different tool like tarsnap that can be built on top of this.
LeoPanthera 11y ago
I'm an rsync.net customer, and I use ZFS locally, but I'm not using ZFS replication because I can't control in what order the files in the filesystem get backed up. (And I'm also unsure whether it's even possible interrupt a transfer and resume it later, if the underlying data has changed. rsync does this no problem.) My filesystems are pretty massive, but my internet connection is pretty crappy. I currently am running a script I hacked together which rsyncs my files up to rsync.net in (roughly) file size order - smallest first, largest last. This way, changes to huge files don't spend hours/days backing up, and preventing the backup of smaller files. I haven't yet found a better way to do this.
- gaadd33 11y ago
  Doesn't ZFS replication send a snapshot? So the underlying data changing doesn't have any impact on the snapshot's view of the world.
  - LeoPanthera 11y ago
    Yes, but after the data has changed, I don't want to waste time sending data in the snapshot that is no longer in the live data. I'd rather just carry on from where it is now. My current system does this just fine.
    - masklinn 11y ago
      So you add a large amount of data, you snapshot, you delete that data and at that point you send the snapshot but don't want to send the deleted data? Why not just re-snapshot and send the new snapshot instead of the old one? > And I'm also unsure whether it's even possible interrupt a transfer and resume it later, if the underlying data has changed There is no "underlying data has changed" in a snapshot, it's a revision of the filesystem. As for interrupting and resuming later, Delphix apparently has contributed a resumable zfs send/receive to OpenZFS, though I don't know if it's available in yours or rsync.net's distribution: http://blog.delphix.com/matt/2015/03/25/resumable-zfs-sendreceive/ http://blog.delphix.com/matt/2015/03/25/resumable-zfs-sendre... As for the concerns of sending small files first, I don't think ZFS could even use that: I don't believe it's content-addressed, so if you send a set of blocks from snapshot 1, interrupt and send snapshot 2 instead the blocks common to both will still be re-sent, ZFS only knows the block commonality from snapshots and since snapshot 1 hasn't been pushed its contents can't be used for snapshot 2. Unless the resumable replication support uses content addressing but that doesn't seem to be the case.
rsync 11y ago
Obligatory: rsync.net has always had a "HN readers discount" which is quite substantial. Just email info@rsync.net and ask about it.
- deleted 11y ago
  [deleted]
KirinDave 11y ago
This is awesome, but it should also remind us of something that is easy to forget. This is tangential to the thread, but relevant becuase of ZFS. Note how they run ZFS, "FreeBSD." We tend to ignore it shipping cloud services and mobile apps and websites, and just focus on Linux based deploys. But FreeBSD is out there. It has all the amenities you'd expect from a modern Linux and more including: 1. Compatibility with nearly all Linux binaries. 2. A more robust implementation of containerization (Jails), with a matching implementation of a docker API. 3. Proven OpenZFS support. 4. A really robust framework for moving and standing up software. 5. Proven performance. In a way, tbh, it's like one of many tech stacks we've neglected for the "mainstream" even though it has a proven track record and great features. ZFS is also in that family, as is stuff like Mono and OCaml.
- PhantomGremlin 11y ago
  We tend to ignore it One thing that FreeBSD doesn't have is a well known public face. Every software project (and company) needs one. Linux is Torvalds. Of course there's a lot more to a distribution than the kernel, but most people don't think that far. OpenBSD is Theo. Love him or hate him. But FreeBSD? A while ago Jordan Hubbard came to mind. But he was never "the" guy, and he left a long time ago. I view it as being run now by a nameless, faceless cabal. Anyway, that's just my opinion on one reason why we tend to ignore it.
jlgaddis 11y ago
ZFS is awesome. I never would have imagined I could ever get excited about a filesystem, but I did. If you've never used it, I encourage everyone to check it out. It's so completely "flexible" and you can do things that you never would've thought of. A while back, my personal mail server ran on FreeBSD 9.x on bare metal. I wanted to rebuild the physical host but run everything in jails on a "clean" 10.x install instead. I created a new virtual machine on our ESXi cluster, booted it with an mfsbsd image, and prep'd the hard disks. Back on the server, I shut down all the services, took snapshots, and shipped them off to the VM using ZFS send/receive. In short order, I had P2V'd my mail server and it worked wonderfully. I left everything running in that VM for a few days while I rebuilt the physical server and got all of my jails set up and then moved it all back. Allan Jude gave a cool talk at vBSDcon a few months ago entitled "Interesting Things You Didn't Know You Could Do With ZFS" [0]. If you've never used ZFS or you're wondering what it might have over your current filesystem of choice, you might check it out. [0]: https://www.youtube.com/watch?v=qXOZmDoy2Co https://www.youtube.com/watch?v=qXOZmDoy2Co
nailer 11y ago
Is 'rsync.net' actually Tridge [1], or just someone else trading on the name of an Open Source project? There's no 'about us' on the site to allow us to determine whether this is legit or not. And the stock photos look really sketchy. [1] https://en.wikipedia.org/wiki/Andrew_Tridgell https://en.wikipedia.org/wiki/Andrew_Tridgell
- rmwaite 11y ago
  I remember looking into this site before and it appears they have rebranded. I know they used to be related to JohnCompanies [http://johncompanies.com http://johncompanies.com] but I see no references to them anymore on rsync.net It's possible they are no longer related. edit: I just looked at http://johncompanies.com/about.html http://johncompanies.com/about.html and they do still mention rsync.net on that page.
- PhantomGremlin 11y ago
  There's no 'about us' on the site I tweaked them about this a few weeks ago, and got a detailed answer here on HN. But I guess they haven't gotten around to updating their site yet. https://news.ycombinator.com/item?id=10674029 https://news.ycombinator.com/item?id=10674029
  - rsync 11y ago
    Sorry - getting to that ...
- rsync 11y ago
  rsync.net has nothing to do with the actual rsync project or its authors. It may interest you to know that in 2006, when we were about to spin off rsync.net from JohnCompanies[1][2] we emailed all of the rsync authors and asked them if they had any objections at all to our naming the new company, and registering the domain name 'rsync.net' ... and they gave us their blessing. As for the stock photos ... the old site had no photos at all. Baby steps ... [1] The first VPS provider (circa 2001) [2] Yes, really.

125 points by wstrange 11y ago

I loved reading this article because it's just pure nerd joy. The guy is so enthusiastic about rsync, measurement, and unix, just reading the article is fun, even if I don't have a use for this particular tool.

matt2000 11y ago

Really fun to read article, but could someone give me a couple examples of what kind of system you'd use this with? I guess I'm so used to S3 now that I've forgotten what it's like to manage your own files.

seiji 11y ago
ZFS supports (and excels at) two necessary things for a good file system: copy on write snapshots and filesystem serialization. ZFS is designed to run one filesystem per user, so ZFS is designed to scale crazy high with little management overhead. It also makes it easy for snapshots to equal isolated per-user or even per-appication backups. You can do neat things like: create a base filesystem, create a snapshot of it (i.e. "freeze" the filesystem in time), send the snapshot to another system (or over the network for S3 storage, etc), continue modifying files, create another snapshot on top of your first one, now you can send/backup the new snapshot which is only a diff of the original snapshot. Many other systems besides ZFS support snapshots/copy-on-write layers, but they are mostly hacks and your performance will drop considerably during the "snapshot existing" time. ZFS is different because the snapshot mechanism doesn't kill your performance (unlike the 18 layer Linux file system organization). "Sending" a snapshot can either be directly over the network to another system to ingest the snapshot (enabling you to stream a new filesystem/snapshot into existence on a remote machine), or you can gzip the snapshot stream and send it off to backup storage. You can layer your snapshot diffs as much as you like. As long as you have all the diffs back to the original file system, you can re-create any point-in-time snapshot layer from your backup archive. Longer details with examples are all over, but a good overview point is: https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.htm... Also, it's basically the only sane way to run containers in production since all the copy-on-write and snapshot sending isn't an 8 layer deep hack cake. In Solaris, containers are aware of filesystems at the deepest levels and the networking is aware of containers too. All Linux-based container fads over the past 5 years still haven't caught up to Solaris+ZFS+Zones+Crossbow from 6 to 10 years ago.
barkingcat 11y ago
This type of ZFS replication / synchronization and rsync as well isn't at all similar to S3. As seiji posted, this is a fast diff'ing mechanism - identifying and trasmitting only what changed from one version of a file to the next, excelling at when files are system images or very large multi-gig or multi-terabyte files. With S3, I don't believe it has any ability to diff and/or store and transmit diffs - S3 is just a data store. ZFS replication / rsync is more for the use case where you want to make sure files on multiple sides of the data transfer equation are the same, and can pass diffs back and forth between them. Basically, if you need and only use S3 and file storage, this isn't want you want at all. The closest analogue is a binary git merge for very very very large files.
takeda 11y ago
Well if you use aws s3 sync, then know that this in fact relies only on timestamp and file size to compare files. If there's a difference it'll send the whole thing. rsync is smarter it actually can compare and send parts of modified files. This article talks about new functionality which utilizes zfs snapshots. Now zfs knows better what has changed (for example whether file was renamed or moved) so it can be even faster when replicating the changes.
jlgaddis 11y ago
I work for an ISP and wear a bunch of different hats but one of them is my "mail server admin hat". I hated our (when I started) Windows-based piece of crap "solution" we were using and have been migrating all of our users and domains to a "proper" Postfix+Dovecot on FreeBSD mail system. All hosted e-mail lives under /var/vmail on its own ZFS dataset ("filesystem"). I snapshot this dataset (via cron) several times per hour and ship those snapshots off to another machine in a different datacenter for safekeeping. If anything tragic were to happen to the mail server itself, I've got a pristine copy of all my users' mailboxes I can turn to that will be, at most, roughly 15 minutes out of date.

swiley 11y ago

Just a note to anyone new to rsync and friends: Rsync is a really fast version of cp that also works over networks. Rsync does not go both ways /and/ handle file deletion (you have to choose one or the other), so it can't replace dropbox and syncthing for example.

LeoPanthera 11y ago
If you do want something that goes both ways, unison is essentially a bidirectional rsync.
- moviuro 11y ago
  An syncthing (http://syncthing.net http://syncthing.net) is a working unison :) (See http://popho.be/#gift http://popho.be/#gift for some more details, search unison)
  - rsync 11y ago
    Also note - rsync.net has supported Unison since 2007, although these days we would recommend using attic or borg, which we now support.[1] [1] As in, full, native support - not just using an sshfs mount...
    - LeoPanthera 11y ago
      Interesting. Both both and attic are new to me. What are the differences between them? They look very similar. Edit: Never mind, I found a handy list of differences here: https://borgbackup.readthedocs.org/en/stable/#differences-between-attic-and-borg https://borgbackup.readthedocs.org/en/stable/#differences-be...
    - darklajid 11y ago
      Not sure if you'll get back to this thread, but .. is something like syncthing impossible to imagine for rsync.net? I guess that's a 'daemon' vs. 'executed binary on the remote side', but I think that'd be awesome.

gjem97 11y ago

What other cloud providers do people use that support ZFS? Is ZFS on Linux a viable option these days?

benjohnson 11y ago
Hetzner.de works fine for my needs - they have a ~220 Euro storage server with 10 x 6TB drives that you can install FreeBSD 10.2 on. With ZFS you can have about 35TG usable (ZFS filesystems degrade in speed at 80% in my experience) My cost is $.007 USD per GB.
- toomuchtodo 11y ago
  You may want to look into Backblaze's new storage offering B2. It's $0.005 USD/GB. https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html

antman 11y ago

OK so how do we setup a fresh ubuntu all with zfs?

moviuro 11y ago
You use FreeBSD + an hypervisor on top? Or you try ZOL (ZFS on Linux): good luck!
- brunoqc 11y ago
  https://wiki.ubuntu.com/Kernel/Reference/ZFS https://wiki.ubuntu.com/Kernel/Reference/ZFS I'm not sure if it's usable now and if it will be good with 16.04.
  - rincebrain 11y ago
    I've been extremely astonished with how stable ZFS on Linux has been for me - I remember when it was unusable.
barkingcat 11y ago
ZFS on linux is not an undertaking that can be easily explained in a hn commment. People have posted links to alternatives below.
aeorgnoieang 11y ago
I don't remember the exact sequence of commands I entered to setup ZFS on several of my Ubuntu (14.04) boxes, but it certainly wasn't hard.
jlgaddis 11y ago
Someone will correct me if I'm wrong because I haven't used ZFS-on-Linux (my remaining Linux boxes are slowly being converted to all FreeBSD) but I don't think you can do "/ on ZFS" on Ubuntu since ZFS support is via a loadable kernel module (i.e., it's not in the mainline kernel -- and apparently never will be). You can install on, e.g., ext4, and then create /home as ZFS and then take advantage of all that ZFS offers (such as sending full/incremental snapshots like this article talks about). Or you could put your databases on ZFS and use snapshots to back them up, and so on.

ClashTheBunny 11y ago

Combine this with https://github.com/jollyjinx/ZFS-TimeMachine https://github.com/jollyjinx/ZFS-TimeMachine and you are in a much better place than Apple's terrible network Time Machine. It's cross-platform (except Windows) and it's much more reliable than SMB or AppleTalk.

rsync 11y ago
Just a note ... ALL rsync.net accounts run on a ZFS infrastructure. The 1TB+ accounts that you can send/recv to do not have snapshots enabled, since it's your zpool and you can set up whatever snapshots you want. But, our normal accounts[1] that you cannot ZFS send/recv to do have ZFS snapshots set up that we create and maintain for you, and they are immutable[2]. So this means you can just do a plain old 1:1 rsync mirror to your rsync.net account, forget all about incrementals or versions, and we will do whatever snapshot schedule you like.[3] So it's just like time machine, but more efficient (block level vs. hard links) and you don't have to set it up or worry about it and an attacker that gets your rsync.net credentials cannot wipe out your historical backups, since they are immutable here. [1] As always, HN-readers discount - just email info@rsync.net [2] even for our local root ... [3] default, for free, is 7 daily, or 7 daily + 4 weekly for TB+ customers, but you can set up any schedule you want (21 days, 8 weeks, 6 months, etc.)
- rincebrain 11y ago
  How do you enforce immutable snapshots for local root?
- tekacs 11y ago
  Any comment on why normal accounts can't ZFS send/recv? Is it because there isn't one filesystem per customer? Is that something that could be done, feasibly? Would be curious to hear. :)

brunoqc 11y ago

Is the data encrypted?

LeoPanthera 11y ago
It runs over SSH, so yes.
- brunoqc 11y ago
  What about on the server side? For the storage.
  - M4v3R 11y ago
    If you want server-side encryption, then Tarsnap is probably better for you.
    - brunoqc 11y ago
      Yes I use Tarsnap but an encrypted backup using file system snapshots would be the best thing ever.
      - barkingcat 11y ago
        One way that you can do this is just to zfs replicate a volume to another system that has an encrypted zfs mount. Probably not what you are looking for, but voila - end to end snapshotted encryption - except it's not a "backup", more like a hot failover volume.
      - rlpb 11y ago
        So just take a ZFS snapshot and give Tarsnap the snapshot to back it up?
barkingcat 11y ago
I don't think this is what you need. This is a base file diff and transfer system - you have to add encryption on top of it yourself. Or you use a different tool like tarsnap that can be built on top of this.

LeoPanthera 11y ago

I'm an rsync.net customer, and I use ZFS locally, but I'm not using ZFS replication because I can't control in what order the files in the filesystem get backed up. (And I'm also unsure whether it's even possible interrupt a transfer and resume it later, if the underlying data has changed. rsync does this no problem.) My filesystems are pretty massive, but my internet connection is pretty crappy. I currently am running a script I hacked together which rsyncs my files up to rsync.net in (roughly) file size order - smallest first, largest last. This way, changes to huge files don't spend hours/days backing up, and preventing the backup of smaller files. I haven't yet found a better way to do this.

gaadd33 11y ago
Doesn't ZFS replication send a snapshot? So the underlying data changing doesn't have any impact on the snapshot's view of the world.
- LeoPanthera 11y ago
  Yes, but after the data has changed, I don't want to waste time sending data in the snapshot that is no longer in the live data. I'd rather just carry on from where it is now. My current system does this just fine.
  - masklinn 11y ago
    So you add a large amount of data, you snapshot, you delete that data and at that point you send the snapshot but don't want to send the deleted data? Why not just re-snapshot and send the new snapshot instead of the old one? > And I'm also unsure whether it's even possible interrupt a transfer and resume it later, if the underlying data has changed There is no "underlying data has changed" in a snapshot, it's a revision of the filesystem. As for interrupting and resuming later, Delphix apparently has contributed a resumable zfs send/receive to OpenZFS, though I don't know if it's available in yours or rsync.net's distribution: http://blog.delphix.com/matt/2015/03/25/resumable-zfs-sendreceive/ http://blog.delphix.com/matt/2015/03/25/resumable-zfs-sendre... As for the concerns of sending small files first, I don't think ZFS could even use that: I don't believe it's content-addressed, so if you send a set of blocks from snapshot 1, interrupt and send snapshot 2 instead the blocks common to both will still be re-sent, ZFS only knows the block commonality from snapshots and since snapshot 1 hasn't been pushed its contents can't be used for snapshot 2. Unless the resumable replication support uses content addressing but that doesn't seem to be the case.

rsync 11y ago

Obligatory: rsync.net has always had a "HN readers discount" which is quite substantial. Just email info@rsync.net and ask about it.

deleted 11y ago
[deleted]

KirinDave 11y ago

This is awesome, but it should also remind us of something that is easy to forget. This is tangential to the thread, but relevant becuase of ZFS. Note how they run ZFS, "FreeBSD." We tend to ignore it shipping cloud services and mobile apps and websites, and just focus on Linux based deploys. But FreeBSD is out there. It has all the amenities you'd expect from a modern Linux and more including: 1. Compatibility with nearly all Linux binaries. 2. A more robust implementation of containerization (Jails), with a matching implementation of a docker API. 3. Proven OpenZFS support. 4. A really robust framework for moving and standing up software. 5. Proven performance. In a way, tbh, it's like one of many tech stacks we've neglected for the "mainstream" even though it has a proven track record and great features. ZFS is also in that family, as is stuff like Mono and OCaml.

PhantomGremlin 11y ago
We tend to ignore it One thing that FreeBSD doesn't have is a well known public face. Every software project (and company) needs one. Linux is Torvalds. Of course there's a lot more to a distribution than the kernel, but most people don't think that far. OpenBSD is Theo. Love him or hate him. But FreeBSD? A while ago Jordan Hubbard came to mind. But he was never "the" guy, and he left a long time ago. I view it as being run now by a nameless, faceless cabal. Anyway, that's just my opinion on one reason why we tend to ignore it.

jlgaddis 11y ago

ZFS is awesome. I never would have imagined I could ever get excited about a filesystem, but I did. If you've never used it, I encourage everyone to check it out. It's so completely "flexible" and you can do things that you never would've thought of. A while back, my personal mail server ran on FreeBSD 9.x on bare metal. I wanted to rebuild the physical host but run everything in jails on a "clean" 10.x install instead. I created a new virtual machine on our ESXi cluster, booted it with an mfsbsd image, and prep'd the hard disks. Back on the server, I shut down all the services, took snapshots, and shipped them off to the VM using ZFS send/receive. In short order, I had P2V'd my mail server and it worked wonderfully. I left everything running in that VM for a few days while I rebuilt the physical server and got all of my jails set up and then moved it all back. Allan Jude gave a cool talk at vBSDcon a few months ago entitled "Interesting Things You Didn't Know You Could Do With ZFS" [0]. If you've never used ZFS or you're wondering what it might have over your current filesystem of choice, you might check it out. [0]: https://www.youtube.com/watch?v=qXOZmDoy2Co https://www.youtube.com/watch?v=qXOZmDoy2Co

nailer 11y ago

Is 'rsync.net' actually Tridge [1], or just someone else trading on the name of an Open Source project? There's no 'about us' on the site to allow us to determine whether this is legit or not. And the stock photos look really sketchy. [1] https://en.wikipedia.org/wiki/Andrew_Tridgell https://en.wikipedia.org/wiki/Andrew_Tridgell

rmwaite 11y ago
I remember looking into this site before and it appears they have rebranded. I know they used to be related to JohnCompanies [http://johncompanies.com http://johncompanies.com] but I see no references to them anymore on rsync.net It's possible they are no longer related. edit: I just looked at http://johncompanies.com/about.html http://johncompanies.com/about.html and they do still mention rsync.net on that page.
PhantomGremlin 11y ago
There's no 'about us' on the site I tweaked them about this a few weeks ago, and got a detailed answer here on HN. But I guess they haven't gotten around to updating their site yet. https://news.ycombinator.com/item?id=10674029 https://news.ycombinator.com/item?id=10674029
- rsync 11y ago
  Sorry - getting to that ...
rsync 11y ago
rsync.net has nothing to do with the actual rsync project or its authors. It may interest you to know that in 2006, when we were about to spin off rsync.net from JohnCompanies[1][2] we emailed all of the rsync authors and asked them if they had any objections at all to our naming the new company, and registering the domain name 'rsync.net' ... and they gave us their blessing. As for the stock photos ... the old site had no photos at all. Baby steps ... [1] The first VPS provider (circa 2001) [2] Yes, really.