5 ms·

Because I'm guessing there might be others thinking the same question, I'll go ahead and ask - How is this different from something like rsync?

by jayroh 16y ago

rlpb 16y ago
rsync is for bringing one tree in sync with another. It can do this relatively efficiently if you already have a previous version of the tree at the destination, which is a fairly common case. rsync compares (more or less) each file with its parallel file on the other side and transmits just the differences. rsync won't save you any storage, just bandwidth. ddar is for storing things de-duplicated, possibly over the network. It will save both storage and bandwidth. However, you won't get a tree on the other side, just an opaque container than you can extract the data out of again. So: rsync to bring two trees in to sync, Tarsnap or ddar to store things (eg. for backups). You could use rsync to store backups, but only really with the help of other tools. Tools like duplicity, rdiff-backup or rsnapshot. Tarsnap was the first common system (to my knowledge) to do full de-duplication for storage. ddar works in the same way. By full de-duplication I mean for the system to have an overview of all data in order to optimise transfers and storage. For example, if I were to download a large image (say a Debian ISO) and then back up my machine using Tarsnap or ddar, then delete the ISO and perform another backup, then download the ISO again and perform a third backup, the third backup will optimise out transmitting the image because it is already stored. I don't think that rdiff-backup, rsnapshot or duplicity can do this, since they only consider one backup at a time. I believe this applies to renaming files around as well.
- jonhohle 16y ago
  Thanks for the clarification. rsync is so fickle with it's deduplication that it doesn't handle file attribute changes either due to hard link limitations. rsync does a lot to maintain file attributes across heterogeneous systems, does ddar address this issue as well?
  - rlpb 16y ago
    ddar doesn't care. The intention is that you use ddar together with a tool that will do what you need, such as tar as in the examples. tar will deal with file attributes; ddar will deal with the de-duplication.
mdasen 16y ago
I think it's easiest to explain with an example. So, every day at 10pm, I want to make a backup of my server to another server. I ddar or rsync the files to another server. Monday at 10pm, I do it; Tuesday at 10pm, I do it; Wednesday at 10pm, yep, did it. Now it's Thursday and for whatever reason, I have to restore the system to the state that it was on Monday. With rsync, changes have been overwritten. ddar and Tarsnap are useful because they allow you to have a full backup of each of those days, rather than just the latest. Likewise, they package it better than the old "do a full backup and then do incrementals on top of it" way so that you don't have to restore the original and then replay all the incrementals. Now, you could just store full backups each night, but that would take an incredible amount of space. ddar and Tarsnap allow you to have the convenience of full-backup while having the data de-duplicated so that it doesn't take up too much disk space. Rsync is more about saving bandwidth while transferring large amounts of files.
- jonhohle 16y ago
  Are you aware of the --link-dest option for rsync? It seems to do the same thing. I've been using this (via dirvish) for years for doing full nightly, deduped backups.
  - cperciva 16y ago
    Hardlinks are only useful if you have files which are unmodified. Tarsnap (and I assume ddar) will take advantage when part of a file is unmodified.
  - rlpb 16y ago
    rsync with --link-dest will store each version of a file separately, including parts that are the same. It also won't hard link files in different areas of the tree that are the same, storing them multiple times also. Nor will it relink files that disappear and reappear, or change and then change back. I wouldn't really call it deduped storage at all. The optimisation it does do might well be sufficient for your needs though.

Because I'm guessing there might be others thinking the same question, I'll go ahead and ask - How is this different from something like rsync?

by jayroh 16y ago

Because I'm guessing there might be others thinking the same question, I'll go ahead and ask - How is this different from something like rsync?

rlpb 16y ago

rsync is for bringing one tree in sync with another. It can do this relatively efficiently if you already have a previous version of the tree at the destination, which is a fairly common case. rsync compares (more or less) each file with its parallel file on the other side and transmits just the differences. rsync won't save you any storage, just bandwidth. ddar is for storing things de-duplicated, possibly over the network. It will save both storage and bandwidth. However, you won't get a tree on the other side, just an opaque container than you can extract the data out of again. So: rsync to bring two trees in to sync, Tarsnap or ddar to store things (eg. for backups). You could use rsync to store backups, but only really with the help of other tools. Tools like duplicity, rdiff-backup or rsnapshot. Tarsnap was the first common system (to my knowledge) to do full de-duplication for storage. ddar works in the same way. By full de-duplication I mean for the system to have an overview of all data in order to optimise transfers and storage. For example, if I were to download a large image (say a Debian ISO) and then back up my machine using Tarsnap or ddar, then delete the ISO and perform another backup, then download the ISO again and perform a third backup, the third backup will optimise out transmitting the image because it is already stored. I don't think that rdiff-backup, rsnapshot or duplicity can do this, since they only consider one backup at a time. I believe this applies to renaming files around as well.

jonhohle 16y ago
Thanks for the clarification. rsync is so fickle with it's deduplication that it doesn't handle file attribute changes either due to hard link limitations. rsync does a lot to maintain file attributes across heterogeneous systems, does ddar address this issue as well?
- rlpb 16y ago
  ddar doesn't care. The intention is that you use ddar together with a tool that will do what you need, such as tar as in the examples. tar will deal with file attributes; ddar will deal with the de-duplication.

mdasen 16y ago

I think it's easiest to explain with an example. So, every day at 10pm, I want to make a backup of my server to another server. I ddar or rsync the files to another server. Monday at 10pm, I do it; Tuesday at 10pm, I do it; Wednesday at 10pm, yep, did it. Now it's Thursday and for whatever reason, I have to restore the system to the state that it was on Monday. With rsync, changes have been overwritten. ddar and Tarsnap are useful because they allow you to have a full backup of each of those days, rather than just the latest. Likewise, they package it better than the old "do a full backup and then do incrementals on top of it" way so that you don't have to restore the original and then replay all the incrementals. Now, you could just store full backups each night, but that would take an incredible amount of space. ddar and Tarsnap allow you to have the convenience of full-backup while having the data de-duplicated so that it doesn't take up too much disk space. Rsync is more about saving bandwidth while transferring large amounts of files.

jonhohle 16y ago
Are you aware of the --link-dest option for rsync? It seems to do the same thing. I've been using this (via dirvish) for years for doing full nightly, deduped backups.
- cperciva 16y ago
  Hardlinks are only useful if you have files which are unmodified. Tarsnap (and I assume ddar) will take advantage when part of a file is unmodified.
- rlpb 16y ago
  rsync with --link-dest will store each version of a file separately, including parts that are the same. It also won't hard link files in different areas of the tree that are the same, storing them multiple times also. Nor will it relink files that disappear and reappear, or change and then change back. I wouldn't really call it deduped storage at all. The optimisation it does do might well be sufficient for your needs though.