7 ms·

A Tarsnap-Alike for Private Storage

71 points by rlpb 16y ago

gte910h 16y ago
"It just so happens that I’ve been working on de-duplication for Synctus" OP should clarify that Synctus has no issue with this project, prove he has rights to release, etc, and stick that in the license. This feels like a copyright morass (although probably isn't).
- rlpb 16y ago
  I'm sorry this isn't clear. I wrote Synctus, am owner/director of the company that owns it, and have released ddar under the GPL having authorised myself through my company to do this.
  - gte910h 16y ago
    Right right, I actually expected that to be the case, but you won't necessarily always be the owner of Synctus, etc. You should probably do the short amount of paperwork required: http://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html http://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html "If you think that the employer or school might have a claim, you can resolve the problem clearly by getting a copyright disclaimer signed by a suitably authorized officer of the company or school. (Your immediate boss or a professor is usually NOT authorized to sign such a disclaimer.)" Stick everyone's mind at ease. Neat project (which is why I care enough to get the ducks in a row).
    - rlpb 16y ago
      A disclaimer doesn't quite make sense for me. Let me explain: I fully own True Blue Logic Ltd and am sole director. Both Synctus and ddar are projects owned by this company. Copyright on both Synctus and ddar are held by this company. Synctus is not open source. This ddar release is open source (GNU GPL 3) as of today. So my company retains copyright but licenses ddar to the public under the terms of the GNU GPL 3. Thus there is no disclaimer to sign, since my company still claims copyright. Of course, now that ddar is released under the GPL, it will be forever (in the same manner as any other open source project). None of this affects Synctus, which is not open source. If anyone contributes code to ddar, those contributions would not be permitted to be used in Synctus without a further licence from the contributor. I've considered this carefully and decided that this won't be a problem for me.
      - gte910h 16y ago
        >So my company retains copyright but licenses ddar to the public under the terms of the GNU GPL 3. Thus there is no disclaimer to sign, since my company still claims copyright. A disclaimer isn't needed now that it's clear in the COPYING file that TBL Ltd (the copyright holder) is the one publishing it. Thanks for clearing the air.
  - carterschonwald 16y ago
    since you may have investigated this: how does this compare with eg: repackaging the tarsnap deduplication code sans the encryption work?
    - rlpb 16y ago
      Tarsnap is not open source, so I intentionally avoided looking at the code to avoid any doubt over provenance. I imagine it would be harder, since Tarsnap is backed by S3 and AIUI Colin's own transactional storage layer to make it work with S3.
ch0wn 16y ago
Oh wow, I started using tarsnap yesterday night!
- JacobAldridge 16y ago
  And they've already launched a major addition. Imagine how incredible the company will be by a week from next Thursday!
elehack 16y ago
Interesting. I've been using rsnapshot for a while for a similar purpose, although its de-duplication is limited to keeping unmodified files as hard links. It'd be nice to see a brief comparison of ddar, rsnapshot, and other things that might fit that role.
- pqs 16y ago
  rdiff-backup also.
  - rlpb 16y ago
    I attempted a comparison with rdiff-backup here: http://www.synctus.com/ddar/rdiff-backup.html http://www.synctus.com/ddar/rdiff-backup.html Some others are briefly mentioned under "Alternatives", including a comparison with Tarsnap. I only have direct experience of Tarsnap and duplicity. Additions/corrections welcome.
- wladimir 16y ago
  I use duplicity for incremental encrypted off-site backups. Would be interesting to see how it compares to the others. As I store the backups on a shared machine, the most important feature for me is that the backupped data never passes through the target machine in unencrypted form, at any stage.
- donio 16y ago
  ddar seems more similar to bup. A comparison between those two would be awesome. I see that ddar supports removing old archives which bup doesn't yet. bup on the other hand already has a fuse filesystem which is on the todo list for ddar.
jayroh 16y ago
Because I'm guessing there might be others thinking the same question, I'll go ahead and ask - How is this different from something like rsync?
- rlpb 16y ago
  rsync is for bringing one tree in sync with another. It can do this relatively efficiently if you already have a previous version of the tree at the destination, which is a fairly common case. rsync compares (more or less) each file with its parallel file on the other side and transmits just the differences. rsync won't save you any storage, just bandwidth. ddar is for storing things de-duplicated, possibly over the network. It will save both storage and bandwidth. However, you won't get a tree on the other side, just an opaque container than you can extract the data out of again. So: rsync to bring two trees in to sync, Tarsnap or ddar to store things (eg. for backups). You could use rsync to store backups, but only really with the help of other tools. Tools like duplicity, rdiff-backup or rsnapshot. Tarsnap was the first common system (to my knowledge) to do full de-duplication for storage. ddar works in the same way. By full de-duplication I mean for the system to have an overview of all data in order to optimise transfers and storage. For example, if I were to download a large image (say a Debian ISO) and then back up my machine using Tarsnap or ddar, then delete the ISO and perform another backup, then download the ISO again and perform a third backup, the third backup will optimise out transmitting the image because it is already stored. I don't think that rdiff-backup, rsnapshot or duplicity can do this, since they only consider one backup at a time. I believe this applies to renaming files around as well.
  - jonhohle 16y ago
    Thanks for the clarification. rsync is so fickle with it's deduplication that it doesn't handle file attribute changes either due to hard link limitations. rsync does a lot to maintain file attributes across heterogeneous systems, does ddar address this issue as well?
    - rlpb 16y ago
      ddar doesn't care. The intention is that you use ddar together with a tool that will do what you need, such as tar as in the examples. tar will deal with file attributes; ddar will deal with the de-duplication.
tropin 16y ago
As always we mention TarSnap, I can't but be amazed Mr Percival, having all the infrastructure in place just sort of a GUI, passes on the gold mine of the millions of Windows users that couldn't care less about the mere existence of a command line, but still would pay for a backup tool as good as TarSnap.
- alexg0 16y ago
  I wonder how much more business tarsnap can get, just by packaging a couple of debs, not even talking about a having a repository.
- getsat 16y ago
  Maybe he decided that the time required to support Windows users isn't worth the money? He is quite a busy guy from what I've seen.
  - icey 16y ago
    If he had a Windows version it should be able to make enough money to pay for a support person AND still be incredibly profitable. He just needs to have someone else do the selling in order to let him focus on the hard technical problems. That being said, the market for a Windows version should NOT be individuals, it should be marketed to the enterprise. Tarsnap solves a problem that many companies have and pay buckets of money to cure.
- thaumaturgy 16y ago
  Colin Percival is another classic example of someone being brilliant in their field, but a mediocre businessperson. This happens a lot; probably the most common examples are great chefs that open restaurants and bakeries only to find that actually running such things requires a set of skills that they don't really have, or are interested in acquiring. (edit: I say that even though I don't even possess the aptitude necessary to fully respect his skills; he's a genius, and that's not a word I ever use lightly. I just hate to see a great work of engineering, like Tarsnap, languish in relative anonymity because it's poorly handled as a business.)
res0nat0r 16y ago
This looks very cool. I'm trying to use the examples, I've created a ddar directory with its db file and sha objects, but when I try a 'ddar tf /path/to/ddar/directory' I only seem to get either a timestamp or the -N name I've passed. Is this right? How to I see my filenames I've archived?
- rlpb 16y ago
  As far as I can tell you've done it right so far. ddar is generic and just stores whatever you put in. You also need to use the tools you used to format the data in to read them back out. If you've stored files using tar, then you want `ddar xf /path/to/ddar/directory|tar t` to see the files. Or if you've also gzipped them, then `ddar xf /path/to/ddar/directory|tar zt`. Does this help?
  - res0nat0r 16y ago
    Ahh that is exactly it. I am just not using the correct syntax. Thanks! This looks to be very cool, looking forward to trying it :]
RexRollman 16y ago
Has there ever been a filesystem that does de-duplication on the fly? Or would something like that be too slow for real world use?
- getsat 16y ago
  ZFS does dedup.
  - wazoox 16y ago
    And so do LessFS, Open dedupe and some others.
    - RexRollman 16y ago
      Thanks for the info guys.
thaumaturgy 16y ago
We've been using BackupPC (http://backuppc.sourceforge.net/ http://backuppc.sourceforge.net/) to accomplish something similar for a long time, and have been amazed at how it's able to store huge amounts of backups on relatively small amounts of disk space. It's highly configurable and has a decent web interface. Sometimes, though, it can be challenging to get it working, and it has a few other issues (e.g., moving backups for systems from one pool to another) that can make it difficult in the long-run for some use-cases.
RexRollman 16y ago
Will this compile on Cygwin like Tarsnap does?
- rlpb 16y ago
  It should only need a couple of minor tweaks. I am looking into it (as well as OS X support) and will post on my blog when it's ready.
  - RexRollman 16y ago
    Thanks for the inforamtion. I have a combination of Linux or Windows machines at home and it would be handy to be able to use the same tool on both.
weaksauce 16y ago
Thanks for putting this up it looks really interesting! Is there a git repo somewhere that we can look at or are we stuck with the tarball?
- rlpb 16y ago
  Just the tarball for now, though it will probably make its way to GitHub soon.

71 points by rlpb 16y ago

"It just so happens that I’ve been working on de-duplication for Synctus" OP should clarify that Synctus has no issue with this project, prove he has rights to release, etc, and stick that in the license. This feels like a copyright morass (although probably isn't).

rlpb 16y ago
I'm sorry this isn't clear. I wrote Synctus, am owner/director of the company that owns it, and have released ddar under the GPL having authorised myself through my company to do this.
- gte910h 16y ago
  Right right, I actually expected that to be the case, but you won't necessarily always be the owner of Synctus, etc. You should probably do the short amount of paperwork required: http://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html http://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html "If you think that the employer or school might have a claim, you can resolve the problem clearly by getting a copyright disclaimer signed by a suitably authorized officer of the company or school. (Your immediate boss or a professor is usually NOT authorized to sign such a disclaimer.)" Stick everyone's mind at ease. Neat project (which is why I care enough to get the ducks in a row).
  - rlpb 16y ago
    A disclaimer doesn't quite make sense for me. Let me explain: I fully own True Blue Logic Ltd and am sole director. Both Synctus and ddar are projects owned by this company. Copyright on both Synctus and ddar are held by this company. Synctus is not open source. This ddar release is open source (GNU GPL 3) as of today. So my company retains copyright but licenses ddar to the public under the terms of the GNU GPL 3. Thus there is no disclaimer to sign, since my company still claims copyright. Of course, now that ddar is released under the GPL, it will be forever (in the same manner as any other open source project). None of this affects Synctus, which is not open source. If anyone contributes code to ddar, those contributions would not be permitted to be used in Synctus without a further licence from the contributor. I've considered this carefully and decided that this won't be a problem for me.
    - gte910h 16y ago
      >So my company retains copyright but licenses ddar to the public under the terms of the GNU GPL 3. Thus there is no disclaimer to sign, since my company still claims copyright. A disclaimer isn't needed now that it's clear in the COPYING file that TBL Ltd (the copyright holder) is the one publishing it. Thanks for clearing the air.
- carterschonwald 16y ago
  since you may have investigated this: how does this compare with eg: repackaging the tarsnap deduplication code sans the encryption work?
  - rlpb 16y ago
    Tarsnap is not open source, so I intentionally avoided looking at the code to avoid any doubt over provenance. I imagine it would be harder, since Tarsnap is backed by S3 and AIUI Colin's own transactional storage layer to make it work with S3.

ch0wn 16y ago

Oh wow, I started using tarsnap yesterday night!

JacobAldridge 16y ago
And they've already launched a major addition. Imagine how incredible the company will be by a week from next Thursday!

elehack 16y ago

Interesting. I've been using rsnapshot for a while for a similar purpose, although its de-duplication is limited to keeping unmodified files as hard links. It'd be nice to see a brief comparison of ddar, rsnapshot, and other things that might fit that role.

pqs 16y ago
rdiff-backup also.
- rlpb 16y ago
  I attempted a comparison with rdiff-backup here: http://www.synctus.com/ddar/rdiff-backup.html http://www.synctus.com/ddar/rdiff-backup.html Some others are briefly mentioned under "Alternatives", including a comparison with Tarsnap. I only have direct experience of Tarsnap and duplicity. Additions/corrections welcome.
wladimir 16y ago
I use duplicity for incremental encrypted off-site backups. Would be interesting to see how it compares to the others. As I store the backups on a shared machine, the most important feature for me is that the backupped data never passes through the target machine in unencrypted form, at any stage.
donio 16y ago
ddar seems more similar to bup. A comparison between those two would be awesome. I see that ddar supports removing old archives which bup doesn't yet. bup on the other hand already has a fuse filesystem which is on the todo list for ddar.

jayroh 16y ago

Because I'm guessing there might be others thinking the same question, I'll go ahead and ask - How is this different from something like rsync?

rlpb 16y ago
rsync is for bringing one tree in sync with another. It can do this relatively efficiently if you already have a previous version of the tree at the destination, which is a fairly common case. rsync compares (more or less) each file with its parallel file on the other side and transmits just the differences. rsync won't save you any storage, just bandwidth. ddar is for storing things de-duplicated, possibly over the network. It will save both storage and bandwidth. However, you won't get a tree on the other side, just an opaque container than you can extract the data out of again. So: rsync to bring two trees in to sync, Tarsnap or ddar to store things (eg. for backups). You could use rsync to store backups, but only really with the help of other tools. Tools like duplicity, rdiff-backup or rsnapshot. Tarsnap was the first common system (to my knowledge) to do full de-duplication for storage. ddar works in the same way. By full de-duplication I mean for the system to have an overview of all data in order to optimise transfers and storage. For example, if I were to download a large image (say a Debian ISO) and then back up my machine using Tarsnap or ddar, then delete the ISO and perform another backup, then download the ISO again and perform a third backup, the third backup will optimise out transmitting the image because it is already stored. I don't think that rdiff-backup, rsnapshot or duplicity can do this, since they only consider one backup at a time. I believe this applies to renaming files around as well.
- jonhohle 16y ago
  Thanks for the clarification. rsync is so fickle with it's deduplication that it doesn't handle file attribute changes either due to hard link limitations. rsync does a lot to maintain file attributes across heterogeneous systems, does ddar address this issue as well?
  - rlpb 16y ago
    ddar doesn't care. The intention is that you use ddar together with a tool that will do what you need, such as tar as in the examples. tar will deal with file attributes; ddar will deal with the de-duplication.

tropin 16y ago

As always we mention TarSnap, I can't but be amazed Mr Percival, having all the infrastructure in place just sort of a GUI, passes on the gold mine of the millions of Windows users that couldn't care less about the mere existence of a command line, but still would pay for a backup tool as good as TarSnap.

alexg0 16y ago
I wonder how much more business tarsnap can get, just by packaging a couple of debs, not even talking about a having a repository.
getsat 16y ago
Maybe he decided that the time required to support Windows users isn't worth the money? He is quite a busy guy from what I've seen.
- icey 16y ago
  If he had a Windows version it should be able to make enough money to pay for a support person AND still be incredibly profitable. He just needs to have someone else do the selling in order to let him focus on the hard technical problems. That being said, the market for a Windows version should NOT be individuals, it should be marketed to the enterprise. Tarsnap solves a problem that many companies have and pay buckets of money to cure.
thaumaturgy 16y ago
Colin Percival is another classic example of someone being brilliant in their field, but a mediocre businessperson. This happens a lot; probably the most common examples are great chefs that open restaurants and bakeries only to find that actually running such things requires a set of skills that they don't really have, or are interested in acquiring. (edit: I say that even though I don't even possess the aptitude necessary to fully respect his skills; he's a genius, and that's not a word I ever use lightly. I just hate to see a great work of engineering, like Tarsnap, languish in relative anonymity because it's poorly handled as a business.)

res0nat0r 16y ago

This looks very cool. I'm trying to use the examples, I've created a ddar directory with its db file and sha objects, but when I try a 'ddar tf /path/to/ddar/directory' I only seem to get either a timestamp or the -N name I've passed. Is this right? How to I see my filenames I've archived?

rlpb 16y ago
As far as I can tell you've done it right so far. ddar is generic and just stores whatever you put in. You also need to use the tools you used to format the data in to read them back out. If you've stored files using tar, then you want `ddar xf /path/to/ddar/directory|tar t` to see the files. Or if you've also gzipped them, then `ddar xf /path/to/ddar/directory|tar zt`. Does this help?
- res0nat0r 16y ago
  Ahh that is exactly it. I am just not using the correct syntax. Thanks! This looks to be very cool, looking forward to trying it :]

RexRollman 16y ago

Has there ever been a filesystem that does de-duplication on the fly? Or would something like that be too slow for real world use?

getsat 16y ago
ZFS does dedup.
- wazoox 16y ago
  And so do LessFS, Open dedupe and some others.
  - RexRollman 16y ago
    Thanks for the info guys.

thaumaturgy 16y ago

We've been using BackupPC (http://backuppc.sourceforge.net/ http://backuppc.sourceforge.net/) to accomplish something similar for a long time, and have been amazed at how it's able to store huge amounts of backups on relatively small amounts of disk space. It's highly configurable and has a decent web interface. Sometimes, though, it can be challenging to get it working, and it has a few other issues (e.g., moving backups for systems from one pool to another) that can make it difficult in the long-run for some use-cases.

RexRollman 16y ago

Will this compile on Cygwin like Tarsnap does?

rlpb 16y ago
It should only need a couple of minor tweaks. I am looking into it (as well as OS X support) and will post on my blog when it's ready.
- RexRollman 16y ago
  Thanks for the inforamtion. I have a combination of Linux or Windows machines at home and it would be handy to be able to use the same tool on both.

weaksauce 16y ago

Thanks for putting this up it looks really interesting! Is there a git repo somewhere that we can look at or are we stuck with the tarball?

rlpb 16y ago
Just the tarball for now, though it will probably make its way to GitHub soon.