4 ms·

Tarsnap works really, really well. Just make a consistent snapshot of your data (I'm using UFS snapshots), point Tarsnap at it, and you're good to go. The docu

by kogir 13y ago

Tarsnap works really, really well. Just make a consistent snapshot of your data (I'm using UFS snapshots), point Tarsnap at it, and you're good to go. The documentation is thorough, and Colin (the owner/operator/author) responds quickly to emails. Finally, compression and deduplication is amazing: [nick@home ~]$ sudo tarsnap --print-stats Total size Compressed size All archives 348 GB 76 GB (unique data) 34 GB 6.3 GB Yep, I've backed up 350GB of data, but since most of it is duplicated, I pay for storing 6.3GB. Win. One word of caution though - this isn't a mainstream consumer backup service. If you lose your keys you lose your data. No chance of recovery. So make sure you back those up properly too, ideally in a different geography.

cperciva 13y ago
Just make a consistent snapshot of your data (I'm using UFS snapshots), point Tarsnap at it, and you're good to go. You're using the --snaptime option, right? It's necessary when you're backing up a filesystem snapshot in order to work around a race condition with them -- if a file is modified, the filesystem snapshot is created, and then the file is modified again, all within a single time quantum, it can trick Tarsnap into thinking that the file hasn't been modified later (which triggers an optimization of "this must be the same blocks as it was last time" in place of the usual "read the file and split it into blocks" behaviour). Finally, compression and deduplication is amazing: Well, if we're going to be posting statistics here... Total size Compressed size All archives 269 TB 121 TB (unique data) 177 GB 72 GB That's 269 TB of data backed up from my laptop, deduplicated and compressed down to 72 GB. This is what I get for taking a backup of my entire home directory every hour...
- kogir 13y ago
  > You're using the --snaptime option, right? Yep, emailed you about this in fact, and appreciated your detailed response.
  - cperciva 13y ago
    Ah, that was you -- I remembered sending an email about snaptime recently but couldn't remember who it was to (and HN user names don't always correlate anyway...)
bluetooth 13y ago
Excuse my ignorance, but what does duplicated in this context mean? Multiple copies of the same file?
- cperciva 13y ago
  Multiple copies of a file, or parts of a file. These can be spatial (your archive contains an SVN checkout, so there's an extra copy of every file in the .svn/pristine directory) or temporal (you take regular backups, and many of the files haven't changed very much from one backup to the next).
  - bluetooth 13y ago
    Ah okay, thanks. Another question: Why wouldn't compression take care of that? Isn't the point of compression to compact as many repeating sequences as possible?
    - cperciva 13y ago
      Deduplication is a form of compression, yes. Most forms of compression are "local" however -- looking to match data against bits from within the past few MB -- so they won't detect duplicated data spread across entire archives.
    - deleted 13y ago
      [deleted]
    - keeperofdakeys 13y ago
      In order for compression to work, the data must be 'solid'. Which means to add or remove something from the archive, you must reprocess the entire archive. This isn't a very good model for backups, especially when you have lots of them. (As an aside, .zip files aren't 'solid', so two copies of a file won't compress well. This is also the reason why most archives on linux are done through tar, to create a single stream of data). Tarsnap uses variable blocks (in such a way that inserting into the middle of a file creates minimal differences). If a new block is detected during a backup, it only needs to send that block, as the rest are already stored on the server. It also means that new archives can refer to the old blocks stored, allowing each archive to be independent, and unused blocks removed when the last archive using it is deleted. Before sending, blocks are compressed then encrypted, so compression can't really help since you might not have all the old data locally. You can also deal with backups using a master and incremental diffs. This doesn't work well with the tarsnap model, as archives are no longer independent.

by kogir 13y ago

cperciva 13y ago

Just make a consistent snapshot of your data (I'm using UFS snapshots), point Tarsnap at it, and you're good to go. You're using the --snaptime option, right? It's necessary when you're backing up a filesystem snapshot in order to work around a race condition with them -- if a file is modified, the filesystem snapshot is created, and then the file is modified again, all within a single time quantum, it can trick Tarsnap into thinking that the file hasn't been modified later (which triggers an optimization of "this must be the same blocks as it was last time" in place of the usual "read the file and split it into blocks" behaviour). Finally, compression and deduplication is amazing: Well, if we're going to be posting statistics here... Total size Compressed size All archives 269 TB 121 TB (unique data) 177 GB 72 GB That's 269 TB of data backed up from my laptop, deduplicated and compressed down to 72 GB. This is what I get for taking a backup of my entire home directory every hour...

kogir 13y ago
> You're using the --snaptime option, right? Yep, emailed you about this in fact, and appreciated your detailed response.
- cperciva 13y ago
  Ah, that was you -- I remembered sending an email about snaptime recently but couldn't remember who it was to (and HN user names don't always correlate anyway...)

bluetooth 13y ago

Excuse my ignorance, but what does duplicated in this context mean? Multiple copies of the same file?

cperciva 13y ago
Multiple copies of a file, or parts of a file. These can be spatial (your archive contains an SVN checkout, so there's an extra copy of every file in the .svn/pristine directory) or temporal (you take regular backups, and many of the files haven't changed very much from one backup to the next).
- bluetooth 13y ago
  Ah okay, thanks. Another question: Why wouldn't compression take care of that? Isn't the point of compression to compact as many repeating sequences as possible?
  - cperciva 13y ago
    Deduplication is a form of compression, yes. Most forms of compression are "local" however -- looking to match data against bits from within the past few MB -- so they won't detect duplicated data spread across entire archives.
  - deleted 13y ago
    [deleted]
  - keeperofdakeys 13y ago
    In order for compression to work, the data must be 'solid'. Which means to add or remove something from the archive, you must reprocess the entire archive. This isn't a very good model for backups, especially when you have lots of them. (As an aside, .zip files aren't 'solid', so two copies of a file won't compress well. This is also the reason why most archives on linux are done through tar, to create a single stream of data). Tarsnap uses variable blocks (in such a way that inserting into the middle of a file creates minimal differences). If a new block is detected during a backup, it only needs to send that block, as the rest are already stored on the server. It also means that new archives can refer to the old blocks stored, allowing each archive to be independent, and unused blocks removed when the last archive using it is deleted. Before sending, blocks are compressed then encrypted, so compression can't really help since you might not have all the old data locally. You can also deal with backups using a master and incremental diffs. This doesn't work well with the tarsnap model, as archives are no longer independent.