6 ms·

Tarsnap pricing is for deduplicated data. You need to divide it by orders of magnitude for a proper comparison. Apples and oranges - what you are doing is on s

by gleb 15y ago

Tarsnap pricing is for deduplicated data. You need to divide it by orders of magnitude for a proper comparison. Apples and oranges - what you are doing is on site backup, Tarsnap is offsite. Both are needed.

jerf 15y ago
Dedupe isn't magic. If your data isn't duplicated on the block level, it isn't going to do anything for you. It does wonders on backing up 10,000 windows machines, it doesn't do anything at all for a server's data store.
- jacques_chester 15y ago
  I disagree. The deduping enables low-cost snapshot semantics which has drastically simplified my life. Every 24 hours, I dump a database and let tarsnap work out what to actually send to S3. And it does such a good job, for such a low price, that it boggles the mind.
  - jerf 15y ago
    Fair enough, but I'm reacting to the unbased claim that you can just divide it by "orders of magnitude" when the original post does not contain enough information to claim that, and indeed enough information that it's probably not true. If you have 100GB of server-type service data, you're not looking at "orders [plural] of magnitude" less space taken up on the backup service. The huge multipliers that dedupe is sometimes cited as giving are for certain datasets, you do not get "orders of magnitude" shrinking in the general case.
    - gleb 15y ago
      The claim is based on personal experience. With Mixpanel I'd expect even better results since their data is append-only by nature. I.e. think about deduplication when backing up a large append-only log file on a daily basis. Tarsnap has weaknesses but cost of storage for Mixpanel type of workload is not one of them.
    - JoachimSchipper 15y ago
      Note that mixpanel is talking about full dumps. Tarsnap does cross-backup deduplication, so while your first 100GB dump may take 80GB, the next may take 100MB.
dstorrs 15y ago
Good point, but I would have said the big win with Tarsnap is the zero-effort encryption. It means that everything you send is safe, so you don't have to worry about whether there is sensitive data in some particular DB / directory / logfile / whatever.
- tptacek 15y ago
  Tarsnap addresses the security concern of cloud backup, making it effectively as secure as on-site backup, but this article seems to stipulate that security isn't a big issue with offsite backup.
cperciva 15y ago
Tarsnap pricing is for deduplicated data. You need to divide it by orders of magnitude for a proper comparison. To be fair, that depends on how much duplicate data you're backing up. If you weren't using Tarsnap, you probably wouldn't have very many mostly-duplicate archives. I think the better way to look at it is that Tarsnap makes it easy to have many archived snapshots of your data without paying very much more. Whether being able to have daily snapshots for the past year is worth more than always having just your latest archive... well, that depends on what you're doing. It's a feature I think is important for my own use cases, but I'm not so bold as to think that everybody has exactly the same needs as me.

by gleb 15y ago

jerf 15y ago

Dedupe isn't magic. If your data isn't duplicated on the block level, it isn't going to do anything for you. It does wonders on backing up 10,000 windows machines, it doesn't do anything at all for a server's data store.

jacques_chester 15y ago
I disagree. The deduping enables low-cost snapshot semantics which has drastically simplified my life. Every 24 hours, I dump a database and let tarsnap work out what to actually send to S3. And it does such a good job, for such a low price, that it boggles the mind.
- jerf 15y ago
  Fair enough, but I'm reacting to the unbased claim that you can just divide it by "orders of magnitude" when the original post does not contain enough information to claim that, and indeed enough information that it's probably not true. If you have 100GB of server-type service data, you're not looking at "orders [plural] of magnitude" less space taken up on the backup service. The huge multipliers that dedupe is sometimes cited as giving are for certain datasets, you do not get "orders of magnitude" shrinking in the general case.
  - gleb 15y ago
    The claim is based on personal experience. With Mixpanel I'd expect even better results since their data is append-only by nature. I.e. think about deduplication when backing up a large append-only log file on a daily basis. Tarsnap has weaknesses but cost of storage for Mixpanel type of workload is not one of them.
  - JoachimSchipper 15y ago
    Note that mixpanel is talking about full dumps. Tarsnap does cross-backup deduplication, so while your first 100GB dump may take 80GB, the next may take 100MB.

dstorrs 15y ago

Good point, but I would have said the big win with Tarsnap is the zero-effort encryption. It means that everything you send is safe, so you don't have to worry about whether there is sensitive data in some particular DB / directory / logfile / whatever.

tptacek 15y ago
Tarsnap addresses the security concern of cloud backup, making it effectively as secure as on-site backup, but this article seems to stipulate that security isn't a big issue with offsite backup.

cperciva 15y ago

Tarsnap pricing is for deduplicated data. You need to divide it by orders of magnitude for a proper comparison. To be fair, that depends on how much duplicate data you're backing up. If you weren't using Tarsnap, you probably wouldn't have very many mostly-duplicate archives. I think the better way to look at it is that Tarsnap makes it easy to have many archived snapshots of your data without paying very much more. Whether being able to have daily snapshots for the past year is worth more than always having just your latest archive... well, that depends on what you're doing. It's a feature I think is important for my own use cases, but I'm not so bold as to think that everybody has exactly the same needs as me.