6 ms·

While we're discussing making Tarsnap more user-friendly: The single reason (other than, er, fecklessness on my part) why I haven't tried Tarsnap is that it se

by gjm11 11y ago

While we're discussing making Tarsnap more user-friendly: The single reason (other than, er, fecklessness on my part) why I haven't tried Tarsnap is that it seems difficult to predict how much using Tarsnap would actually cost me. I would need to know (1) how much space my data will take up after compression and deduplication, and (2) how much bandwidth my incremental backups will need -- again, after compression and deduplication. Wouldn't It Be Nice If there were a "predict my Tarsnap costs" tool? You download it and point it at your data. It compresses it and identifies duplicated blocks, and says "You would initially be storing about 25 GB of data on the Tarsnap servers, which will cost you about $6.70 per month." It records a bunch of hashes. Then, a little later, you run it again. It identifies changed blocks and compresses the differences (or something), and says "If you do an incremental update like this, you will transfer about 2GB of data, which will cost you about $0.54." With, each time, a disclaimer: "This is only a crude estimate and if you think Tarsnap, Inc., will be in any way bound by it then you're out of your mind." I'm not sure whether this is a thing that any random third-party person could write, or whether getting the numbers right would require information about exactly what happens on the Tarsnap servers that only Colin has. Perhaps the answer is that doing it right requires secret information, but doing it well enough (e.g., getting within 20%, 95% of the time) is easy with some naive algorithm like "divide everything into 4kB blocks, hash them all, identify duplicates, compress individual unique blocks with gzip; do the same for incremental updates but ignore blocks whose hash hasn't changed". I'm curious: is this a factor stopping other people signing up with Tarsnap? Is it perhaps only a factor for cheapskate individuals like me, and not for the larger organizations that probably represent most of Tarsnap's profits?

neilk 11y ago
The thing that's keeping you from using Tarsnap is its costs? The deltas you're talking about look like they are under $10. It sounds like you are a programmer. At an hourly rate, it probably cost you more just to write that comment.
- recursive 11y ago
  No, predictability of costs.
- gjm11 11y ago
  The deltas were completely made up. If I knew they were right, I'd sign right up today. (I think.) What I don't want to do is guess at the deltas, sign up for Tarsnap, start up my first backup or my first incremental thereafter, and find I'm on the hook for hugely more than I originally budgeted. I think the way it actually works, what would happen is that after my first backup completes my account would go overdrawn and I would lose access to Tarsnap. And then I could walk away and all I'd actually lose would be whatever I'd prepaid. But -- and this is surely irrational, but I bet it isn't only my brain that works this way -- I would then feel really bad because (1) I would be failing to pay for something I had bought and (2) I would have spent money and got nothing in return. Even though (1) Colin surely budgets for a certain amount of attrition and (2) the sum of money involved needn't be bigger than $5. I don't want to do something that seems like it has a substantial chance of leaving me feeling simultaneously guilty and wasteful.
  - neilk 11y ago
    I did the math and I see your point. I'm also a CrashPlan user. Right now I'm paying CrashPlan about $60/year. My best guess at Tarsnap costs is about 10x that, $600/year. Normally I'm a very privacy-conscious person, but that difference does give me pause. I could probably slim down what I do with Crashplan to only personal files rather than the whole disk, but why would I want to do that? The whole point is to "fire and forget", and restore painlessly. Perhaps Tarsnap makes sense if you're Stripe, with lots of sensitive data to store, easily partitioned from your other data. But if my calculations are correct, probably not if you're an average individual.
    - gaadd33 11y ago
      Does CrashPlan dedup and compress the data? Also I have to say, tarsnap is so much more lightweight than CrashPlan. If they came out with a nice command line client I would have to reconsider them.
- zippergz 11y ago
  When I last tried to do a calculation, I came out at well over $100 per month which, yes, I can afford, but is way more expensive than what I'm paying Crashplan. There are many reasons I'd rather use Tarsnap than Crashplan, but it's hard to justify that price difference.
deleted 11y ago
[deleted]
avian 11y ago
> (1) how much space my data will take up after compression and deduplication In my experience, deduplication typically doesn't do much for the first backup. Further backups take approximately as much space as usual incremental backups. So the answer to this is "as much space as your existing backup". > (2) how much bandwidth my incremental backups will need I recently looked into this and for me the cost of bandwidth used for daily backups is about 10-15% of the cost of data storage.
fweespeech 11y ago
Yeah, I'd much prefer a $10/month for 10 GB of storage plan without any variable pricing. But honestly, it is what truly stops me from using Tarsnap. The part that stops me is the fact I have a dev server at home I have on 24/7 and I can just wait 9-10 hours for it to pull gpg'd backups.
cbhl 11y ago
Are you sure something like Backblaze isn't a better fit for you, then? "Unlimited" backups at $5/month/computer. Install their proprietary agent on your Windows/Mac OS X machine, log in, and forget about it. No Linux support, though.
- gjm11 11y ago
  Absolutely sure. The machine I am most interested in backing up is a FreeBSD box.
cperciva 11y ago
how much space my data will take up after compression and deduplication Valid point. You can use tarsnap's --dry-run option (along with --print-stats) to find out how much space your data will take up; unfortunately you need to have a key file before you can run that, and you can't create a key file until you've created a tarsnap account and added money to it. I will be adding a mechanism to allow people to run "keyless" dry runs so that you can install tarsnap and find out how much space your data would take before you spend any money.
- gjm11 11y ago
  Excellent! Would I be right in thinking that, if the storage cost is small enough not to bother me, then the bandwidth costs are extremely unlikely to be a problem?
  - cperciva 11y ago
    The bandwidth used to upload data is going to be around 1% more than the size of the data (protocol overhead). So to a close approximation the cost of the bandwidth for uploading is going to be the same as the cost of a month of storage.
    - gjm11 11y ago
      That's for the initial upload, right? (And then incremental updates should typically be much smaller.)
      - cperciva 11y ago
        Yes. (Although even with incrementals, until you start deleting data your total bandwidth usage to date will be roughly the same as your current monthly storage.)
adamtj 11y ago
This might be one of those situations where the simplest and cheapest way to figure out if something will work for you is to just try it for a bit. You can always stop, but right now you're at risk of data loss. Perfect is the enemy of good, worse is better, etc. Consider: Your current approach is basically to model the problem, evaluate the model, and then decide whether to use the service. That approach seems to protect you from risk, but it really doesn't. As you've noted, there isn't an existing model, which makes the modeling-first approach too expensive. As a result, you don't have backups, which is riskier that one or two months of abnormally expensive tarsnap service. Your deciding factor is probably the long-term cost. You can ballpark that by paying only a month or two and extrapolating. That should relatively cheap in absolute terms. Most numbers are not outliers, so you're unlikely to be very surprised. You would probably know already if you're atypical in some way. Even if it turns out you'd have an abnormally high cost, figuring that out by evaluating the service will be cheaper that building a model. Perhaps you're worried that the costs are fine right now but will grow too high over time. If so, that'll happen relatively slowly, over the course of a month or two at the fastest. At worst, you'll have to delete some backups or switch to a difference service. That might be annoying, but it won't be overly expensive and it's unlikely to even happen. If it does, at least you will have had affordable backups in the meantime. Right now, you're at risk.
pw 11y ago
It will cost you approximately nothing.
gjm11 11y ago
Just by way of information for Colin (though I'm not sure how much information it really gives): this comment is currently on +25, which to my mind suggests that there are a reasonable number of others who would also like this feature. I think the ability to do a dry-run upload without prepaying (which Colin has already said is on the way) will be pretty much a complete solution.

by gjm11 11y ago

neilk 11y ago

The thing that's keeping you from using Tarsnap is its costs? The deltas you're talking about look like they are under $10. It sounds like you are a programmer. At an hourly rate, it probably cost you more just to write that comment.

recursive 11y ago
No, predictability of costs.
gjm11 11y ago
The deltas were completely made up. If I knew they were right, I'd sign right up today. (I think.) What I don't want to do is guess at the deltas, sign up for Tarsnap, start up my first backup or my first incremental thereafter, and find I'm on the hook for hugely more than I originally budgeted. I think the way it actually works, what would happen is that after my first backup completes my account would go overdrawn and I would lose access to Tarsnap. And then I could walk away and all I'd actually lose would be whatever I'd prepaid. But -- and this is surely irrational, but I bet it isn't only my brain that works this way -- I would then feel really bad because (1) I would be failing to pay for something I had bought and (2) I would have spent money and got nothing in return. Even though (1) Colin surely budgets for a certain amount of attrition and (2) the sum of money involved needn't be bigger than $5. I don't want to do something that seems like it has a substantial chance of leaving me feeling simultaneously guilty and wasteful.
- neilk 11y ago
  I did the math and I see your point. I'm also a CrashPlan user. Right now I'm paying CrashPlan about $60/year. My best guess at Tarsnap costs is about 10x that, $600/year. Normally I'm a very privacy-conscious person, but that difference does give me pause. I could probably slim down what I do with Crashplan to only personal files rather than the whole disk, but why would I want to do that? The whole point is to "fire and forget", and restore painlessly. Perhaps Tarsnap makes sense if you're Stripe, with lots of sensitive data to store, easily partitioned from your other data. But if my calculations are correct, probably not if you're an average individual.
  - gaadd33 11y ago
    Does CrashPlan dedup and compress the data? Also I have to say, tarsnap is so much more lightweight than CrashPlan. If they came out with a nice command line client I would have to reconsider them.
zippergz 11y ago
When I last tried to do a calculation, I came out at well over $100 per month which, yes, I can afford, but is way more expensive than what I'm paying Crashplan. There are many reasons I'd rather use Tarsnap than Crashplan, but it's hard to justify that price difference.

deleted 11y ago

[deleted]

avian 11y ago

> (1) how much space my data will take up after compression and deduplication In my experience, deduplication typically doesn't do much for the first backup. Further backups take approximately as much space as usual incremental backups. So the answer to this is "as much space as your existing backup". > (2) how much bandwidth my incremental backups will need I recently looked into this and for me the cost of bandwidth used for daily backups is about 10-15% of the cost of data storage.

fweespeech 11y ago

Yeah, I'd much prefer a $10/month for 10 GB of storage plan without any variable pricing. But honestly, it is what truly stops me from using Tarsnap. The part that stops me is the fact I have a dev server at home I have on 24/7 and I can just wait 9-10 hours for it to pull gpg'd backups.

cbhl 11y ago

Are you sure something like Backblaze isn't a better fit for you, then? "Unlimited" backups at $5/month/computer. Install their proprietary agent on your Windows/Mac OS X machine, log in, and forget about it. No Linux support, though.

gjm11 11y ago
Absolutely sure. The machine I am most interested in backing up is a FreeBSD box.

cperciva 11y ago

how much space my data will take up after compression and deduplication Valid point. You can use tarsnap's --dry-run option (along with --print-stats) to find out how much space your data will take up; unfortunately you need to have a key file before you can run that, and you can't create a key file until you've created a tarsnap account and added money to it. I will be adding a mechanism to allow people to run "keyless" dry runs so that you can install tarsnap and find out how much space your data would take before you spend any money.

gjm11 11y ago
Excellent! Would I be right in thinking that, if the storage cost is small enough not to bother me, then the bandwidth costs are extremely unlikely to be a problem?
- cperciva 11y ago
  The bandwidth used to upload data is going to be around 1% more than the size of the data (protocol overhead). So to a close approximation the cost of the bandwidth for uploading is going to be the same as the cost of a month of storage.
  - gjm11 11y ago
    That's for the initial upload, right? (And then incremental updates should typically be much smaller.)
    - cperciva 11y ago
      Yes. (Although even with incrementals, until you start deleting data your total bandwidth usage to date will be roughly the same as your current monthly storage.)

adamtj 11y ago

This might be one of those situations where the simplest and cheapest way to figure out if something will work for you is to just try it for a bit. You can always stop, but right now you're at risk of data loss. Perfect is the enemy of good, worse is better, etc. Consider: Your current approach is basically to model the problem, evaluate the model, and then decide whether to use the service. That approach seems to protect you from risk, but it really doesn't. As you've noted, there isn't an existing model, which makes the modeling-first approach too expensive. As a result, you don't have backups, which is riskier that one or two months of abnormally expensive tarsnap service. Your deciding factor is probably the long-term cost. You can ballpark that by paying only a month or two and extrapolating. That should relatively cheap in absolute terms. Most numbers are not outliers, so you're unlikely to be very surprised. You would probably know already if you're atypical in some way. Even if it turns out you'd have an abnormally high cost, figuring that out by evaluating the service will be cheaper that building a model. Perhaps you're worried that the costs are fine right now but will grow too high over time. If so, that'll happen relatively slowly, over the course of a month or two at the fastest. At worst, you'll have to delete some backups or switch to a difference service. That might be annoying, but it won't be overly expensive and it's unlikely to even happen. If it does, at least you will have had affordable backups in the meantime. Right now, you're at risk.

pw 11y ago

It will cost you approximately nothing.

gjm11 11y ago

Just by way of information for Colin (though I'm not sure how much information it really gives): this comment is currently on +25, which to my mind suggests that there are a reasonable number of others who would also like this feature. I think the ability to do a dry-run upload without prepaying (which Colin has already said is on the way) will be pretty much a complete solution.