4 ms·

that's certainly interesting. as there will be migration from s3 to glacier, it would be nice if tarsnap had an option to store only the (say) last week in s3 (

by ibotty 14y ago

that's certainly interesting. as there will be migration from s3 to glacier, it would be nice if tarsnap had an option to store only the (say) last week in s3 (with .3$/gb/month) and the rest in glacier (with, say, .03$/gb/month). that would certainly be very nice. cperciva, what do you think?

cperciva 14y ago
I can't see any way for Tarsnap to use this right now. When you create a new archive, you're only uploading new blocks of data; the server has no way of knowing which old blocks of data are being re-used. As a result, storing any significant portion of a user's data in Amazon Glacier would mean that all archive extracts would need to go out to Glacier for data... Also, with Tarsnap's average block size (~ 64 kB uncompressed, typically ~ 32 kB compressed) the 50 microdollar cost per Glacier RETRIEVAL request means that I'd need to bump the pricing for tarsnap downloads up to about $1.75 / GB just to cover the AWS costs. I may find a use for Glacier at some point, but it's not something Tarsnap is going to be using in the near future.
- PanMan 14y ago
  While I have no idea how you would fit it in your current infrastructure, I certainly see a (BIG) use-case for: I have this 100 GB, store it somewhere safe (in glacier), I won't need it for the next year (unless my house burns down). I agree that is a bit different from ongoing daily backups with changes, but its also not THAT different from a customer perspective. That it doesn't fit with how you store blocks on the backend won't matter to a lot of customers.
  - cperciva 14y ago
    Oh, I absolutely agree that Glacier has lots of great use cases. I wish Tarsnap was able to make good use of it.
- ibotty 14y ago
  i understand. i hoped tarsnap knew what blocks (do not) get reused. it's unfortunate, because some backups happen to just lie around for very long. it would be nice to take advantage of (the low cost of) glacier for that. that said, if it's not possible with tarsnap now, it's not possible now. :D. if you find a satisfying possibility to incorporate it in the new backend(s) design (if that's fixable in the backend(s) alone), i'd surely be pleased.

by ibotty 14y ago

cperciva 14y ago

I can't see any way for Tarsnap to use this right now. When you create a new archive, you're only uploading new blocks of data; the server has no way of knowing which old blocks of data are being re-used. As a result, storing any significant portion of a user's data in Amazon Glacier would mean that all archive extracts would need to go out to Glacier for data... Also, with Tarsnap's average block size (~ 64 kB uncompressed, typically ~ 32 kB compressed) the 50 microdollar cost per Glacier RETRIEVAL request means that I'd need to bump the pricing for tarsnap downloads up to about $1.75 / GB just to cover the AWS costs. I may find a use for Glacier at some point, but it's not something Tarsnap is going to be using in the near future.

PanMan 14y ago
While I have no idea how you would fit it in your current infrastructure, I certainly see a (BIG) use-case for: I have this 100 GB, store it somewhere safe (in glacier), I won't need it for the next year (unless my house burns down). I agree that is a bit different from ongoing daily backups with changes, but its also not THAT different from a customer perspective. That it doesn't fit with how you store blocks on the backend won't matter to a lot of customers.
- cperciva 14y ago
  Oh, I absolutely agree that Glacier has lots of great use cases. I wish Tarsnap was able to make good use of it.
ibotty 14y ago
i understand. i hoped tarsnap knew what blocks (do not) get reused. it's unfortunate, because some backups happen to just lie around for very long. it would be nice to take advantage of (the low cost of) glacier for that. that said, if it's not possible with tarsnap now, it's not possible now. :D. if you find a satisfying possibility to incorporate it in the new backend(s) design (if that's fixable in the backend(s) alone), i'd surely be pleased.