5 ms·

Not sure why Tarsnap has bubbled up on HN again, but I'm always happy to answer questions about Tarsnap if there's anything anyone here wants to know.

soneca 10y ago
A "truly paranoid" that is not a software developer can use Tarsnap?
- cperciva 10y ago
  I'm not sure what you're asking, but if your question is "can paranoid non-developers use Tarsnap", absolutely. You'll need to have some basic comfort with UNIX, though.
  - soneca 10y ago
    Yes, the question was if it is possible, regarding necessary technical knowledge to use Tarsnap for a non-developer to use it. You answered it, thanks.
Osmium 10y ago
Always wanted to try Tarsnap for personal use, but I can only budget if the cost was comparable to Dropbox/Microsoft OneDrive/Google Drive/iCloud/Amazon Drive which are all ~$10/TB/mo, but instead Tarsnap seems about an order of magnitude more expensive? Or is the product just not aimed at me? I currently use Arq[1] for backups (which has its own encryption[2] with a user choice of cloud backend), but Tarsnap has such a stellar reputation I'd definitely try it if I could :) [1] https://www.arqbackup.com https://www.arqbackup.com [2] AES-256 with PKCS5_PBKDF2_HMAC_SHA1 for key derivation, implemented by OpenSSL, with an open file format https://www.arqbackup.com/arq_data_format.txt https://www.arqbackup.com/arq_data_format.txt
- CodeWriter23 10y ago
  Tarsnap stores in S3, which is the primary cost driver. For the extra expense, you get multi-region durability up to I think it's six nines.
- y4mi 10y ago
  i haven't used tarsnap either, but i see it as cold storage. dropbox, onedrive, etc are all hot storage and would get rendered useless by a cryptolocker cant say anything about arq, as ive never head of them.
- hobarrera 10y ago
  Are you really sure you need all that storage? Tarsnap has deduplication, so you might end up using a lot less. Or not. It depends on your use-case, but I'm just suggesting you consider that factor too.
diafygi 10y ago
Is using spiped viable for redis servers? Can all the servers that need to connect to the redis server all connect via spiped using the same secret key? Or do multiple connections mess things up? How many simultaneous incoming connections can one listener deal with?
- cperciva 10y ago
  I don't use redis, but I understand that lots of people use spiped with redis, yes. Each connection going through spiped is just another connection; spiped is intended to be a "drop in and change where you point your services at" way to make TCP connections secure. Re simultaneous connections: The current release is limited to 500, but the next release (a matter of weeks) will be able to handle an arbitrarily large number of connections. That said, it uses select/poll based non-blocking I/O, so performance may suffer if most of your connections are idle. (This hasn't been an issue for any spiped uses I'm aware of.)
nodesocket 10y ago
Colin, Any interest or thoughts about adding Google Cloud Storage as an option vs AWS S3? S3 (us-east-1) runs $0.023 per GB. Google Cloud Storage in a single region runs slightly less at $0.02 per GB. Howerver, bandwidth may be the biggest cost factor for you not storage.
- cperciva 10y ago
  The price difference between S3 and Google Cloud Storage isn't enough to make a meaningful difference, particularly since I'd need to set up server code in Google Cloud to manage it. Bandwidth isn't a huge cost; but the server which keeps track of where all the bits are in S3 and shuffles them around is surprisingly expensive.
  - nodesocket 10y ago
    Curious what AWS instance type you are using for that machine that keeps track of all the Tarsnap bits? I'm guessing it is highly available, so multiple instances of that type?
  - boulos 10y ago
    Disclosure: I work on Google Cloud (and love the idea of Tarsnap, including the picodollars). What about if you were to age things into Nearline and Coldline using lifecycle policies? Also, someone claims you do multiple regions, which our "default" multiregional does at less than half the cost of S3. Admittedly we only recently (finally!) added per-object storage classes. But unless I misunderstand how your blocks work in tarsnap, wouldn't that be amenable to having some large portion as Nearline and possibly even Coldline? If not, I'd love to understand! (contact info in my profile).
nickpsecurity 10y ago
How many copies do you keep of the data to avoid corruption in storage?
- cperciva 10y ago
  Tarsnap data is stored in S3, with cryptographic authentication codes to detect any corruption; while data corruption has been detected in transit on many occasions (and requests retried as a result), I am not aware of S3 ever corrupting any data in situ.
  - nickpsecurity 10y ago
    Thank you.
neurostimulant 10y ago
Downloading large amount of data from tarsnap archive is very slow. Any tips/tricks to speed them up?
- cperciva 10y ago
  Scott Wheeler wrote a tool for speeding up extracts by running multiple tarsnap processes: https://github.com/directededge/redsnapper https://github.com/directededge/redsnapper There's also the trivial (but often unhelpful) advice of "be closer to the server" -- the problem is one of round trip latencies since the internal design of tar (and thus tarsnap) works on the principle of "read a block; do something with it; repeat". This is what I feel to be the biggest technical pain point in tarsnap, and I know in theory how to fix it, but every time I've started focusing my energy on it other things have come up requiring more urgent attention. :-/
danohu 10y ago
How can I download backups into a different directory from where they originated? [I was just trying out tarsnap, go to the restore test section, and was a bit stumped. It seems like "-xC /other/dir" might do it, but the man page is pretty opaque. I'd like to test a restore without the risk of clobbering my data with some misconfigured backup!]
- crzwdjk 10y ago
  It works just like tar for the most part, so it extracts into whatever the current working directory happens to be. I'd just do (cd /other/dir; tarsnap -x -f whatever-archive).
  - danohu 10y ago
    Thanks, that makes sense
caivvoacmh 10y ago
Hi cperciva can you please explain the following: 1) "Every Tarsnap archive acts like it is completely independent of all other archives" Lets say I'm backing up `/somedir` that looks like this: # archive1 - initial backup ── somedir ├── file1.txt ├── file2.txt └── file3.txt # archive 2 - 2nd backup ── somedir └── file3.txt (modified) Are you suggesting that if I delete `archive1` and `rm -rf /somedir` that I can restore `/somedir` with `archive2` alone? If so, how is that possible? 2) Does tarsnap stream the backup data to the tarsnap service or does it create the encrypted archive and only after this upload it? For instance, if I've got 100 GB to backup (first time), do I need 100 GB of free space for tarsnap to work?
- cperciva 10y ago
  Are you suggesting that if I delete `archive1` and `rm -rf /somedir` that I can restore `/somedir` with `archive2` alone? If archive2 only contains somedir/file3.txt, then if you extract archive2 you'll get somedir/file3.txt but not file1.txt or file2.txt. But if you create archive2 containing all three files, tarsnap will recognize the duplicate data in file1.txt and file2.txt and not re-upload it; but when you extract archive2 you'll get all three blocks. Tarsnap reference-counts blocks of data so that when you delete archive1 it only deletes the data which is not used by any of the remaining archives. I often tell new Tarsnap users that they should start by forgetting everything they know about incremental backups. Tell Tarsnap what data you want to have in an archive, and let it do the work of figuring out what's new. Does tarsnap stream the backup data to the tarsnap service or does it create the encrypted archive and only after this upload it? Tarsnap uploads data as it collects it. Tarsnap needs a small amount of disk space to keep track of which blocks have been uploaded previously, but it's less than 1% of the size of the data archived.

soneca 10y ago

A "truly paranoid" that is not a software developer can use Tarsnap?

cperciva 10y ago
I'm not sure what you're asking, but if your question is "can paranoid non-developers use Tarsnap", absolutely. You'll need to have some basic comfort with UNIX, though.
- soneca 10y ago
  Yes, the question was if it is possible, regarding necessary technical knowledge to use Tarsnap for a non-developer to use it. You answered it, thanks.

Osmium 10y ago

Always wanted to try Tarsnap for personal use, but I can only budget if the cost was comparable to Dropbox/Microsoft OneDrive/Google Drive/iCloud/Amazon Drive which are all ~$10/TB/mo, but instead Tarsnap seems about an order of magnitude more expensive? Or is the product just not aimed at me? I currently use Arq[1] for backups (which has its own encryption[2] with a user choice of cloud backend), but Tarsnap has such a stellar reputation I'd definitely try it if I could :) [1] https://www.arqbackup.com https://www.arqbackup.com [2] AES-256 with PKCS5_PBKDF2_HMAC_SHA1 for key derivation, implemented by OpenSSL, with an open file format https://www.arqbackup.com/arq_data_format.txt https://www.arqbackup.com/arq_data_format.txt

CodeWriter23 10y ago
Tarsnap stores in S3, which is the primary cost driver. For the extra expense, you get multi-region durability up to I think it's six nines.
y4mi 10y ago
i haven't used tarsnap either, but i see it as cold storage. dropbox, onedrive, etc are all hot storage and would get rendered useless by a cryptolocker cant say anything about arq, as ive never head of them.
hobarrera 10y ago
Are you really sure you need all that storage? Tarsnap has deduplication, so you might end up using a lot less. Or not. It depends on your use-case, but I'm just suggesting you consider that factor too.

diafygi 10y ago

Is using spiped viable for redis servers? Can all the servers that need to connect to the redis server all connect via spiped using the same secret key? Or do multiple connections mess things up? How many simultaneous incoming connections can one listener deal with?

cperciva 10y ago
I don't use redis, but I understand that lots of people use spiped with redis, yes. Each connection going through spiped is just another connection; spiped is intended to be a "drop in and change where you point your services at" way to make TCP connections secure. Re simultaneous connections: The current release is limited to 500, but the next release (a matter of weeks) will be able to handle an arbitrarily large number of connections. That said, it uses select/poll based non-blocking I/O, so performance may suffer if most of your connections are idle. (This hasn't been an issue for any spiped uses I'm aware of.)

nodesocket 10y ago

Colin, Any interest or thoughts about adding Google Cloud Storage as an option vs AWS S3? S3 (us-east-1) runs $0.023 per GB. Google Cloud Storage in a single region runs slightly less at $0.02 per GB. Howerver, bandwidth may be the biggest cost factor for you not storage.

cperciva 10y ago
The price difference between S3 and Google Cloud Storage isn't enough to make a meaningful difference, particularly since I'd need to set up server code in Google Cloud to manage it. Bandwidth isn't a huge cost; but the server which keeps track of where all the bits are in S3 and shuffles them around is surprisingly expensive.
- nodesocket 10y ago
  Curious what AWS instance type you are using for that machine that keeps track of all the Tarsnap bits? I'm guessing it is highly available, so multiple instances of that type?
- boulos 10y ago
  Disclosure: I work on Google Cloud (and love the idea of Tarsnap, including the picodollars). What about if you were to age things into Nearline and Coldline using lifecycle policies? Also, someone claims you do multiple regions, which our "default" multiregional does at less than half the cost of S3. Admittedly we only recently (finally!) added per-object storage classes. But unless I misunderstand how your blocks work in tarsnap, wouldn't that be amenable to having some large portion as Nearline and possibly even Coldline? If not, I'd love to understand! (contact info in my profile).

nickpsecurity 10y ago

How many copies do you keep of the data to avoid corruption in storage?

cperciva 10y ago
Tarsnap data is stored in S3, with cryptographic authentication codes to detect any corruption; while data corruption has been detected in transit on many occasions (and requests retried as a result), I am not aware of S3 ever corrupting any data in situ.
- nickpsecurity 10y ago
  Thank you.

neurostimulant 10y ago

Downloading large amount of data from tarsnap archive is very slow. Any tips/tricks to speed them up?

cperciva 10y ago
Scott Wheeler wrote a tool for speeding up extracts by running multiple tarsnap processes: https://github.com/directededge/redsnapper https://github.com/directededge/redsnapper There's also the trivial (but often unhelpful) advice of "be closer to the server" -- the problem is one of round trip latencies since the internal design of tar (and thus tarsnap) works on the principle of "read a block; do something with it; repeat". This is what I feel to be the biggest technical pain point in tarsnap, and I know in theory how to fix it, but every time I've started focusing my energy on it other things have come up requiring more urgent attention. :-/

danohu 10y ago

How can I download backups into a different directory from where they originated? [I was just trying out tarsnap, go to the restore test section, and was a bit stumped. It seems like "-xC /other/dir" might do it, but the man page is pretty opaque. I'd like to test a restore without the risk of clobbering my data with some misconfigured backup!]

crzwdjk 10y ago
It works just like tar for the most part, so it extracts into whatever the current working directory happens to be. I'd just do (cd /other/dir; tarsnap -x -f whatever-archive).
- danohu 10y ago
  Thanks, that makes sense

caivvoacmh 10y ago

Hi cperciva can you please explain the following: 1) "Every Tarsnap archive acts like it is completely independent of all other archives" Lets say I'm backing up `/somedir` that looks like this: # archive1 - initial backup ── somedir ├── file1.txt ├── file2.txt └── file3.txt # archive 2 - 2nd backup ── somedir └── file3.txt (modified) Are you suggesting that if I delete `archive1` and `rm -rf /somedir` that I can restore `/somedir` with `archive2` alone? If so, how is that possible? 2) Does tarsnap stream the backup data to the tarsnap service or does it create the encrypted archive and only after this upload it? For instance, if I've got 100 GB to backup (first time), do I need 100 GB of free space for tarsnap to work?

cperciva 10y ago
Are you suggesting that if I delete `archive1` and `rm -rf /somedir` that I can restore `/somedir` with `archive2` alone? If archive2 only contains somedir/file3.txt, then if you extract archive2 you'll get somedir/file3.txt but not file1.txt or file2.txt. But if you create archive2 containing all three files, tarsnap will recognize the duplicate data in file1.txt and file2.txt and not re-upload it; but when you extract archive2 you'll get all three blocks. Tarsnap reference-counts blocks of data so that when you delete archive1 it only deletes the data which is not used by any of the remaining archives. I often tell new Tarsnap users that they should start by forgetting everything they know about incremental backups. Tell Tarsnap what data you want to have in an archive, and let it do the work of figuring out what's new. Does tarsnap stream the backup data to the tarsnap service or does it create the encrypted archive and only after this upload it? Tarsnap uploads data as it collects it. Tarsnap needs a small amount of disk space to keep track of which blocks have been uploaded previously, but it's less than 1% of the size of the data archived.