5 ms·

There is one recommendation there I find a bit questionable and thats encryption. If you are out of options and restoring from tape, might be better to have it

There is one recommendation there I find a bit questionable and thats encryption. If you are out of options and restoring from tape, might be better to have it uncompressed and not encrypted. Its possible, after some physical disaster that you are on somebody elses infrastructure and having some encryption on your data doubles the problems you might have. I use an ancient LTO2 drive for last resort backups that are off cloud and off premises. Its more peace of mind than practical on a daily basis but I did find myself restoring a few files a couple of weeks ago as I had fat fingered an rm command. It was quicker than getting them from S3 glacier.

benjojo12 5y ago
I agree somewhat. Encryption is more critical on tape because there is no easy path to wiping a tape, and in a company situation if you need to erase something in your backups too (think GDPR erasure), then encryption is reasonably critical unless you want to go though all of your cold backups. For my archival use (the reason why I got into this in the first place) I do not encrypt nor compress the data going to tape. For server/desktop backups. they are compressed and encrypted.
- throw0101a 5y ago
  > […] nor compress the data going to tape. Just to note that tape drives have built-in compression that generally is done transparently in the background. So while using something like zstd (per the article) may get more bits on a given tape, there is some compression that one gets "for free" without doing anything at all. * https://en.wikipedia.org/wiki/Linear_Tape-Open#Optional_technology https://en.wikipedia.org/wiki/Linear_Tape-Open#Optional_tech... * https://en.wikipedia.org/wiki/Magnetic_tape_data_storage#Data_compression https://en.wikipedia.org/wiki/Magnetic_tape_data_storage#Dat...
  - benjojo12 5y ago
    I mention this in the post itself
    - lights0123 5y ago
      You mention that they're advertised in the amount of compressed data that can be stored, not that they actually compress data themselves. I thought you meant that they assume you use a compression algorithm yourself.
      - benjojo12 5y ago
        Ah, ok fair enough! I should have pointed that out more clearly!
    - throw0101a 5y ago
      You wrote: > Drives above LTO-4 have built-in hardware encryption, however I would steer away from using it and instead just encrypt data yourself (possibly with the tool I helped make called age!). Like most things, you should also consider compressing your data before encrypting and writing it to tape. LTO tape capacities are often quoted in their “compressed capacity” which is a little cheeky since it assumes basically over a 50% compression ratio, this is not at all likely to be true if you are writing video or other lossy mediums like images etc to the tape. I generally run my data through zstd to compress and then age to encrypt. Zstd and age are quite fast and I’ve not found them to impede performance noticeably. If someone is not familiar with tape drives, I think it would be easy not to realize that the compression is built into drives like the explicitly called out "built-in hardware encryption".
- lostapathy 5y ago
  > Encryption is more critical on tape because there is no easy path to wiping a tape. I used to work for a government agency. We ran backup tapes that rotated out through a degaussing machine that spun them around for like 10 minutes to wipe them. It’s not common to have, but it’s definitely easy.
- rowanG077 5y ago
  It's trivial to wipe data on a tape with a degausser. You destroy the Tape in the process since it also wipes out factory written servo tracks.
  - kortex 5y ago
    Is there a way to restore the servo tracks? This sounds like the kind of hack a dedicated nerd could pull off with an arduino and duct tape.
    - ansible 5y ago
      Without looking into the specs, at the very least, you'd need to modify the LTO drive firmware. The drive itself isn't designed to operate without the servo tracks. Those are written to newly-manufactured tapes with special equipment at the factory. So, it would take a very dedicated nerd indeed.
    - rowanG077 5y ago
      Not that I know off. But the positioning on recent Gen LTOs is pretty tight. I don't think it's out of the realm of possibility for a dedicated nerd but it won't be trivial.
op00to 5y ago
The tapes compress themselves. There's no real need for file compression.
El_RIDO 5y ago
I'd like to suggest two arguments that made me use software encryption on my tapes instead: 1. You don't have to trust the hardware and can use tool I trust and have the sources for. 2. If you encrypt yourself you can combine it with something like par2 to generate error detection and recovery data, letting you restore the encrypted file off a damaged tape. A downside of encrypting yourself is that you can't benefit from the hardware compression either, hence the articles suggestion to do that in software before compressing as well. Personally, my tape writing workflow is: dar (per file compression, skips uncompressable mime types + encryption) followed by par2cmdline with 30% redundancy. For comparison: CD-ROMs have 33% redundancy information (8 bits per 24 bits, CIRC encoding).

benjojo12 5y ago

I agree somewhat. Encryption is more critical on tape because there is no easy path to wiping a tape, and in a company situation if you need to erase something in your backups too (think GDPR erasure), then encryption is reasonably critical unless you want to go though all of your cold backups. For my archival use (the reason why I got into this in the first place) I do not encrypt nor compress the data going to tape. For server/desktop backups. they are compressed and encrypted.

throw0101a 5y ago
> […] nor compress the data going to tape. Just to note that tape drives have built-in compression that generally is done transparently in the background. So while using something like zstd (per the article) may get more bits on a given tape, there is some compression that one gets "for free" without doing anything at all. * https://en.wikipedia.org/wiki/Linear_Tape-Open#Optional_technology https://en.wikipedia.org/wiki/Linear_Tape-Open#Optional_tech... * https://en.wikipedia.org/wiki/Magnetic_tape_data_storage#Data_compression https://en.wikipedia.org/wiki/Magnetic_tape_data_storage#Dat...
- benjojo12 5y ago
  I mention this in the post itself
  - lights0123 5y ago
    You mention that they're advertised in the amount of compressed data that can be stored, not that they actually compress data themselves. I thought you meant that they assume you use a compression algorithm yourself.
    - benjojo12 5y ago
      Ah, ok fair enough! I should have pointed that out more clearly!
  - throw0101a 5y ago
    You wrote: > Drives above LTO-4 have built-in hardware encryption, however I would steer away from using it and instead just encrypt data yourself (possibly with the tool I helped make called age!). Like most things, you should also consider compressing your data before encrypting and writing it to tape. LTO tape capacities are often quoted in their “compressed capacity” which is a little cheeky since it assumes basically over a 50% compression ratio, this is not at all likely to be true if you are writing video or other lossy mediums like images etc to the tape. I generally run my data through zstd to compress and then age to encrypt. Zstd and age are quite fast and I’ve not found them to impede performance noticeably. If someone is not familiar with tape drives, I think it would be easy not to realize that the compression is built into drives like the explicitly called out "built-in hardware encryption".
lostapathy 5y ago
> Encryption is more critical on tape because there is no easy path to wiping a tape. I used to work for a government agency. We ran backup tapes that rotated out through a degaussing machine that spun them around for like 10 minutes to wipe them. It’s not common to have, but it’s definitely easy.
rowanG077 5y ago
It's trivial to wipe data on a tape with a degausser. You destroy the Tape in the process since it also wipes out factory written servo tracks.
- kortex 5y ago
  Is there a way to restore the servo tracks? This sounds like the kind of hack a dedicated nerd could pull off with an arduino and duct tape.
  - ansible 5y ago
    Without looking into the specs, at the very least, you'd need to modify the LTO drive firmware. The drive itself isn't designed to operate without the servo tracks. Those are written to newly-manufactured tapes with special equipment at the factory. So, it would take a very dedicated nerd indeed.
  - rowanG077 5y ago
    Not that I know off. But the positioning on recent Gen LTOs is pretty tight. I don't think it's out of the realm of possibility for a dedicated nerd but it won't be trivial.

op00to 5y ago

The tapes compress themselves. There's no real need for file compression.

El_RIDO 5y ago

I'd like to suggest two arguments that made me use software encryption on my tapes instead: 1. You don't have to trust the hardware and can use tool I trust and have the sources for. 2. If you encrypt yourself you can combine it with something like par2 to generate error detection and recovery data, letting you restore the encrypted file off a damaged tape. A downside of encrypting yourself is that you can't benefit from the hardware compression either, hence the articles suggestion to do that in software before compressing as well. Personally, my tape writing workflow is: dar (per file compression, skips uncompressable mime types + encryption) followed by par2cmdline with 30% redundancy. For comparison: CD-ROMs have 33% redundancy information (8 bits per 24 bits, CIRC encoding).