4 ms·

What about tape gives it superior compression? Why can’t that same compression be applied to any storage?

jarfil 7y ago
Nothing. They estimate an average compression based on an average dataset with uncompressed text data. That wouldn't work for things like already compressed video, audio, encrypted data, and so on.
- cl0ckt0wer 7y ago
  works great on giant uncompressed database backups though
  - sliken 7y ago
    Sure, or just datadump | gzip > /dev/tape or similar. Tape loves to brag about bandwidth/capacity and then hide the assumed 2.5x compression in a footnote.
    - namibj 7y ago
      Please consider zstd instead. It is superior in every way except memory usage and compatibility with old systems.
      - sliken 7y ago
        Sure. Anything, not like text compression is hard.
      - erik_seaberg 7y ago
        zstd looks good but it's only three years old, so use cases need to fit in the lifetime of today's platform. gzip is about to turn 27 and everyone has heard of it, which makes me pretty confident in finding decoders after another decade or two. (Actually this problem is why I haven't owned a tape drive for a while.)
      - nightfly 7y ago
        pigz!
        terrelln 7y ago
        Zstd also has a multithreaded mode. On the CLI `zstd -T0` tells zstd to use all available cores.
        namibj 7y ago
        Please be aware that this will significantly impact compression ratio. So much actually, that I don't dare use it by default. I'd often use the non-multithreaded mode as for some data (perf.data files with large stack captures, most recently) and reasonable compression speeds of >=50MB/s input multi-threaded modes are not pareto-optimal. I'll get a small benchmark script for that done over lunch. Come back here later.
        Cyan4973 7y ago
        > Please be aware that this will significantly impact compression ratio. Is this a continuation from previous message ? Because multithreading mode of zstd doesn't impact its compression ratio. It's actually an interesting property of zstd : whatever the nb of threads used, the compression ratio remains the same (aka. reproducible).
    - jolmg 7y ago
      Before reading up on LTO, I thought maybe it was some kind of physical compression where it would write the data closer together at the cost of read/write speed or something (doesn't really make sense in retrospect). I'm kind of disappointed that it was just regular data compression with an assumed ratio. :/
  - perl4ever 7y ago
    I spend most of my time these days with a database that seems to compress to 4% of its size...
- userbinator 7y ago
  I really wish they wouldn't add stuff like compression and so forth, because I feel like it's an "abstraction layer violation" and it's misleading in a way that the majority of other storage media isn't; for example, a 1TB hard drive is going to store (at least) 10^12 bytes regardless of what those bytes contain. A 60 minute audio tape will always record 60 minutes of pure silence, white noise, or anything in between. Yet somehow data tape is marketed with a theoretical compressed capacity.
  - nightfly 7y ago
    It's super annoying, and it's annoying to (attempt) to disable the hardware compression on the tape drives as well. We've got a tape library with 2 two LTO7 tape drives, and even though we're accessing it through the linux device file that should be accessing the tapes in an un-compressed mode diagnostic tools indicate that compression is still be used when writing to tape. Since we're pre-compressing our data with gzip having the tape drive try and apply compression is a wasted effort and I wonder if we're losing write speed to it.
jolmg 7y ago
The compression ratio seems to actually be much worse than gzip, but it's also faster.
- makapuf 7y ago
  on a core i5 you can use lz4 and compress LZ4 for a ratio around 2 at 370 MB/s (ans decompress it faster) - for one core.
userbinator 7y ago
It's not about compression, but total storage capacity. Imagine a disk. It has two sides and a certain surface area, as well as a certain density (bits per square in/cm). A standard 3.5" HDD platter has a bit more than 4200mm^2 of surface area (having been unable to quickly find the dimensions of an HDD platter, I grabbed the dimensions of a floppy disk from the standard) per side, or a little less than 10000mm^2 per platter; I'm being deliberately rough with the numbers since all we need is an order of magnitude. Imagine a tape. It's long and thin, and can be rolled up. An LTO-8 is 960m long and 12.65mm wide, for a total surface area of 12,144,000mm^2. Tape has a lower recording density than HDDs (LTO-8: ~20Kbit/mm * 6656 tracks, or ~128MB/mm^2 compared to ~1Tbit/in^2 = ~185MB/mm^2 for a HDD), but more than makes up for it with its length.
- caf 7y ago
  After multiplying the single track linear density by the number of tracks, you also need to divide by the tape width to convert to a recording area density number. So you end up with about 10.9Mbit/mm².
- sliken 7y ago
  That's all well and good, but tapes are smaller than disks. LTO-8 is 12TB and the 14TB disks have been out for a good while. Also with inefficient tapes you end up spending many more tapes for inefficient incrementals, incredibly long seeks, and lack of efficient deduplication. Then there's other nasty surprises like a relatively narrow temperature range for tapes and an improper alignment of the head can result in tapes you can read and write... but when the drive dies you can't read the tapes anywhere else. Tapes still make sense above a certainly scale, but I think it's around 10PB before the real world costs of tapes, drives, robot, support agreements, extra capacity for inefficient incrementals/dedupe, etc. The big win with tapes is it's much more feasible to rotate tapes offsite.
  - adrian_b 7y ago
    > but tapes are smaller than disks Tapes are also physically smaller, therfore their capacity per volume is better than for HDDs. Two LTO Ultrium catridges have about the same volume as a 3.5" HDDs, so you have 24 TB in the same space as a 14 TB HDD. Moreover, the HDDs are much heavier. You can easily carry a suitcase with 20 tape cartridges, i.e. with 240 TB. If you would carry a suitcase with 17 HDDs (238 TB), that would be around 14 kg (around 31 lbs). Possible but not at all comfortable.

Nothing. They estimate an average compression based on an average dataset with uncompressed text data. That wouldn't work for things like already compressed video, audio, encrypted data, and so on.

cl0ckt0wer 7y ago
works great on giant uncompressed database backups though
- sliken 7y ago
  Sure, or just datadump | gzip > /dev/tape or similar. Tape loves to brag about bandwidth/capacity and then hide the assumed 2.5x compression in a footnote.
  - namibj 7y ago
    Please consider zstd instead. It is superior in every way except memory usage and compatibility with old systems.
    - sliken 7y ago
      Sure. Anything, not like text compression is hard.
    - erik_seaberg 7y ago
      zstd looks good but it's only three years old, so use cases need to fit in the lifetime of today's platform. gzip is about to turn 27 and everyone has heard of it, which makes me pretty confident in finding decoders after another decade or two. (Actually this problem is why I haven't owned a tape drive for a while.)
    - nightfly 7y ago
      pigz!
      - terrelln 7y ago
        Zstd also has a multithreaded mode. On the CLI `zstd -T0` tells zstd to use all available cores.
        namibj 7y ago
        Please be aware that this will significantly impact compression ratio. So much actually, that I don't dare use it by default. I'd often use the non-multithreaded mode as for some data (perf.data files with large stack captures, most recently) and reasonable compression speeds of >=50MB/s input multi-threaded modes are not pareto-optimal. I'll get a small benchmark script for that done over lunch. Come back here later.
        Cyan4973 7y ago
        > Please be aware that this will significantly impact compression ratio. Is this a continuation from previous message ? Because multithreading mode of zstd doesn't impact its compression ratio. It's actually an interesting property of zstd : whatever the nb of threads used, the compression ratio remains the same (aka. reproducible).
  - jolmg 7y ago
    Before reading up on LTO, I thought maybe it was some kind of physical compression where it would write the data closer together at the cost of read/write speed or something (doesn't really make sense in retrospect). I'm kind of disappointed that it was just regular data compression with an assumed ratio. :/
- perl4ever 7y ago
  I spend most of my time these days with a database that seems to compress to 4% of its size...
userbinator 7y ago
I really wish they wouldn't add stuff like compression and so forth, because I feel like it's an "abstraction layer violation" and it's misleading in a way that the majority of other storage media isn't; for example, a 1TB hard drive is going to store (at least) 10^12 bytes regardless of what those bytes contain. A 60 minute audio tape will always record 60 minutes of pure silence, white noise, or anything in between. Yet somehow data tape is marketed with a theoretical compressed capacity.
- nightfly 7y ago
  It's super annoying, and it's annoying to (attempt) to disable the hardware compression on the tape drives as well. We've got a tape library with 2 two LTO7 tape drives, and even though we're accessing it through the linux device file that should be accessing the tapes in an un-compressed mode diagnostic tools indicate that compression is still be used when writing to tape. Since we're pre-compressing our data with gzip having the tape drive try and apply compression is a wasted effort and I wonder if we're losing write speed to it.

jolmg 7y ago

The compression ratio seems to actually be much worse than gzip, but it's also faster.

makapuf 7y ago
on a core i5 you can use lz4 and compress LZ4 for a ratio around 2 at 370 MB/s (ans decompress it faster) - for one core.

userbinator 7y ago

It's not about compression, but total storage capacity. Imagine a disk. It has two sides and a certain surface area, as well as a certain density (bits per square in/cm). A standard 3.5" HDD platter has a bit more than 4200mm^2 of surface area (having been unable to quickly find the dimensions of an HDD platter, I grabbed the dimensions of a floppy disk from the standard) per side, or a little less than 10000mm^2 per platter; I'm being deliberately rough with the numbers since all we need is an order of magnitude. Imagine a tape. It's long and thin, and can be rolled up. An LTO-8 is 960m long and 12.65mm wide, for a total surface area of 12,144,000mm^2. Tape has a lower recording density than HDDs (LTO-8: ~20Kbit/mm * 6656 tracks, or ~128MB/mm^2 compared to ~1Tbit/in^2 = ~185MB/mm^2 for a HDD), but more than makes up for it with its length.

caf 7y ago
After multiplying the single track linear density by the number of tracks, you also need to divide by the tape width to convert to a recording area density number. So you end up with about 10.9Mbit/mm².
sliken 7y ago
That's all well and good, but tapes are smaller than disks. LTO-8 is 12TB and the 14TB disks have been out for a good while. Also with inefficient tapes you end up spending many more tapes for inefficient incrementals, incredibly long seeks, and lack of efficient deduplication. Then there's other nasty surprises like a relatively narrow temperature range for tapes and an improper alignment of the head can result in tapes you can read and write... but when the drive dies you can't read the tapes anywhere else. Tapes still make sense above a certainly scale, but I think it's around 10PB before the real world costs of tapes, drives, robot, support agreements, extra capacity for inefficient incrementals/dedupe, etc. The big win with tapes is it's much more feasible to rotate tapes offsite.
- adrian_b 7y ago
  > but tapes are smaller than disks Tapes are also physically smaller, therfore their capacity per volume is better than for HDDs. Two LTO Ultrium catridges have about the same volume as a 3.5" HDDs, so you have 24 TB in the same space as a 14 TB HDD. Moreover, the HDDs are much heavier. You can easily carry a suitcase with 20 tape cartridges, i.e. with 240 TB. If you would carry a suitcase with 17 HDDs (238 TB), that would be around 14 kg (around 31 lbs). Possible but not at all comfortable.