5 ms·

works great on giant uncompressed database backups though

sliken 7y ago
Sure, or just datadump | gzip > /dev/tape or similar. Tape loves to brag about bandwidth/capacity and then hide the assumed 2.5x compression in a footnote.
- namibj 7y ago
  Please consider zstd instead. It is superior in every way except memory usage and compatibility with old systems.
  - sliken 7y ago
    Sure. Anything, not like text compression is hard.
  - erik_seaberg 7y ago
    zstd looks good but it's only three years old, so use cases need to fit in the lifetime of today's platform. gzip is about to turn 27 and everyone has heard of it, which makes me pretty confident in finding decoders after another decade or two. (Actually this problem is why I haven't owned a tape drive for a while.)
  - nightfly 7y ago
    pigz!
    - terrelln 7y ago
      Zstd also has a multithreaded mode. On the CLI `zstd -T0` tells zstd to use all available cores.
      - namibj 7y ago
        Please be aware that this will significantly impact compression ratio. So much actually, that I don't dare use it by default. I'd often use the non-multithreaded mode as for some data (perf.data files with large stack captures, most recently) and reasonable compression speeds of >=50MB/s input multi-threaded modes are not pareto-optimal. I'll get a small benchmark script for that done over lunch. Come back here later.
        Cyan4973 7y ago
        > Please be aware that this will significantly impact compression ratio. Is this a continuation from previous message ? Because multithreading mode of zstd doesn't impact its compression ratio. It's actually an interesting property of zstd : whatever the nb of threads used, the compression ratio remains the same (aka. reproducible).
- jolmg 7y ago
  Before reading up on LTO, I thought maybe it was some kind of physical compression where it would write the data closer together at the cost of read/write speed or something (doesn't really make sense in retrospect). I'm kind of disappointed that it was just regular data compression with an assumed ratio. :/
perl4ever 7y ago
I spend most of my time these days with a database that seems to compress to 4% of its size...

Sure, or just datadump | gzip > /dev/tape or similar. Tape loves to brag about bandwidth/capacity and then hide the assumed 2.5x compression in a footnote.

namibj 7y ago
Please consider zstd instead. It is superior in every way except memory usage and compatibility with old systems.
- sliken 7y ago
  Sure. Anything, not like text compression is hard.
- erik_seaberg 7y ago
  zstd looks good but it's only three years old, so use cases need to fit in the lifetime of today's platform. gzip is about to turn 27 and everyone has heard of it, which makes me pretty confident in finding decoders after another decade or two. (Actually this problem is why I haven't owned a tape drive for a while.)
- nightfly 7y ago
  pigz!
  - terrelln 7y ago
    Zstd also has a multithreaded mode. On the CLI `zstd -T0` tells zstd to use all available cores.
    - namibj 7y ago
      Please be aware that this will significantly impact compression ratio. So much actually, that I don't dare use it by default. I'd often use the non-multithreaded mode as for some data (perf.data files with large stack captures, most recently) and reasonable compression speeds of >=50MB/s input multi-threaded modes are not pareto-optimal. I'll get a small benchmark script for that done over lunch. Come back here later.
      - Cyan4973 7y ago
        > Please be aware that this will significantly impact compression ratio. Is this a continuation from previous message ? Because multithreading mode of zstd doesn't impact its compression ratio. It's actually an interesting property of zstd : whatever the nb of threads used, the compression ratio remains the same (aka. reproducible).
jolmg 7y ago
Before reading up on LTO, I thought maybe it was some kind of physical compression where it would write the data closer together at the cost of read/write speed or something (doesn't really make sense in retrospect). I'm kind of disappointed that it was just regular data compression with an assumed ratio. :/

perl4ever 7y ago

I spend most of my time these days with a database that seems to compress to 4% of its size...