4 ms·

Is there some nuance in the phrase “compressed storage” here? Like, is that a subset of overall storage, with some objects not being compressed? I imagine that

by jl6 4y ago

Is there some nuance in the phrase “compressed storage” here? Like, is that a subset of overall storage, with some objects not being compressed? I imagine that a great proportion of S3 objects are not amenable to zstd compression, either because they are natively in a compressed format (e.g. JPEG) or because the customer has already compressed them to realise the saving themselves.

badpun 4y ago
Certainly, if used as a data store for a data-lake-like solution, it's much more convenient to store data in uncompressed form, knowing that the underlying storage solution will take care of that for you. This way, you can just dump Json files in without it feeling like a collosal waste (or having to do extra work of converting them to a compressed format, such as Avro).
- paranoidrobot 4y ago
  Storage level compression usually benefits whomever owns the storage. I doubt Amazon intends to bill on compressed size.
infogulch 4y ago
I believe zstd can detect incompressible data quickly and falls back to a low-overhead passthrough mode in that case. So you can just run everything through zstd and it handles those cases pretty optimally already.
- le-mark 4y ago
  Hmm this made me chuckle, is that not equivalent to a test for randomness, which is very very hard indeed?
  - Cyberdog 4y ago
    I think a useful test for this case could be pretty simple. Compress maybe 10% of the file (up to some reasonable maximum, like maybe a few megabytes) and check the compression ratio. If it's not something better than maybe 1.1x, give up, throw out the compressed data, and just pass through the file. Otherwise keep going.
  - hexxagone 4y ago
    It is not. Just look at the frequencies in the data (histogram) and compute the order 0 entropy. Then you know if your data is compressible.
hyuijk 4y ago
If the customers stored .gz/.zip/... files you could transparently transcompress them to .zstd and back (with an added test of course)
- sgtnoodle 4y ago
  It seems like that would cause problems. Right off the bat, any integrity hashes like md5 or sha256 for the original compressed files would likely be corrupted. Also, the compressed archive could have been structurally baked in a specific way that's meaningful to the customer. Zip archives in particular can have arbitrary data pretended to them. I suppose you could speculatively decompress and then re-compress and see if you get the original compressed file back, and maybe most people happen to use the same compression implementations with default settings.
  - mike_hock 4y ago
    That wastes a lot of CPU compared to just running it through zstd.
  - spockz 4y ago
    You would only apply the compression on the internally stored file and then decompress when retrieving it for the customer again. That way all the hashes and original structure of the user are retained.
    - klauspost 4y ago
      You would need to be able to reconstruct the input file bit-by-bit. S3 Clients expect to get back what they sent, exactly. This puts a serious limitation on your compression. You would only be able to re-do the entropy coding part of DEFLATE, which is actually pretty good. You would still need to store the original Huffman tables for each block, so you can reconstruct the entropy coding exactly. I doubt this would even gain you a single percentage.
  - mkup 4y ago
    Besides ZIP metadata, there may be flushes in specific points in the deflate stream (which applies to .gz files as well). These flushes reset the compression dictionary and make further compressed data independent from previous data (at the expense of losing some compression efficiency). So: AWS S3 customer may have injected these flushes to their .gz files (for whatever reason, e.g. steganography), and after gzip-to-zstd-to-gzip transcompression this steganographic data will be be lost (and of course sha256 and other similar hashes will be different, as you already said).
scrame 4y ago
It's written through a virtual file system which stores in blocks and not files, the implementation of that could happily run zstd on the way to the multiple physical disks. This isn't about them shrinking your individual files, its compression on their internal transport.

by jl6 4y ago

badpun 4y ago

Certainly, if used as a data store for a data-lake-like solution, it's much more convenient to store data in uncompressed form, knowing that the underlying storage solution will take care of that for you. This way, you can just dump Json files in without it feeling like a collosal waste (or having to do extra work of converting them to a compressed format, such as Avro).

paranoidrobot 4y ago
Storage level compression usually benefits whomever owns the storage. I doubt Amazon intends to bill on compressed size.

infogulch 4y ago

I believe zstd can detect incompressible data quickly and falls back to a low-overhead passthrough mode in that case. So you can just run everything through zstd and it handles those cases pretty optimally already.

le-mark 4y ago
Hmm this made me chuckle, is that not equivalent to a test for randomness, which is very very hard indeed?
- Cyberdog 4y ago
  I think a useful test for this case could be pretty simple. Compress maybe 10% of the file (up to some reasonable maximum, like maybe a few megabytes) and check the compression ratio. If it's not something better than maybe 1.1x, give up, throw out the compressed data, and just pass through the file. Otherwise keep going.
- hexxagone 4y ago
  It is not. Just look at the frequencies in the data (histogram) and compute the order 0 entropy. Then you know if your data is compressible.

hyuijk 4y ago

If the customers stored .gz/.zip/... files you could transparently transcompress them to .zstd and back (with an added test of course)

sgtnoodle 4y ago
It seems like that would cause problems. Right off the bat, any integrity hashes like md5 or sha256 for the original compressed files would likely be corrupted. Also, the compressed archive could have been structurally baked in a specific way that's meaningful to the customer. Zip archives in particular can have arbitrary data pretended to them. I suppose you could speculatively decompress and then re-compress and see if you get the original compressed file back, and maybe most people happen to use the same compression implementations with default settings.
- mike_hock 4y ago
  That wastes a lot of CPU compared to just running it through zstd.
- spockz 4y ago
  You would only apply the compression on the internally stored file and then decompress when retrieving it for the customer again. That way all the hashes and original structure of the user are retained.
  - klauspost 4y ago
    You would need to be able to reconstruct the input file bit-by-bit. S3 Clients expect to get back what they sent, exactly. This puts a serious limitation on your compression. You would only be able to re-do the entropy coding part of DEFLATE, which is actually pretty good. You would still need to store the original Huffman tables for each block, so you can reconstruct the entropy coding exactly. I doubt this would even gain you a single percentage.
- mkup 4y ago
  Besides ZIP metadata, there may be flushes in specific points in the deflate stream (which applies to .gz files as well). These flushes reset the compression dictionary and make further compressed data independent from previous data (at the expense of losing some compression efficiency). So: AWS S3 customer may have injected these flushes to their .gz files (for whatever reason, e.g. steganography), and after gzip-to-zstd-to-gzip transcompression this steganographic data will be be lost (and of course sha256 and other similar hashes will be different, as you already said).

scrame 4y ago

It's written through a virtual file system which stores in blocks and not files, the implementation of that could happily run zstd on the way to the multiple physical disks. This isn't about them shrinking your individual files, its compression on their internal transport.