5 ms·

For those interested in the question, it's worth looking at how zstd was made. I have to archive hundreds of TB of JSON about tweets, so I did a compressor bake

For those interested in the question, it's worth looking at how zstd was made. I have to archive hundreds of TB of JSON about tweets, so I did a compressor bake-off last year. I was surprised at how much better zstd did. Turns out designing for the low-level details of modern processors can make a big difference: https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard/ https://engineering.fb.com/2016/08/31/core-data/smaller-and-...

userbinator 5y ago
Somehow, I suspect not using JSON in the first place might yield better absolute results since the compressor wouldn't have to deal with all the redundancy of JSON and instead focus on the content only.
- wpietri 5y ago
  For many applications, this one included, "the first place" is out of reach. What I've got is an awful lot of incoming JSON data. I'd like to keep a copy in case we need it later. Is it possible for me to write and test a robust custom compressor that does better in both storage and compression costs than any of the modern compressors? Possibly. Is that true even though one of FB's goals was specifically optimizing JSON compression? Maybe. Can I build it for the few hundred bucks per month in S3 costs I might save? Definitely not.
  - kragen 5y ago
    Also, surprisingly often, doing something reasonably workable but far from optimal, and then throwing generic compression algorithms at it, can work very nearly as well as carefully doing the precisely optimal thing. SNMP uses BER, which has lots of carefully thought out compact representations for data types, and COAP has CBOR, and in both cases you can usually do as well or better on space by running a simple ASCII syntax through LZW. Usually not that much slower, either.
    - rixed 5y ago
      Not fair, unless you also compress the BER or CBOR (which are primarily intended to serialize/deserialize faster, not to compress)
      - kragen 5y ago
        I don't know if you've ever benchmarked BER and CBOR implementations, but I have, and they're astoundingly shitty at serializing and deserializing faster, even when compared against things like JSON. They do usually beat LZW though.

userbinator 5y ago

Somehow, I suspect not using JSON in the first place might yield better absolute results since the compressor wouldn't have to deal with all the redundancy of JSON and instead focus on the content only.

wpietri 5y ago
For many applications, this one included, "the first place" is out of reach. What I've got is an awful lot of incoming JSON data. I'd like to keep a copy in case we need it later. Is it possible for me to write and test a robust custom compressor that does better in both storage and compression costs than any of the modern compressors? Possibly. Is that true even though one of FB's goals was specifically optimizing JSON compression? Maybe. Can I build it for the few hundred bucks per month in S3 costs I might save? Definitely not.
- kragen 5y ago
  Also, surprisingly often, doing something reasonably workable but far from optimal, and then throwing generic compression algorithms at it, can work very nearly as well as carefully doing the precisely optimal thing. SNMP uses BER, which has lots of carefully thought out compact representations for data types, and COAP has CBOR, and in both cases you can usually do as well or better on space by running a simple ASCII syntax through LZW. Usually not that much slower, either.
  - rixed 5y ago
    Not fair, unless you also compress the BER or CBOR (which are primarily intended to serialize/deserialize faster, not to compress)
    - kragen 5y ago
      I don't know if you've ever benchmarked BER and CBOR implementations, but I have, and they're astoundingly shitty at serializing and deserializing faster, even when compared against things like JSON. They do usually beat LZW though.