4 ms·

I think this was the main reason (from the linked article) LOL: "Brotli is a compression algorithm developed by Google." They have no idea about Zstandard nor

I think this was the main reason (from the linked article) LOL: "Brotli is a compression algorithm developed by Google." They have no idea about Zstandard nor ANS/FSE comparing it with LZ77. Sheer incompetence.

mort96 9mo ago
EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 https://news.ycombinator.com/item?id=46723158 I just took all PDFs I had in my downloads folder (55, totaling 47M). These are invoices, data sheets, employment contracts, schematics, research reports, a bunch of random stuff really. I compressed them all with 'zstd --ultra -22', 'brotli -9', 'xz -9' and 'gzip -9'. Here are the results: +------+------+-----+------+--------+ | none | zstd | xz | gzip | brotli | +------|------|-----|------|--------| | 47M | 45M | 39M | 38M | 37M | +------+------+-----+------+--------+ Here's a table with all the files: +------+------+------+------+--------+ | raw | zstd | xz | gzip | brotli | +------+------+------+------+--------+ | 12K | 12K | 12K | 12K | 12K | | 20K | 20K | 20K | 20K | 20K | x5 | 24K | 20K | 20K | 20K | 20K | x5 | 28K | 24K | 24K | 24K | 24K | | 28K | 24K | 24K | 24K | 24K | | 32K | 20K | 20K | 20K | 20K | x3 | 32K | 24K | 24K | 24K | 24K | | 40K | 32K | 32K | 32K | 32K | | 44K | 40K | 40K | 40K | 40K | | 44K | 40K | 40K | 40K | 40K | | 48K | 36K | 36K | 36K | 36K | | 48K | 48K | 48K | 48K | 48K | | 76K | 128K | 72K | 72K | 72K | | 84K | 140K | 84K | 80K | 80K | x7 | 88K | 136K | 76K | 76K | 76K | | 124K | 152K | 88K | 92K | 92K | | 124K | 152K | 92K | 96K | 92K | | 140K | 160K | 100K | 100K | 100K | | 152K | 188K | 128K | 128K | 132K | | 188K | 192K | 184K | 184K | 184K | | 264K | 256K | 240K | 244K | 240K | | 320K | 256K | 228K | 232K | 228K | | 440K | 448K | 408K | 408K | 408K | | 448K | 448K | 432K | 432K | 432K | | 516K | 384K | 376K | 384K | 376K | | 992K | 320K | 260K | 296K | 280K | | 1.0M | 2.0M | 1.0M | 1.0M | 1.0M | | 1.1M | 192K | 192K | 228K | 200K | | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M | | 1.2M | 1.1M | 1.0M | 1.0M | 1.0M | | 1.3M | 2.0M | 1.1M | 1.1M | 1.1M | | 1.7M | 2.0M | 1.7M | 1.7M | 1.7M | | 1.9M | 960K | 896K | 952K | 916K | | 2.9M | 2.0M | 1.3M | 1.4M | 1.4M | | 3.2M | 4.0M | 3.1M | 3.1M | 3.0M | | 3.7M | 4.0M | 3.5M | 3.5M | 3.5M | | 6.4M | 4.0M | 4.1M | 3.7M | 3.5M | | 6.4M | 6.0M | 6.1M | 5.8M | 5.7M | | 9.7M | 10M | 10M | 9.5M | 9.4M | +------+------+------+------+--------+ Zstd is surprisingly bad on this data set. I'm guessing it struggles with the already-compressed image data in some of these PDFs. Going by only compression ratio, brotli is clearly better than the rest here and zstd is the worst. You'd have to find some other reason (maybe decompression speed, maybe spec complexity, or maybe you just trust Facebook more than Google) to choose zstd over brotli, going by my results. I wish I could share the data set for reproducibility, but I obviously can't just share every PDF I happened to have laying around in my downloads folder :p
- noname120 9mo ago
  Why not use a more widespread compression algorithm (e.g. gzip) considering that Brotli barely performs better at all? Sounds like a pain for portability
  - mort96 9mo ago
    I'm not sold on the idea of adding compression to PDF at all, I'm not convinced that the space savings are worth breaking compatibility with older readers. Especially when you consider that you can just compress it in transit with e.g HTTP's 'Content-Encoding' without any special PDF reader support. (You can even use 'Content-Encoding: br' for brotli!) If you do wanna change PDF backwards-incompatibly, I don't think there's a significant advantage to choosing gzip to be honest, both brotli and zstd are pretty widely available these days and should be fairly easy to vendor. But yeah, it's a slight advantage I guess. Though I would expect that there are other PDF data sets where brotli has a larger advantage compared to gzip. But what I really don't get is all the calls to use zstd instead of brotli and treating the choise to use brotli instead of zstd as some form of Google conspiracy. (Is Facebook really better?)
    - ksec 9mo ago
      >But what I really don't get is all the calls to use zstd instead of brotli and treating the choise to use brotli instead of zstd as some form of Google conspiracy. (Is Facebook really better?) I may dislike Google. But my support of JPEG XL and Zstd has nothing to do with competition tech being Google at all. I simply think JPEG XL and Zstd are better technology.
- noname120 9mo ago
  Could you add compression and decompression speeds to your table?
  - deleted 9mo ago
    [deleted]
  - 9mo ago
cortesoft 9mo ago
I can’t imagine the people actually doing the technical work don’t know about Zstandard.

mort96 9mo ago

EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 https://news.ycombinator.com/item?id=46723158 I just took all PDFs I had in my downloads folder (55, totaling 47M). These are invoices, data sheets, employment contracts, schematics, research reports, a bunch of random stuff really. I compressed them all with 'zstd --ultra -22', 'brotli -9', 'xz -9' and 'gzip -9'. Here are the results: +------+------+-----+------+--------+ | none | zstd | xz | gzip | brotli | +------|------|-----|------|--------| | 47M | 45M | 39M | 38M | 37M | +------+------+-----+------+--------+ Here's a table with all the files: +------+------+------+------+--------+ | raw | zstd | xz | gzip | brotli | +------+------+------+------+--------+ | 12K | 12K | 12K | 12K | 12K | | 20K | 20K | 20K | 20K | 20K | x5 | 24K | 20K | 20K | 20K | 20K | x5 | 28K | 24K | 24K | 24K | 24K | | 28K | 24K | 24K | 24K | 24K | | 32K | 20K | 20K | 20K | 20K | x3 | 32K | 24K | 24K | 24K | 24K | | 40K | 32K | 32K | 32K | 32K | | 44K | 40K | 40K | 40K | 40K | | 44K | 40K | 40K | 40K | 40K | | 48K | 36K | 36K | 36K | 36K | | 48K | 48K | 48K | 48K | 48K | | 76K | 128K | 72K | 72K | 72K | | 84K | 140K | 84K | 80K | 80K | x7 | 88K | 136K | 76K | 76K | 76K | | 124K | 152K | 88K | 92K | 92K | | 124K | 152K | 92K | 96K | 92K | | 140K | 160K | 100K | 100K | 100K | | 152K | 188K | 128K | 128K | 132K | | 188K | 192K | 184K | 184K | 184K | | 264K | 256K | 240K | 244K | 240K | | 320K | 256K | 228K | 232K | 228K | | 440K | 448K | 408K | 408K | 408K | | 448K | 448K | 432K | 432K | 432K | | 516K | 384K | 376K | 384K | 376K | | 992K | 320K | 260K | 296K | 280K | | 1.0M | 2.0M | 1.0M | 1.0M | 1.0M | | 1.1M | 192K | 192K | 228K | 200K | | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M | | 1.2M | 1.1M | 1.0M | 1.0M | 1.0M | | 1.3M | 2.0M | 1.1M | 1.1M | 1.1M | | 1.7M | 2.0M | 1.7M | 1.7M | 1.7M | | 1.9M | 960K | 896K | 952K | 916K | | 2.9M | 2.0M | 1.3M | 1.4M | 1.4M | | 3.2M | 4.0M | 3.1M | 3.1M | 3.0M | | 3.7M | 4.0M | 3.5M | 3.5M | 3.5M | | 6.4M | 4.0M | 4.1M | 3.7M | 3.5M | | 6.4M | 6.0M | 6.1M | 5.8M | 5.7M | | 9.7M | 10M | 10M | 9.5M | 9.4M | +------+------+------+------+--------+ Zstd is surprisingly bad on this data set. I'm guessing it struggles with the already-compressed image data in some of these PDFs. Going by only compression ratio, brotli is clearly better than the rest here and zstd is the worst. You'd have to find some other reason (maybe decompression speed, maybe spec complexity, or maybe you just trust Facebook more than Google) to choose zstd over brotli, going by my results. I wish I could share the data set for reproducibility, but I obviously can't just share every PDF I happened to have laying around in my downloads folder :p

noname120 9mo ago
Why not use a more widespread compression algorithm (e.g. gzip) considering that Brotli barely performs better at all? Sounds like a pain for portability
- mort96 9mo ago
  I'm not sold on the idea of adding compression to PDF at all, I'm not convinced that the space savings are worth breaking compatibility with older readers. Especially when you consider that you can just compress it in transit with e.g HTTP's 'Content-Encoding' without any special PDF reader support. (You can even use 'Content-Encoding: br' for brotli!) If you do wanna change PDF backwards-incompatibly, I don't think there's a significant advantage to choosing gzip to be honest, both brotli and zstd are pretty widely available these days and should be fairly easy to vendor. But yeah, it's a slight advantage I guess. Though I would expect that there are other PDF data sets where brotli has a larger advantage compared to gzip. But what I really don't get is all the calls to use zstd instead of brotli and treating the choise to use brotli instead of zstd as some form of Google conspiracy. (Is Facebook really better?)
  - ksec 9mo ago
    >But what I really don't get is all the calls to use zstd instead of brotli and treating the choise to use brotli instead of zstd as some form of Google conspiracy. (Is Facebook really better?) I may dislike Google. But my support of JPEG XL and Zstd has nothing to do with competition tech being Google at all. I simply think JPEG XL and Zstd are better technology.
noname120 9mo ago
Could you add compression and decompression speeds to your table?
- deleted 9mo ago
  [deleted]
- 9mo ago

cortesoft 9mo ago

I can’t imagine the people actually doing the technical work don’t know about Zstandard.