5 ms·

Why not zstd?

by ksec 9mo ago

Why not zstd?

PunchyHamster 9mo ago
incompetence
- deleted 9mo ago
  [deleted]
- whizzx 9mo ago
  You can read about it here https://pdfa.org/brotli-compression-coming-to-pdf/ https://pdfa.org/brotli-compression-coming-to-pdf/
  - jeffbee 9mo ago
    That mentions zstd in a weird incomplete sentence, but never compares it.
    - F3nd0 9mo ago
      They don’t seem to provide a detailed comparison showing how each compression scheme fared at every task, but they do list (some of) their criteria and say they found Brotli the best of the bunch. I can’t tell if that’s a sensible conclusion or not, though. Maybe Brotli did better on code size or memory use?
    - eviks 9mo ago
      Hey, they did all the work and more, trust them!!! > Experts in the PDF Association’s PDF TWG undertook theoretical and experimental analysis of these schemes, reviewing decompression speed, compression speed, compression ratio achieved, memory usage, code size, standardisation, IP, interoperability, prototyping, sample file creation, and other due diligence tasks.
      - LoganDark 9mo ago
        I love when I perform all the due diligence tasks. You just can't counter that. Yes but, they did all the due diligence tasks. They considered all the factors. Every one. Think you have one they didn't consider? Nope.
        jsnell 9mo ago
        But they didn't write "all". They wrote "other", which absolutely does not imply full coverage. Maybe read things a bit more carefully before going all out on the snide comments?
        wizzwizz4 9mo ago
        In fact, they wrote "reviewing […] other due diligence tasks", which doesn't imply any coverage! This close, literal reading is an appropriate – nay, the only appropriate – way to draw conclusions about the degree of responsibility exhibited by the custodians of a living standard. By corollary, any criticism of this form could be rebuffed by appeal to a sufficiently-carefully-written press release.
        LoganDark 9mo ago
        It implies potential coverage of anything one could bring up. It creates a similar impression in my mind, because it becomes easy to claim you already considered something.
HackerThemAll 9mo ago
I think this was the main reason (from the linked article) LOL: "Brotli is a compression algorithm developed by Google." They have no idea about Zstandard nor ANS/FSE comparing it with LZ77. Sheer incompetence.
- mort96 9mo ago
  EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 https://news.ycombinator.com/item?id=46723158 I just took all PDFs I had in my downloads folder (55, totaling 47M). These are invoices, data sheets, employment contracts, schematics, research reports, a bunch of random stuff really. I compressed them all with 'zstd --ultra -22', 'brotli -9', 'xz -9' and 'gzip -9'. Here are the results: +------+------+-----+------+--------+ | none | zstd | xz | gzip | brotli | +------|------|-----|------|--------| | 47M | 45M | 39M | 38M | 37M | +------+------+-----+------+--------+ Here's a table with all the files: +------+------+------+------+--------+ | raw | zstd | xz | gzip | brotli | +------+------+------+------+--------+ | 12K | 12K | 12K | 12K | 12K | | 20K | 20K | 20K | 20K | 20K | x5 | 24K | 20K | 20K | 20K | 20K | x5 | 28K | 24K | 24K | 24K | 24K | | 28K | 24K | 24K | 24K | 24K | | 32K | 20K | 20K | 20K | 20K | x3 | 32K | 24K | 24K | 24K | 24K | | 40K | 32K | 32K | 32K | 32K | | 44K | 40K | 40K | 40K | 40K | | 44K | 40K | 40K | 40K | 40K | | 48K | 36K | 36K | 36K | 36K | | 48K | 48K | 48K | 48K | 48K | | 76K | 128K | 72K | 72K | 72K | | 84K | 140K | 84K | 80K | 80K | x7 | 88K | 136K | 76K | 76K | 76K | | 124K | 152K | 88K | 92K | 92K | | 124K | 152K | 92K | 96K | 92K | | 140K | 160K | 100K | 100K | 100K | | 152K | 188K | 128K | 128K | 132K | | 188K | 192K | 184K | 184K | 184K | | 264K | 256K | 240K | 244K | 240K | | 320K | 256K | 228K | 232K | 228K | | 440K | 448K | 408K | 408K | 408K | | 448K | 448K | 432K | 432K | 432K | | 516K | 384K | 376K | 384K | 376K | | 992K | 320K | 260K | 296K | 280K | | 1.0M | 2.0M | 1.0M | 1.0M | 1.0M | | 1.1M | 192K | 192K | 228K | 200K | | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M | | 1.2M | 1.1M | 1.0M | 1.0M | 1.0M | | 1.3M | 2.0M | 1.1M | 1.1M | 1.1M | | 1.7M | 2.0M | 1.7M | 1.7M | 1.7M | | 1.9M | 960K | 896K | 952K | 916K | | 2.9M | 2.0M | 1.3M | 1.4M | 1.4M | | 3.2M | 4.0M | 3.1M | 3.1M | 3.0M | | 3.7M | 4.0M | 3.5M | 3.5M | 3.5M | | 6.4M | 4.0M | 4.1M | 3.7M | 3.5M | | 6.4M | 6.0M | 6.1M | 5.8M | 5.7M | | 9.7M | 10M | 10M | 9.5M | 9.4M | +------+------+------+------+--------+ Zstd is surprisingly bad on this data set. I'm guessing it struggles with the already-compressed image data in some of these PDFs. Going by only compression ratio, brotli is clearly better than the rest here and zstd is the worst. You'd have to find some other reason (maybe decompression speed, maybe spec complexity, or maybe you just trust Facebook more than Google) to choose zstd over brotli, going by my results. I wish I could share the data set for reproducibility, but I obviously can't just share every PDF I happened to have laying around in my downloads folder :p

Why not zstd?

by ksec 9mo ago

Why not zstd?

PunchyHamster 9mo ago

incompetence

deleted 9mo ago
[deleted]
whizzx 9mo ago
You can read about it here https://pdfa.org/brotli-compression-coming-to-pdf/ https://pdfa.org/brotli-compression-coming-to-pdf/
- jeffbee 9mo ago
  That mentions zstd in a weird incomplete sentence, but never compares it.
  - F3nd0 9mo ago
    They don’t seem to provide a detailed comparison showing how each compression scheme fared at every task, but they do list (some of) their criteria and say they found Brotli the best of the bunch. I can’t tell if that’s a sensible conclusion or not, though. Maybe Brotli did better on code size or memory use?
  - eviks 9mo ago
    Hey, they did all the work and more, trust them!!! > Experts in the PDF Association’s PDF TWG undertook theoretical and experimental analysis of these schemes, reviewing decompression speed, compression speed, compression ratio achieved, memory usage, code size, standardisation, IP, interoperability, prototyping, sample file creation, and other due diligence tasks.
    - LoganDark 9mo ago
      I love when I perform all the due diligence tasks. You just can't counter that. Yes but, they did all the due diligence tasks. They considered all the factors. Every one. Think you have one they didn't consider? Nope.
      - jsnell 9mo ago
        But they didn't write "all". They wrote "other", which absolutely does not imply full coverage. Maybe read things a bit more carefully before going all out on the snide comments?
        wizzwizz4 9mo ago
        In fact, they wrote "reviewing […] other due diligence tasks", which doesn't imply any coverage! This close, literal reading is an appropriate – nay, the only appropriate – way to draw conclusions about the degree of responsibility exhibited by the custodians of a living standard. By corollary, any criticism of this form could be rebuffed by appeal to a sufficiently-carefully-written press release.
        LoganDark 9mo ago
        It implies potential coverage of anything one could bring up. It creates a similar impression in my mind, because it becomes easy to claim you already considered something.

HackerThemAll 9mo ago

I think this was the main reason (from the linked article) LOL: "Brotli is a compression algorithm developed by Google." They have no idea about Zstandard nor ANS/FSE comparing it with LZ77. Sheer incompetence.

mort96 9mo ago
EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 https://news.ycombinator.com/item?id=46723158 I just took all PDFs I had in my downloads folder (55, totaling 47M). These are invoices, data sheets, employment contracts, schematics, research reports, a bunch of random stuff really. I compressed them all with 'zstd --ultra -22', 'brotli -9', 'xz -9' and 'gzip -9'. Here are the results: +------+------+-----+------+--------+ | none | zstd | xz | gzip | brotli | +------|------|-----|------|--------| | 47M | 45M | 39M | 38M | 37M | +------+------+-----+------+--------+ Here's a table with all the files: +------+------+------+------+--------+ | raw | zstd | xz | gzip | brotli | +------+------+------+------+--------+ | 12K | 12K | 12K | 12K | 12K | | 20K | 20K | 20K | 20K | 20K | x5 | 24K | 20K | 20K | 20K | 20K | x5 | 28K | 24K | 24K | 24K | 24K | | 28K | 24K | 24K | 24K | 24K | | 32K | 20K | 20K | 20K | 20K | x3 | 32K | 24K | 24K | 24K | 24K | | 40K | 32K | 32K | 32K | 32K | | 44K | 40K | 40K | 40K | 40K | | 44K | 40K | 40K | 40K | 40K | | 48K | 36K | 36K | 36K | 36K | | 48K | 48K | 48K | 48K | 48K | | 76K | 128K | 72K | 72K | 72K | | 84K | 140K | 84K | 80K | 80K | x7 | 88K | 136K | 76K | 76K | 76K | | 124K | 152K | 88K | 92K | 92K | | 124K | 152K | 92K | 96K | 92K | | 140K | 160K | 100K | 100K | 100K | | 152K | 188K | 128K | 128K | 132K | | 188K | 192K | 184K | 184K | 184K | | 264K | 256K | 240K | 244K | 240K | | 320K | 256K | 228K | 232K | 228K | | 440K | 448K | 408K | 408K | 408K | | 448K | 448K | 432K | 432K | 432K | | 516K | 384K | 376K | 384K | 376K | | 992K | 320K | 260K | 296K | 280K | | 1.0M | 2.0M | 1.0M | 1.0M | 1.0M | | 1.1M | 192K | 192K | 228K | 200K | | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M | | 1.2M | 1.1M | 1.0M | 1.0M | 1.0M | | 1.3M | 2.0M | 1.1M | 1.1M | 1.1M | | 1.7M | 2.0M | 1.7M | 1.7M | 1.7M | | 1.9M | 960K | 896K | 952K | 916K | | 2.9M | 2.0M | 1.3M | 1.4M | 1.4M | | 3.2M | 4.0M | 3.1M | 3.1M | 3.0M | | 3.7M | 4.0M | 3.5M | 3.5M | 3.5M | | 6.4M | 4.0M | 4.1M | 3.7M | 3.5M | | 6.4M | 6.0M | 6.1M | 5.8M | 5.7M | | 9.7M | 10M | 10M | 9.5M | 9.4M | +------+------+------+------+--------+ Zstd is surprisingly bad on this data set. I'm guessing it struggles with the already-compressed image data in some of these PDFs. Going by only compression ratio, brotli is clearly better than the rest here and zstd is the worst. You'd have to find some other reason (maybe decompression speed, maybe spec complexity, or maybe you just trust Facebook more than Google) to choose zstd over brotli, going by my results. I wish I could share the data set for reproducibility, but I obviously can't just share every PDF I happened to have laying around in my downloads folder :p