5 ms·

Some real cognitive dissonance in this article… “The PDF Association operates under a strict principle—any new feature must work seamlessly with existing reade

Some real cognitive dissonance in this article… “The PDF Association operates under a strict principle—any new feature must work seamlessly with existing readers” followed by introducing compression as a breaking change in the same paragraph. All this for brotli… on a read-many format like pdf zstd’s decompression speed is a much better fit.

xxs 9mo ago
yup, zstd is better. Overall use zstd for pretty much anything that can benefit from a general purpose compression. It's a beyond excellent library, tool, and an algorithm (set of). Brotli w/o a custom dictionary is a weird choice to begin with.
- greenavocado 9mo ago
  This bizzare move has all the hallmarks of embrace-extend-extinguish rather than technical excellence
- adzm 9mo ago
  Brotli makes a bit of sense considering this is a static asset; it compresses somewhat more than zstd. This is why brotli is pretty ubiquitous for precompressed static assets on the Web. That said, I personally prefer zstd as well, it's been a great general use lib.
  - dist-epoch 9mo ago
    You need to crank up zstd compression level. zstd is Pareto better than brotli - compresses better and faster
    - jeffbee 9mo ago
      Are you sure? Admittedly I only have 1 PDF in my homedir, but no combination of flags to zstd gets it to match the size of brotli's output on that particular file. Even zstd --long --ultra -22.
      - xxs 9mo ago
        on max compression (11 vs zstd's 22) of text brotli will be around 3-4% denser... and a lot slower. Decompression wise zstd is over 2x faster. The pdfs you have are already compressed with deflate (zip).
    - atiedebee 9mo ago
      I thought the same, so I ran brotli and zstd on some PDFs I had laying around. brotli 1.0.7 args: -q 11 -w 24 zstd v1.5.0 args: --ultra -22 --long=31 | Original | zstd | brotli RandomBook.pdf | 15M | 4.6M | 4.5M Invoice.pdf | 19.3K | 16.3K | 16.1K I made a table because I wanted to test more files, but almost all PDFs I downloaded/had stored locally were already compressed and I couldn't quickly find a way to decompress them. Brotli seemed to have a very slight edge over zstd, even on the larger pdf, which I did not expect.
      - order-matters 9mo ago
        Whats the assumption we can potentially target as reason for the counter-intuitive result? that data in pdf files are noisy and zstd should perform better on noisy files?
        jeffbee 9mo ago
        What's counter-intuitive about this outcome?
        order-matters 9mo ago
        maybe that was too strongly worded but there was an expectation for zstd to outperform. So the fact it didnt means the result was unexpected. i generally find it helpful to understand why something performs better than expected.
        mort96 9mo ago
        Isn't zstd primarily designed to provide decent compression ratios at amazing speeds? The reason it's exciting is mainly that you can add compression to places where it didn't necessarily make sense before because it's almost free in terms of CPU and memory consumption. I don't think it has ever had a stated goal of beating compression ratio focused algorithms like brotli on compression ratio.
    - dchest 9mo ago
      Not with small files.
      - Dylan16807 9mo ago
        If that's about using predefined dictionaries, zstd can use them too. If brotli has a different advantage on small source files, you have my curiosity. If you're talking about max compression, zstd likely loses out there, the answer seems to vary based on the tests I look at, but it seems to be better across a very wide range.
        dchest 9mo ago
        No, it's literally just compressing small files without training zstd dict or plugging external dictionaries (not counting the built-in one that brotli has). Especially for English text, brotli at the same speed as zstd gives better results for small data (in kilobyte to a few of megabyte range).
    - DetroitThrow 9mo ago
      I love zstd but this isn't necessarily true.
    - itsdesmond 9mo ago
      > Pareto I don’t think you’re using that correctly.
      - wizzwizz4 9mo ago
        It's correct use of Pareto, short for Pareto frontier, if the claim being made is "for every needed compression ratio, zstd is faster; and for every needed time budget, zstd is faster". (Whether this claim is true is another matter.)
  - stonogo 9mo ago
    brotli is ubiquitous because Google recommends it. While Deflate definitely sucks and is old, Google ships brotli in Chrome, and since Chrome is the de facto default platform nowadays, I'd imagine it was chosen because it was the lowest-effort lift. Nevertheless, I expect this to be JBIG2 all over again: almost nobody will use this because we've got decades of devices and software in the wild that can't, and 20% filesize savings is pointless if your destination can't read the damn thing.
- deepsun 9mo ago
  Brotli compresses my files way better, but it's doing it way slower. Anyway, universal statement "zstd is better" is not valid.
  - xxs 9mo ago
    On max compression "--ultra -22", zstd is likely to be 2-4% less dense (larger) on text alike input. While taking over 2x times times to compress. Decompression is also much faster, usually over 2x. I have not tried using a dictionary for zstd.
deepsun 9mo ago
Well, except for speed, compression algorithms need to be compared in terms of compression, you know. Here's discussion by brotli's and zstd's staff: https://news.ycombinator.com/item?id=19678985 https://news.ycombinator.com/item?id=19678985
mmooss 9mo ago
Note the language: "You're not creating broken files—you're creating files that are ahead of their time." Imagine a sales meeting where someone pitched that to you. They have to be joking, right? I have no objection to adding Brotli, but I hope they take the compatability more seriously. You may need readers to deploy it for a long time - ten years? - before you deploy it in PDF creation tools.
- nxobject 9mo ago
  (sarcasm warning...) You're absolutely right! It's not just an inaccurate slogan—it's a patronizing use of artificial intelligence. What you're describing is not just true, it's precise.
  - mmooss 9mo ago
    I don't understand your point ...
    - eventualcomp 9mo ago
      The commenter is making a joke about the style of delivery of the sentence you quoted, because the style is [1]characteristic of AI generated writing. [1]https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
spider-mario 9mo ago
> on a read-many format like pdf zstd’s decompression speed is a much better fit. brotli decompression is already plenty fast. For PDFs, zstd’s advantage in decompression speed is academic.

xxs 9mo ago

yup, zstd is better. Overall use zstd for pretty much anything that can benefit from a general purpose compression. It's a beyond excellent library, tool, and an algorithm (set of). Brotli w/o a custom dictionary is a weird choice to begin with.

greenavocado 9mo ago
This bizzare move has all the hallmarks of embrace-extend-extinguish rather than technical excellence
adzm 9mo ago
Brotli makes a bit of sense considering this is a static asset; it compresses somewhat more than zstd. This is why brotli is pretty ubiquitous for precompressed static assets on the Web. That said, I personally prefer zstd as well, it's been a great general use lib.
- dist-epoch 9mo ago
  You need to crank up zstd compression level. zstd is Pareto better than brotli - compresses better and faster
  - jeffbee 9mo ago
    Are you sure? Admittedly I only have 1 PDF in my homedir, but no combination of flags to zstd gets it to match the size of brotli's output on that particular file. Even zstd --long --ultra -22.
    - xxs 9mo ago
      on max compression (11 vs zstd's 22) of text brotli will be around 3-4% denser... and a lot slower. Decompression wise zstd is over 2x faster. The pdfs you have are already compressed with deflate (zip).
  - atiedebee 9mo ago
    I thought the same, so I ran brotli and zstd on some PDFs I had laying around. brotli 1.0.7 args: -q 11 -w 24 zstd v1.5.0 args: --ultra -22 --long=31 | Original | zstd | brotli RandomBook.pdf | 15M | 4.6M | 4.5M Invoice.pdf | 19.3K | 16.3K | 16.1K I made a table because I wanted to test more files, but almost all PDFs I downloaded/had stored locally were already compressed and I couldn't quickly find a way to decompress them. Brotli seemed to have a very slight edge over zstd, even on the larger pdf, which I did not expect.
    - order-matters 9mo ago
      Whats the assumption we can potentially target as reason for the counter-intuitive result? that data in pdf files are noisy and zstd should perform better on noisy files?
      - jeffbee 9mo ago
        What's counter-intuitive about this outcome?
        order-matters 9mo ago
        maybe that was too strongly worded but there was an expectation for zstd to outperform. So the fact it didnt means the result was unexpected. i generally find it helpful to understand why something performs better than expected.
        mort96 9mo ago
        Isn't zstd primarily designed to provide decent compression ratios at amazing speeds? The reason it's exciting is mainly that you can add compression to places where it didn't necessarily make sense before because it's almost free in terms of CPU and memory consumption. I don't think it has ever had a stated goal of beating compression ratio focused algorithms like brotli on compression ratio.
  - dchest 9mo ago
    Not with small files.
    - Dylan16807 9mo ago
      If that's about using predefined dictionaries, zstd can use them too. If brotli has a different advantage on small source files, you have my curiosity. If you're talking about max compression, zstd likely loses out there, the answer seems to vary based on the tests I look at, but it seems to be better across a very wide range.
      - dchest 9mo ago
        No, it's literally just compressing small files without training zstd dict or plugging external dictionaries (not counting the built-in one that brotli has). Especially for English text, brotli at the same speed as zstd gives better results for small data (in kilobyte to a few of megabyte range).
  - DetroitThrow 9mo ago
    I love zstd but this isn't necessarily true.
  - itsdesmond 9mo ago
    > Pareto I don’t think you’re using that correctly.
    - wizzwizz4 9mo ago
      It's correct use of Pareto, short for Pareto frontier, if the claim being made is "for every needed compression ratio, zstd is faster; and for every needed time budget, zstd is faster". (Whether this claim is true is another matter.)
- stonogo 9mo ago
  brotli is ubiquitous because Google recommends it. While Deflate definitely sucks and is old, Google ships brotli in Chrome, and since Chrome is the de facto default platform nowadays, I'd imagine it was chosen because it was the lowest-effort lift. Nevertheless, I expect this to be JBIG2 all over again: almost nobody will use this because we've got decades of devices and software in the wild that can't, and 20% filesize savings is pointless if your destination can't read the damn thing.
deepsun 9mo ago
Brotli compresses my files way better, but it's doing it way slower. Anyway, universal statement "zstd is better" is not valid.
- xxs 9mo ago
  On max compression "--ultra -22", zstd is likely to be 2-4% less dense (larger) on text alike input. While taking over 2x times times to compress. Decompression is also much faster, usually over 2x. I have not tried using a dictionary for zstd.

deepsun 9mo ago

Well, except for speed, compression algorithms need to be compared in terms of compression, you know. Here's discussion by brotli's and zstd's staff: https://news.ycombinator.com/item?id=19678985 https://news.ycombinator.com/item?id=19678985

mmooss 9mo ago

Note the language: "You're not creating broken files—you're creating files that are ahead of their time." Imagine a sales meeting where someone pitched that to you. They have to be joking, right? I have no objection to adding Brotli, but I hope they take the compatability more seriously. You may need readers to deploy it for a long time - ten years? - before you deploy it in PDF creation tools.

nxobject 9mo ago
(sarcasm warning...) You're absolutely right! It's not just an inaccurate slogan—it's a patronizing use of artificial intelligence. What you're describing is not just true, it's precise.
- mmooss 9mo ago
  I don't understand your point ...
  - eventualcomp 9mo ago
    The commenter is making a joke about the style of delivery of the sentence you quoted, because the style is [1]characteristic of AI generated writing. [1]https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing

spider-mario 9mo ago

> on a read-many format like pdf zstd’s decompression speed is a much better fit. brotli decompression is already plenty fast. For PDFs, zstd’s advantage in decompression speed is academic.