5 ms·

You need to crank up zstd compression level. zstd is Pareto better than brotli - compresses better and faster

jeffbee 9mo ago
Are you sure? Admittedly I only have 1 PDF in my homedir, but no combination of flags to zstd gets it to match the size of brotli's output on that particular file. Even zstd --long --ultra -22.
- xxs 9mo ago
  on max compression (11 vs zstd's 22) of text brotli will be around 3-4% denser... and a lot slower. Decompression wise zstd is over 2x faster. The pdfs you have are already compressed with deflate (zip).
atiedebee 9mo ago
I thought the same, so I ran brotli and zstd on some PDFs I had laying around. brotli 1.0.7 args: -q 11 -w 24 zstd v1.5.0 args: --ultra -22 --long=31 | Original | zstd | brotli RandomBook.pdf | 15M | 4.6M | 4.5M Invoice.pdf | 19.3K | 16.3K | 16.1K I made a table because I wanted to test more files, but almost all PDFs I downloaded/had stored locally were already compressed and I couldn't quickly find a way to decompress them. Brotli seemed to have a very slight edge over zstd, even on the larger pdf, which I did not expect.
- order-matters 9mo ago
  Whats the assumption we can potentially target as reason for the counter-intuitive result? that data in pdf files are noisy and zstd should perform better on noisy files?
  - jeffbee 9mo ago
    What's counter-intuitive about this outcome?
    - order-matters 9mo ago
      maybe that was too strongly worded but there was an expectation for zstd to outperform. So the fact it didnt means the result was unexpected. i generally find it helpful to understand why something performs better than expected.
      - mort96 9mo ago
        Isn't zstd primarily designed to provide decent compression ratios at amazing speeds? The reason it's exciting is mainly that you can add compression to places where it didn't necessarily make sense before because it's almost free in terms of CPU and memory consumption. I don't think it has ever had a stated goal of beating compression ratio focused algorithms like brotli on compression ratio.
        sgerenser 9mo ago
        I actually thought zstd was supposed to be better than Brotli in most cases, but a bit of searching reveals you're right... Brotli, especially at the highest compression levels (10/11), often exceeds zstd at the highest compression levels (20-22). Both are very slow at those levels, although perfectly suitable for "compress once, decompress many" applications which the PDF spec is obviously one of them.
- mort96 9mo ago
  EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 https://news.ycombinator.com/item?id=46723158 I did my own testing where Brotli also ended up better than ZSTD: https://news.ycombinator.com/item?id=46722044 https://news.ycombinator.com/item?id=46722044 Results by compression type across 55 PDFs: +------+------+-----+------+--------+ | none | zstd | xz | gzip | brotli | +------|------|-----|------|--------| | 47M | 45M | 39M | 38M | 37M | +------+------+-----+------+--------+
  - mort96 9mo ago
    Turns out that these numbers are caused by APFS weirdness. I used 'du' to get them which reports the size on disk, which is weirdly bloated for some reason when compressing in parallel. I should've used 'du -A', which reports the apparent size. Here's a table with the correct sizes, reported by 'du -A' (which shows the apparent size): +---------+---------+--------+--------+--------+ | none | zstd | xz | gzip | brotli | +---------|---------|--------|--------|--------| | 47.81M | 37.92M | 37.96M | 38.80M | 37.06M | +---------+---------+--------+--------+--------+ These numbers are much more impressive. Still, Brotli has a slight edge.
    - tracker1 9mo ago
      Worth considering the compress/decompress overhead, which is also lower in brotli than zstd from my understanding. Also, worth testing zopfli since it's decompression is gzip compatible.
- mrspuratic 9mo ago
  > I couldn't quickly find a way to decompress them pdftk in.pdf output out.pdf decompress
- Thoreandan 9mo ago
  Does your source .pdf material have FlateDecode'd chunks or did you fully uncompress it?
  - atiedebee 9mo ago
    I wasn't sure. I just went in with the (probably faulty) assumption that if it compresses to less than 90% of the original size that it had enough "non-randomness" to compare compression performance.
- atiedebee 9mo ago
  Ran the tests again with some more files, this time decompressing the pdf in advance. I picked some widely available PDFs to make the experiment reproducable. file | raw | zstd (%) | brotli (%) | gawk.pdf | 8.068.092 | 1.437.529 (17.8%) | 1.376.106 (17.1%) | shannon.pdf | 335.009 | 68.739 (20.5%) | 65.978 (19.6%) | attention.pdf | 24.742.418 | 367.367 (1.4%) | 362.578 (1.4%) | learnopengl.pdf | 253.041.425 | 37.756.229 (14.9%) | 35.223.532 (13.9%) | For learnopengl.pdf I also tested the decompression performance, since it is such a large file, and got the following (less surprising) results using 'perf stat -r 5': zstd: 0.4532 +- 0.0216 seconds time elapsed ( +- 4.77% ) brotli: 0.7641 +- 0.0242 seconds time elapsed ( +- 3.17% ) The conclusion seems to be consistent with what brotli's authors have said: brotli achieves slightly better compression, at the cost of a little over half the decompression speed.
dchest 9mo ago
Not with small files.
- Dylan16807 9mo ago
  If that's about using predefined dictionaries, zstd can use them too. If brotli has a different advantage on small source files, you have my curiosity. If you're talking about max compression, zstd likely loses out there, the answer seems to vary based on the tests I look at, but it seems to be better across a very wide range.
  - dchest 9mo ago
    No, it's literally just compressing small files without training zstd dict or plugging external dictionaries (not counting the built-in one that brotli has). Especially for English text, brotli at the same speed as zstd gives better results for small data (in kilobyte to a few of megabyte range).
DetroitThrow 9mo ago
I love zstd but this isn't necessarily true.
itsdesmond 9mo ago
> Pareto I don’t think you’re using that correctly.
- wizzwizz4 9mo ago
  It's correct use of Pareto, short for Pareto frontier, if the claim being made is "for every needed compression ratio, zstd is faster; and for every needed time budget, zstd is faster". (Whether this claim is true is another matter.)

Are you sure? Admittedly I only have 1 PDF in my homedir, but no combination of flags to zstd gets it to match the size of brotli's output on that particular file. Even zstd --long --ultra -22.

xxs 9mo ago
on max compression (11 vs zstd's 22) of text brotli will be around 3-4% denser... and a lot slower. Decompression wise zstd is over 2x faster. The pdfs you have are already compressed with deflate (zip).

atiedebee 9mo ago

I thought the same, so I ran brotli and zstd on some PDFs I had laying around. brotli 1.0.7 args: -q 11 -w 24 zstd v1.5.0 args: --ultra -22 --long=31 | Original | zstd | brotli RandomBook.pdf | 15M | 4.6M | 4.5M Invoice.pdf | 19.3K | 16.3K | 16.1K I made a table because I wanted to test more files, but almost all PDFs I downloaded/had stored locally were already compressed and I couldn't quickly find a way to decompress them. Brotli seemed to have a very slight edge over zstd, even on the larger pdf, which I did not expect.

order-matters 9mo ago
Whats the assumption we can potentially target as reason for the counter-intuitive result? that data in pdf files are noisy and zstd should perform better on noisy files?
- jeffbee 9mo ago
  What's counter-intuitive about this outcome?
  - order-matters 9mo ago
    maybe that was too strongly worded but there was an expectation for zstd to outperform. So the fact it didnt means the result was unexpected. i generally find it helpful to understand why something performs better than expected.
    - mort96 9mo ago
      Isn't zstd primarily designed to provide decent compression ratios at amazing speeds? The reason it's exciting is mainly that you can add compression to places where it didn't necessarily make sense before because it's almost free in terms of CPU and memory consumption. I don't think it has ever had a stated goal of beating compression ratio focused algorithms like brotli on compression ratio.
      - sgerenser 9mo ago
        I actually thought zstd was supposed to be better than Brotli in most cases, but a bit of searching reveals you're right... Brotli, especially at the highest compression levels (10/11), often exceeds zstd at the highest compression levels (20-22). Both are very slow at those levels, although perfectly suitable for "compress once, decompress many" applications which the PDF spec is obviously one of them.
mort96 9mo ago
EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 https://news.ycombinator.com/item?id=46723158 I did my own testing where Brotli also ended up better than ZSTD: https://news.ycombinator.com/item?id=46722044 https://news.ycombinator.com/item?id=46722044 Results by compression type across 55 PDFs: +------+------+-----+------+--------+ | none | zstd | xz | gzip | brotli | +------|------|-----|------|--------| | 47M | 45M | 39M | 38M | 37M | +------+------+-----+------+--------+
- mort96 9mo ago
  Turns out that these numbers are caused by APFS weirdness. I used 'du' to get them which reports the size on disk, which is weirdly bloated for some reason when compressing in parallel. I should've used 'du -A', which reports the apparent size. Here's a table with the correct sizes, reported by 'du -A' (which shows the apparent size): +---------+---------+--------+--------+--------+ | none | zstd | xz | gzip | brotli | +---------|---------|--------|--------|--------| | 47.81M | 37.92M | 37.96M | 38.80M | 37.06M | +---------+---------+--------+--------+--------+ These numbers are much more impressive. Still, Brotli has a slight edge.
  - tracker1 9mo ago
    Worth considering the compress/decompress overhead, which is also lower in brotli than zstd from my understanding. Also, worth testing zopfli since it's decompression is gzip compatible.
mrspuratic 9mo ago
> I couldn't quickly find a way to decompress them pdftk in.pdf output out.pdf decompress
Thoreandan 9mo ago
Does your source .pdf material have FlateDecode'd chunks or did you fully uncompress it?
- atiedebee 9mo ago
  I wasn't sure. I just went in with the (probably faulty) assumption that if it compresses to less than 90% of the original size that it had enough "non-randomness" to compare compression performance.
atiedebee 9mo ago
Ran the tests again with some more files, this time decompressing the pdf in advance. I picked some widely available PDFs to make the experiment reproducable. file | raw | zstd (%) | brotli (%) | gawk.pdf | 8.068.092 | 1.437.529 (17.8%) | 1.376.106 (17.1%) | shannon.pdf | 335.009 | 68.739 (20.5%) | 65.978 (19.6%) | attention.pdf | 24.742.418 | 367.367 (1.4%) | 362.578 (1.4%) | learnopengl.pdf | 253.041.425 | 37.756.229 (14.9%) | 35.223.532 (13.9%) | For learnopengl.pdf I also tested the decompression performance, since it is such a large file, and got the following (less surprising) results using 'perf stat -r 5': zstd: 0.4532 +- 0.0216 seconds time elapsed ( +- 4.77% ) brotli: 0.7641 +- 0.0242 seconds time elapsed ( +- 3.17% ) The conclusion seems to be consistent with what brotli's authors have said: brotli achieves slightly better compression, at the cost of a little over half the decompression speed.

dchest 9mo ago

Not with small files.

Dylan16807 9mo ago
If that's about using predefined dictionaries, zstd can use them too. If brotli has a different advantage on small source files, you have my curiosity. If you're talking about max compression, zstd likely loses out there, the answer seems to vary based on the tests I look at, but it seems to be better across a very wide range.
- dchest 9mo ago
  No, it's literally just compressing small files without training zstd dict or plugging external dictionaries (not counting the built-in one that brotli has). Especially for English text, brotli at the same speed as zstd gives better results for small data (in kilobyte to a few of megabyte range).

DetroitThrow 9mo ago

I love zstd but this isn't necessarily true.

itsdesmond 9mo ago

> Pareto I don’t think you’re using that correctly.

wizzwizz4 9mo ago
It's correct use of Pareto, short for Pareto frontier, if the claim being made is "for every needed compression ratio, zstd is faster; and for every needed time budget, zstd is faster". (Whether this claim is true is another matter.)