4 ms·

Actually brotli compresses some more, often around 5 % more density. If you compress to same density, brotli tends to be significantly faster. Zstd decompresses

Actually brotli compresses some more, often around 5 % more density. If you compress to same density, brotli tends to be significantly faster. Zstd decompresses faster. Brotli is naturally limited to 16 MB of memory use whereas ZStd makes less attempts to reduce memory use. If you want to use more memory with Brotli, you need to use another flag '--large_window' instead of '--window', to indicate that you really want it, whereas zstd uses a lot of memory more silently. When you compare compression levels, you should compare with same memory consumption -- and if you do that, brotli tends to win by 5 %. When in doubt in benchmarking, run brotli with --large_window 30 https://encode.ru/threads/2947-large-window-brotli-results-available-for-a-several-corpora https://encode.ru/threads/2947-large-window-brotli-results-a... is a summary of a recent large-scale experiment of a large variety of compressors, including brotli, zstd and LZMA.

twitch_checksum 8y ago
> ZStd makes less attempts to reduce memory use Ever heard of flag `--ultra` ? Without it, zstd memory usage will be limited to 8 MB. It has been there since, forever.
- JyrkiAlakuijala 8y ago
  -22 doesn't imply 128 MB window silently?
  - terrelln 8y ago
    Levels above 19, which has a 8 MB window size, may only be used with the --ultra flag, whose documentation says "requires more memory".
    - JyrkiAlakuijala 8y ago
      cool, just that most zstd benchmarking against other algos that I ever saw was with -22, i.e., disregarding memory use and multi-processing performance would be wonderful to see benchmarking with real data and with realistic (and same from one algorithm to the next) window sizes
      - terrelln 8y ago
        Looking at the brotli API, I believe I spotted the source of our disagreements. Brotli separates out the window size from the quality, but in zstd the level implies the window size, unless you use advanced parameters. For example, level 3 has a 1 MB window size. When you are comparing brotli and zstd, are you focusing on the highest quality/compression level, but varying the window size? For us, lower compression levels (1-3) are our primary focus. We care about the highest levels for benchmarks, but it isn't our focus. When we are benchmarking zstd we focus on the lower levels, since they are the most important to us. When we think about a smaller window size, we generally think about faster compression. When we comparing against brotli, the question we're asking is, how does brotli compare to level 3. When you're benchmarking zstd, I suspect you're asking the question, how does the highest zstd level compare to the highest brotli quality for the same window size, is that right?
        JyrkiAlakuijala 8y ago
        In my thinking window size is something that is optimized for the decoding resources -- whereas the encoding effort is optimized for how expensive the transfer is in relation to encoding cost. For example, a mobile client might prefer to have no more than 512 kB sliding window due to design of its memory system or for multi-processing reasons. There, a static resource can still be encoded to the smallest size with extremely slow encoding (quality 9-11) -- but for one-use we'd still prefer a faster encoding (quality 6 or so). Thank you for explaining your focus. Did you arrive to that by doing economic calculations or was it to have minimal changes to situation with no compression?
        terrelln 8y ago
        A lot of compression ends up being compress once, decompress one to a few times, so the faster end of the spectrum fits well. Additionally, compression has to fit into the existing system, and existing systems already have tight constraints, so it is an easier sell to say "hey, we can compress faster AND stronger" than what you have right now. For our larger services, we will tune the compression level more carefully, and end up at different places for different services, but still normally around the faster levels. That said, we still see users of the stronger compression levels.

twitch_checksum 8y ago

> ZStd makes less attempts to reduce memory use Ever heard of flag `--ultra` ? Without it, zstd memory usage will be limited to 8 MB. It has been there since, forever.

JyrkiAlakuijala 8y ago
-22 doesn't imply 128 MB window silently?
- terrelln 8y ago
  Levels above 19, which has a 8 MB window size, may only be used with the --ultra flag, whose documentation says "requires more memory".
  - JyrkiAlakuijala 8y ago
    cool, just that most zstd benchmarking against other algos that I ever saw was with -22, i.e., disregarding memory use and multi-processing performance would be wonderful to see benchmarking with real data and with realistic (and same from one algorithm to the next) window sizes
    - terrelln 8y ago
      Looking at the brotli API, I believe I spotted the source of our disagreements. Brotli separates out the window size from the quality, but in zstd the level implies the window size, unless you use advanced parameters. For example, level 3 has a 1 MB window size. When you are comparing brotli and zstd, are you focusing on the highest quality/compression level, but varying the window size? For us, lower compression levels (1-3) are our primary focus. We care about the highest levels for benchmarks, but it isn't our focus. When we are benchmarking zstd we focus on the lower levels, since they are the most important to us. When we think about a smaller window size, we generally think about faster compression. When we comparing against brotli, the question we're asking is, how does brotli compare to level 3. When you're benchmarking zstd, I suspect you're asking the question, how does the highest zstd level compare to the highest brotli quality for the same window size, is that right?
      - JyrkiAlakuijala 8y ago
        In my thinking window size is something that is optimized for the decoding resources -- whereas the encoding effort is optimized for how expensive the transfer is in relation to encoding cost. For example, a mobile client might prefer to have no more than 512 kB sliding window due to design of its memory system or for multi-processing reasons. There, a static resource can still be encoded to the smallest size with extremely slow encoding (quality 9-11) -- but for one-use we'd still prefer a faster encoding (quality 6 or so). Thank you for explaining your focus. Did you arrive to that by doing economic calculations or was it to have minimal changes to situation with no compression?
        terrelln 8y ago
        A lot of compression ends up being compress once, decompress one to a few times, so the faster end of the spectrum fits well. Additionally, compression has to fit into the existing system, and existing systems already have tight constraints, so it is an easier sell to say "hey, we can compress faster AND stronger" than what you have right now. For our larger services, we will tune the compression level more carefully, and end up at different places for different services, but still normally around the faster levels. That said, we still see users of the stronger compression levels.