4 ms·

The plot of compression ratio against speed for the various compression levels is pretty helpful for understanding its performance: https://scontent.fsnc1-3.fna

by cbr 10y ago

The plot of compression ratio against speed for the various compression levels is pretty helpful for understanding its performance: https://scontent.fsnc1-3.fna.fbcdn.net/t39.2365-6/14146892_944159239044397_638267599_n.jpg https://scontent.fsnc1-3.fna.fbcdn.net/t39.2365-6/14146892_9... "The x-axis is a decreasing logarithmic scale in megabytes per second; the y-axis is the compression ratio achieved." I'd love to see a version of this chart that also included Brotli. (And I'm somewhat surprised Brotli isn't mentioned at all.) (Disclaimer: I work at Google, which made Brotli)

justinsaccount 10y ago
The README at https://github.com/facebook/zstd https://github.com/facebook/zstd mentions brotli
jithesh 10y ago
Brotli is available in this graph - https://github.com/facebook/zstd/blob/master/images/DCspeed5.png https://github.com/facebook/zstd/blob/master/images/DCspeed5... This seems to be slightly old doc - from April.
bryanlarsen 10y ago
I thought that brotli was tuned for typical web workloads, that it contained a dictionary tuned for web workloads. Our internal testing shows that it performs very poorly for binary 3D vector data. So a test between zstd and brotli would show brotli in a poor light if it used a mixed corpus, but a test between zstd and brotli on a web corpus would give an advantage to brotli...
- niftich 10y ago
  'tuned' for the brotli dictionary is a bit generous [1], but yes, it contains a smattering of strings [2] you'd find in plain text and web documents. Because of its seemingly haphazard dictionary, I wouldn't rule out zstd outperforming brotli, if trained on a good dataset. [1] https://news.ycombinator.com/item?id=12010313 https://news.ycombinator.com/item?id=12010313 [2] https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e
- duskwuff 10y ago
  Not just tuned for web workloads in general, but for specific web workloads. The Brotli dictionary is mostly composed of English words and phrases, and fragments of HTML, CSS, and Javascript. It would perform poorly on non-English text. I have a feeling that the dictionary was designed with the specific goal of performing well on a specific corpus similar to the Large Text Compression Benchmark[1]. It has quite a few words and phrases that I'd associate with Wikipedia's "house style". [1]: https://cs.fit.edu/~mmahoney/compression/textdata.html https://cs.fit.edu/~mmahoney/compression/textdata.html
  - legulere 10y ago
    The dictionary also contains lots of Chinese, Russian and Arabic
    - duskwuff 10y ago
      By the numbers: - 9216 phrases total - 5857 (63.5%) pure ASCII phrases, mostly English with a few Spanish words thrown in - 1372 (14.8%) code fragments -- mostly HTML, CSS, and Javascript - 1027 (11.1%) CJK (Chinese, Japanese, and Korean, and mostly the first two) phrases -- it's very hard to tell Chinese and Japanese apart in this context; I didn't try - 158 (1.7%) phrases containing extended Latin-1 characters (nearly all Spanish words) - 303 (3.3%) Cyrillic script (probably Russian) phrases - 322 (3.5%) Arabic phrases - 172 (1.9%) Devanagari script (Hindi) phrases Plus a few miscellaneous other scripts and generally unclassifiable content.
- gribbly 10y ago
  >Our internal testing shows that it performs very poorly for binary 3D vector data. Uh ? What format of data was this ? I did a pretty large test (2gb+) on OBJ/STL 3d data and brotli compressed within ~5% margin of lzma, and this holds true on other binary data I've compared. It also compressed better than zstd (as in compression ratio) on the same data at their highest respective compression settings: bro -quality 11 -window 24 zstd --ultra -22 So I find it baffling that you find brotli to be poor for any binary data, could you share the data in question ?
  - haughter 10y ago
    What is the compression time for brotli to achieve compression ratios comparable to LZMA on these types of data sets: https://github.com/google/brotli/issues/165 https://github.com/google/brotli/issues/165 I understand that brotli is incredibly slow as compared to LZMA when using these high ratio settings (q11, w24), so slow as to be impractical in production even in a write-once, read-many scenario if you have any non-trivial amount of data being regularly produced. I do not want to have a farm of machines just to handle the brotli compression load of our data sets just because it is 10x slower than LZMA.
    - gribbly 10y ago
      >What is the compression time for brotli to achieve compression ratios comparable to LZMA... Compression time is indeed the achilles heel of Brotli, which is why it's something I would only use for compress once (and preferably decompress very often) scenarios. Compared to lzma at it's best compression setting for this particular data (-mx9 -m0=LZMA:d512m:fb273:lc8), brotli took 6 minutes and 4 seconds to compress, while the same data took 1 minute and 47 seconds for lzma. On the other hand, brotli decompressed the same data in 0.6 seconds, while it took lzma 2.2 seconds.
  - deleted 10y ago
    [deleted]
cmurphycode 10y ago
There's a nice comparison of compression algorithms (including zlib, zstd, brotli, snappy, etc) here: https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ It's nice because it uses many datasets and machine platforms. Unfortunately the graphs leave a bit to be desired, especially when you're trying to compare two algorithms. They provide the raw data, but it needs a little munging to make it workable. For my day job I made the following graph to compare zlib level1 and zstd. This was using the "peltast" platform which is a Xeon based system, because that was most relevant for us. http://imgur.com/a/0a3kK http://imgur.com/a/0a3kK I'll make one with brotli now.
- cmurphycode 10y ago
  Imgur album now compares brotli and zstd. Looks like zstd is always significantly faster, but Brotli compresses slightly better for every dataset except the XML one. This is with brotli level 1, by the way. My understanding is that brotli is pretty quick through the first few levels, but the levels that ask for the highest compression are insanely slow (which is a valuable thing to have as an option, for things like game assets or something, which are compressed once and delivered many times!)
  - wolf550e 10y ago
    For "compress once, decompress many times" workloads like game assets, you might want to check out Oodle, a proprietary library that kills zstd in decompression speed while achieving better compression ratios. http://cbloomrants.blogspot.com/2016/07/introducing-oodle-mermaid-and-selkie.html http://cbloomrants.blogspot.com/2016/07/introducing-oodle-me...
DougBTX 10y ago
Slide 4 here has one: https://dl.dropboxusercontent.com/u/12405967/ANSsem.pdf https://dl.dropboxusercontent.com/u/12405967/ANSsem.pdf
legulere 10y ago
Would be especially interesting to have zstd work with brotli's dictionary for the comparison.

by cbr 10y ago

justinsaccount 10y ago

The README at https://github.com/facebook/zstd https://github.com/facebook/zstd mentions brotli

jithesh 10y ago

Brotli is available in this graph - https://github.com/facebook/zstd/blob/master/images/DCspeed5.png https://github.com/facebook/zstd/blob/master/images/DCspeed5... This seems to be slightly old doc - from April.

bryanlarsen 10y ago

I thought that brotli was tuned for typical web workloads, that it contained a dictionary tuned for web workloads. Our internal testing shows that it performs very poorly for binary 3D vector data. So a test between zstd and brotli would show brotli in a poor light if it used a mixed corpus, but a test between zstd and brotli on a web corpus would give an advantage to brotli...

niftich 10y ago
'tuned' for the brotli dictionary is a bit generous [1], but yes, it contains a smattering of strings [2] you'd find in plain text and web documents. Because of its seemingly haphazard dictionary, I wouldn't rule out zstd outperforming brotli, if trained on a good dataset. [1] https://news.ycombinator.com/item?id=12010313 https://news.ycombinator.com/item?id=12010313 [2] https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e
duskwuff 10y ago
Not just tuned for web workloads in general, but for specific web workloads. The Brotli dictionary is mostly composed of English words and phrases, and fragments of HTML, CSS, and Javascript. It would perform poorly on non-English text. I have a feeling that the dictionary was designed with the specific goal of performing well on a specific corpus similar to the Large Text Compression Benchmark[1]. It has quite a few words and phrases that I'd associate with Wikipedia's "house style". [1]: https://cs.fit.edu/~mmahoney/compression/textdata.html https://cs.fit.edu/~mmahoney/compression/textdata.html
- legulere 10y ago
  The dictionary also contains lots of Chinese, Russian and Arabic
  - duskwuff 10y ago
    By the numbers: - 9216 phrases total - 5857 (63.5%) pure ASCII phrases, mostly English with a few Spanish words thrown in - 1372 (14.8%) code fragments -- mostly HTML, CSS, and Javascript - 1027 (11.1%) CJK (Chinese, Japanese, and Korean, and mostly the first two) phrases -- it's very hard to tell Chinese and Japanese apart in this context; I didn't try - 158 (1.7%) phrases containing extended Latin-1 characters (nearly all Spanish words) - 303 (3.3%) Cyrillic script (probably Russian) phrases - 322 (3.5%) Arabic phrases - 172 (1.9%) Devanagari script (Hindi) phrases Plus a few miscellaneous other scripts and generally unclassifiable content.
gribbly 10y ago
>Our internal testing shows that it performs very poorly for binary 3D vector data. Uh ? What format of data was this ? I did a pretty large test (2gb+) on OBJ/STL 3d data and brotli compressed within ~5% margin of lzma, and this holds true on other binary data I've compared. It also compressed better than zstd (as in compression ratio) on the same data at their highest respective compression settings: bro -quality 11 -window 24 zstd --ultra -22 So I find it baffling that you find brotli to be poor for any binary data, could you share the data in question ?
- haughter 10y ago
  What is the compression time for brotli to achieve compression ratios comparable to LZMA on these types of data sets: https://github.com/google/brotli/issues/165 https://github.com/google/brotli/issues/165 I understand that brotli is incredibly slow as compared to LZMA when using these high ratio settings (q11, w24), so slow as to be impractical in production even in a write-once, read-many scenario if you have any non-trivial amount of data being regularly produced. I do not want to have a farm of machines just to handle the brotli compression load of our data sets just because it is 10x slower than LZMA.
  - gribbly 10y ago
    >What is the compression time for brotli to achieve compression ratios comparable to LZMA... Compression time is indeed the achilles heel of Brotli, which is why it's something I would only use for compress once (and preferably decompress very often) scenarios. Compared to lzma at it's best compression setting for this particular data (-mx9 -m0=LZMA:d512m:fb273:lc8), brotli took 6 minutes and 4 seconds to compress, while the same data took 1 minute and 47 seconds for lzma. On the other hand, brotli decompressed the same data in 0.6 seconds, while it took lzma 2.2 seconds.
- deleted 10y ago
  [deleted]

cmurphycode 10y ago

There's a nice comparison of compression algorithms (including zlib, zstd, brotli, snappy, etc) here: https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ It's nice because it uses many datasets and machine platforms. Unfortunately the graphs leave a bit to be desired, especially when you're trying to compare two algorithms. They provide the raw data, but it needs a little munging to make it workable. For my day job I made the following graph to compare zlib level1 and zstd. This was using the "peltast" platform which is a Xeon based system, because that was most relevant for us. http://imgur.com/a/0a3kK http://imgur.com/a/0a3kK I'll make one with brotli now.

cmurphycode 10y ago
Imgur album now compares brotli and zstd. Looks like zstd is always significantly faster, but Brotli compresses slightly better for every dataset except the XML one. This is with brotli level 1, by the way. My understanding is that brotli is pretty quick through the first few levels, but the levels that ask for the highest compression are insanely slow (which is a valuable thing to have as an option, for things like game assets or something, which are compressed once and delivered many times!)
- wolf550e 10y ago
  For "compress once, decompress many times" workloads like game assets, you might want to check out Oodle, a proprietary library that kills zstd in decompression speed while achieving better compression ratios. http://cbloomrants.blogspot.com/2016/07/introducing-oodle-mermaid-and-selkie.html http://cbloomrants.blogspot.com/2016/07/introducing-oodle-me...

DougBTX 10y ago

Slide 4 here has one: https://dl.dropboxusercontent.com/u/12405967/ANSsem.pdf https://dl.dropboxusercontent.com/u/12405967/ANSsem.pdf

legulere 10y ago

Would be especially interesting to have zstd work with brotli's dictionary for the comparison.