9 ms·

CDN77 Now Supports Brotli

227 points by clarinois 11y ago

pmlnr 11y ago
Are there any memory & CPU consumption graphs? I also don't understand why not xz, though I know that requires significantly more resource than gzip.
- TheCondor 11y ago
  Brotli actually has levels of compression that are more dense than gzip and compress/decompress faster. There is also a spec, xz has a reference implementation but no spec.
  - nacs 11y ago
    > more dense than gzip and compress/decompress faster If you're saying brotli compresses/decompresses faster that gzip, it doesn't according to Cloudflare tests [1] (see table near bottom). Even the fastest compression level of Brotli is slower than the highest/slowest compression level of gzip in most cases. [1]: https://blog.cloudflare.com/results-experimenting-brotli/ https://blog.cloudflare.com/results-experimenting-brotli/
    - cornstalks 11y ago
      > If you're saying brotli compresses/decompresses faster that gzip, it doesn't according to Cloudflare tests. It doesn't always compress faster, but as far as I can tell they didn't measure or say anything about decompression.
    - Nelson69 11y ago
      I just did this: time bro --quality 6 --input linux-4.5.tar --output linux-4.5.tar.bro real 0m18.436s user 0m18.228s sys 0m0.184s time gzip -9 linux-4.5.tar real 0m30.555s user 0m30.424s sys 0m0.172s ls -lh (I removed the metadata): 106M linux-4.5.tar.bro 129M linux-4.5.tar.gz Only 82% of the size at 60% of the time. Now this is pure text so that's not a good example of everything.
- doomrobo 11y ago
  >XZ is a file format, LZMA is the compression algorithm. https://bugzilla.mozilla.org/show_bug.cgi?id=366559#c18 https://bugzilla.mozilla.org/show_bug.cgi?id=366559#c18
- sp332 11y ago
  If it takes a lot of CPU, it might not be worth the CPU time or cost. https://i.imgur.com/bAc1Saq.png https://i.imgur.com/bAc1Saq.png Image from https://dl.acm.org/citation.cfm?id=1084786&dl=ACM&coll=DL&CFID=763952065&CFTOKEN=92459762 https://dl.acm.org/citation.cfm?id=1084786&dl=ACM&coll=DL&CF... And gzip speed has improved since 2005, so over a 10 Mbps connection, you'd have to get much better compression to be worth the switch.
jakubstr 11y ago
Nice. Hope to see it in FF soon. And in Internet Explorer 87.
- KurvaKde1334 11y ago
  In 2020 maybe :)
- TheRealPomax 11y ago
  http://caniuse.com/#search=brotli http://caniuse.com/#search=brotli tells us FF already supports this, and IE has it marked as being under implementation consideration (which one would expect to turn into active development once they finish their current WOFF2 support work; WOFF2 has a hard dependency on a working Brotli implementation)
  - KurvaKde1334 11y ago
    Try nginx & FF45 -> doesnt work as FF uses deprecated (and now obsolete & removed) API
    - bzbarsky 11y ago
      Reference for this, please? Because I see no bugs filed on FF regarding brotli apart from https://bugzilla.mozilla.org/show_bug.cgi?id=1222541 https://bugzilla.mozilla.org/show_bug.cgi?id=1222541 and https://bugzilla.mozilla.org/show_bug.cgi?id=1207234 https://bugzilla.mozilla.org/show_bug.cgi?id=1207234 neither of which seems to match your problem description. I'm happy to get a bug filed for you so whatever problem this is can be fixed, but I need a bit more to go on here... I did look around for known brotli+Firefox+nginx issues, but the only one that comes up is https://bugzilla.mozilla.org/show_bug.cgi?id=1215724 https://bugzilla.mozilla.org/show_bug.cgi?id=1215724 which was fixed in shipping Firefox months ago, so I'm assuming that's not the one you're talking about.
      - KurvaKde1334 11y ago
        Sure, there is no official bug for this. The only way to see it is to check the dates when the new API was applied - its far too late after releasing versions already supporting brotli, but this commit was not backported anywhere so in a real life, its broken in FF. Just get an nginx, nginx brotli module from google and use FF 45. Good luck :) https://hg.mozilla.org/releases/mozilla-aurora/log/10e1774de80e/modules/brotli/dec/decode.h https://hg.mozilla.org/releases/mozilla-aurora/log/10e1774de...
jgrahamc 11y ago
We took a very detailed look at Brotli performance (both speed and compression) and concluded that for dynamic content it was only useful if the file size was greater than 64k on slow connections and there's a very detailed description here: https://blog.cloudflare.com/results-experimenting-brotli/ https://blog.cloudflare.com/results-experimenting-brotli/ If you want to experiment with it, it's enabled on CloudFlare's test server https://http2.cloudflare.com/ https://http2.cloudflare.com/. Since we don't charge by byte the focus is on end-user performance and not 'saving money' and we concluded that it wasn't currently worth implementing Brotli widely but will continue to experiment with it.
- KurvaKde1334 11y ago
  There's a lot of stuff greater then 64k nowadays ;)
  - mwpmaybe 11y ago
    Especially those pesky XML and JSON payloads!
  - clarinois 11y ago
    Exactly :) If Cloudflare decides it’s worth several man-days to offer this feature to their clients who deliver larger files than 64k, feel free to reach out to us, we’ll be happy to share our experience :) devs@cdn77.com
    - jgrahamc 11y ago
      Thanks. We actually wrote and open sourced our nginx modifications for choosing between Brotli and gzip. https://github.com/cloudflare/ngx_brotli_module https://github.com/cloudflare/ngx_brotli_module
    - deleted 11y ago
      [deleted]
  - acqq 11y ago
    From the Cloduflare's tests: "Most files are smaller than 64KB, and if we look only at those files then Brotli 4 is actually 1.48X slower than zlib level 8!" And the faster zlib levels (1-7) are even faster! See the table. I especially like zlib 1.
bhouston 11y ago
I find that Brotli is over sold for the case of compressing generic binary files. The claims that it is comparable to xz/lzma for generic binary data are not accurate. In my real-world tests of compressing 3D data it far underperformed xz/lzma although it was still better than gzip: https://github.com/google/brotli/issues/165 https://github.com/google/brotli/issues/165
- kevingadd 11y ago
  It's fairly competitive with LZHAM, at least, even if it's way slower to compress. You will get much better results out of Brotli if you restructure your data to be more compressible, and that will also improve your lzma and gzip (especially gzip) compression ratios, to a tremendous degree. Have you done any of this? If not, ping me, and I can explain some techniques to apply.
  - detaro 11y ago
    >You will get much better results out of Brotli if you restructure your data to be more compressible, and that will also improve your lzma and gzip (especially gzip) compression ratios, to a tremendous degree. This sounds interesting, I'd like to read some examples/links/explanations.
    - EdHominem 11y ago
      I imagine a lot of it is segregating your content. All strings in one file, etc. It'd be an interesting test to take our eight-language set of localization strings and compress them in UI order and language order and see if there's much of a difference. (UI order is all languages for one dialog element, then all eight for the next, etc. Language order is all the English first, then ...) I'd definitely like to hear Kevingadd's tips though.
sremani 11y ago
Is it true that Brotli wanted the .bro file extension and moved away from it, because it was deemed offensive?
- return0 11y ago
  Interesting. Is it also middle-out?
- Bud 11y ago
  Yes, that's confirmed. "In late September, Google released a compression algorithm called Brotli and gave files it makes the extension “.bro”. But last week the extension was changed to “.br”. The reason for the change is threads like this one, in which posters suggest that “'bro' has a gender problem” and “comes of[f] misogynistic and unprofessional due to the world it lives in.” http://www.theregister.co.uk/2015/10/11/googles_bro_file_format_changed_to_br_after_gender_politics_worries/ http://www.theregister.co.uk/2015/10/11/googles_bro_file_for...
  - ant6n 11y ago
    I would've preferred .brot
    - oftenwrong 11y ago
      or just ".brotli" modern filesystems can handle a few extra characters
      - chocolatebunny 11y ago
        but we don't want to type them. when was the last time you typed chdir instead of cd?
        ctrl-j 11y ago
        when was the last time you were in a shell that didn't have tab completion? .br<tab> is the same number of characters as .bro
KurvaKde1334 11y ago
Official comparison by google https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf https://cran.r-project.org/web/packages/brotli/vignettes/bro...
superiphone77 11y ago
Heh, Brotli + http2, it looks so Fast: http://www.http2demo.io http://www.http2demo.io
grandalf 11y ago
Like Facebook's Dragon, Brotli is an algorithm that is optimized for typical usage patterns. Similarly, an entire IOS device could be fabricated as a single ASIC, and (for example) uikit could be fabricated as part of that ASIC. There is always a tradeoff between generic optimization and usage-specific optimization which comes at the expense of flexibility. Google can do a statistical analysis of all the data it serves compressed with gzip, and determine exactly the characteristics of a compression algorithm that would save the most money. These are small, evolutionary optimizations that save tons of money by incrementally increasing efficiency in a large system.
frewsxcv 11y ago
If anyone wants to check out a Rust implementation: https://github.com/ende76/brotli-rs https://github.com/ende76/brotli-rs It's currently in use in Servo
ape4 11y ago
I see its dictionary is optimized for web content. Seems a bit like cheating ;) https://en.wikipedia.org/wiki/Brotli https://en.wikipedia.org/wiki/Brotli Unlike most general purpose compression algorithms, Brotli uses a pre-defined 120 kilobyte dictionary. The dictionary contains over 13000 common words, phrases and other substrings derived from a large corpus of text and HTML documents.[6][7] A pre-defined algorithm can give a compression density boost for short data files.
- return0 11y ago
  How is it cheating if it was developed specifically for that?
  - Lagged2Death 11y ago
    "Cheating" might be a little strong, but "not actually comparable to general purpose tools like Gzip" seems justifiable.
    - kami8845 11y ago
      If we are talking about web content then how exactly is it not comparable?
- donatj 11y ago
  Huh, yeah that does feel like cheating. Even having some sort of sheared dictionary feels better than a hard-coded one. I wonder how well it performs on things that aren't text based?
  - efuquen 11y ago
    Since when has being designed and optimized for a specific use case ever been "cheating" in software development?
  - IncRnd 11y ago
    Static dictionaries are used frequently in data communications to "break the bounds" of Shannon's theorem. See static Huffman coding in fax machines.
charlesju 11y ago
Is this a Silicon Valley TV show joke or is this real? Not sarcasm, real question.
- shawabawa3 11y ago
  It's real
- detaro 11y ago
  What made you wonder about it being a joke?
cromwellian 11y ago
I think decompression speed is more important. I'm willing to burn compression time precompressing resources. For more resources, especially JS and images, you can achieve precompression. Or, if you compress on the fly, you can cache the result.
GigabyteCoin 11y ago
Title should be changed to "...25% improvement over Gzip" re: "Brotli should bring 25% reduction in data size compared to Gzip for the most common assets like Javascript and CSS files. For HTML, Brotli promises up to 40% difference (with median around 25%)."
Joky 11y ago
So since this is a "dictionary" based algorithm, what about specializing the dictionary per content (content/type=javascript would have a different dictionary than html for instance). Also to embed Brotli for my custom application format, what should I expect by specializing the dictionary for my use case?
SureshG 11y ago
Do we have any implementation for Brotli encoder/decoder on JVM or JNI/JNA is the best option available right now for using Brotli on jvm apps?

227 points by clarinois 11y ago

pmlnr 11y ago

Are there any memory & CPU consumption graphs? I also don't understand why not xz, though I know that requires significantly more resource than gzip.

TheCondor 11y ago
Brotli actually has levels of compression that are more dense than gzip and compress/decompress faster. There is also a spec, xz has a reference implementation but no spec.
- nacs 11y ago
  > more dense than gzip and compress/decompress faster If you're saying brotli compresses/decompresses faster that gzip, it doesn't according to Cloudflare tests [1] (see table near bottom). Even the fastest compression level of Brotli is slower than the highest/slowest compression level of gzip in most cases. [1]: https://blog.cloudflare.com/results-experimenting-brotli/ https://blog.cloudflare.com/results-experimenting-brotli/
  - cornstalks 11y ago
    > If you're saying brotli compresses/decompresses faster that gzip, it doesn't according to Cloudflare tests. It doesn't always compress faster, but as far as I can tell they didn't measure or say anything about decompression.
  - Nelson69 11y ago
    I just did this: time bro --quality 6 --input linux-4.5.tar --output linux-4.5.tar.bro real 0m18.436s user 0m18.228s sys 0m0.184s time gzip -9 linux-4.5.tar real 0m30.555s user 0m30.424s sys 0m0.172s ls -lh (I removed the metadata): 106M linux-4.5.tar.bro 129M linux-4.5.tar.gz Only 82% of the size at 60% of the time. Now this is pure text so that's not a good example of everything.
doomrobo 11y ago
>XZ is a file format, LZMA is the compression algorithm. https://bugzilla.mozilla.org/show_bug.cgi?id=366559#c18 https://bugzilla.mozilla.org/show_bug.cgi?id=366559#c18
sp332 11y ago
If it takes a lot of CPU, it might not be worth the CPU time or cost. https://i.imgur.com/bAc1Saq.png https://i.imgur.com/bAc1Saq.png Image from https://dl.acm.org/citation.cfm?id=1084786&dl=ACM&coll=DL&CFID=763952065&CFTOKEN=92459762 https://dl.acm.org/citation.cfm?id=1084786&dl=ACM&coll=DL&CF... And gzip speed has improved since 2005, so over a 10 Mbps connection, you'd have to get much better compression to be worth the switch.

jakubstr 11y ago

Nice. Hope to see it in FF soon. And in Internet Explorer 87.

KurvaKde1334 11y ago
In 2020 maybe :)
TheRealPomax 11y ago
http://caniuse.com/#search=brotli http://caniuse.com/#search=brotli tells us FF already supports this, and IE has it marked as being under implementation consideration (which one would expect to turn into active development once they finish their current WOFF2 support work; WOFF2 has a hard dependency on a working Brotli implementation)
- KurvaKde1334 11y ago
  Try nginx & FF45 -> doesnt work as FF uses deprecated (and now obsolete & removed) API
  - bzbarsky 11y ago
    Reference for this, please? Because I see no bugs filed on FF regarding brotli apart from https://bugzilla.mozilla.org/show_bug.cgi?id=1222541 https://bugzilla.mozilla.org/show_bug.cgi?id=1222541 and https://bugzilla.mozilla.org/show_bug.cgi?id=1207234 https://bugzilla.mozilla.org/show_bug.cgi?id=1207234 neither of which seems to match your problem description. I'm happy to get a bug filed for you so whatever problem this is can be fixed, but I need a bit more to go on here... I did look around for known brotli+Firefox+nginx issues, but the only one that comes up is https://bugzilla.mozilla.org/show_bug.cgi?id=1215724 https://bugzilla.mozilla.org/show_bug.cgi?id=1215724 which was fixed in shipping Firefox months ago, so I'm assuming that's not the one you're talking about.
    - KurvaKde1334 11y ago
      Sure, there is no official bug for this. The only way to see it is to check the dates when the new API was applied - its far too late after releasing versions already supporting brotli, but this commit was not backported anywhere so in a real life, its broken in FF. Just get an nginx, nginx brotli module from google and use FF 45. Good luck :) https://hg.mozilla.org/releases/mozilla-aurora/log/10e1774de80e/modules/brotli/dec/decode.h https://hg.mozilla.org/releases/mozilla-aurora/log/10e1774de...

jgrahamc 11y ago

We took a very detailed look at Brotli performance (both speed and compression) and concluded that for dynamic content it was only useful if the file size was greater than 64k on slow connections and there's a very detailed description here: https://blog.cloudflare.com/results-experimenting-brotli/ https://blog.cloudflare.com/results-experimenting-brotli/ If you want to experiment with it, it's enabled on CloudFlare's test server https://http2.cloudflare.com/ https://http2.cloudflare.com/. Since we don't charge by byte the focus is on end-user performance and not 'saving money' and we concluded that it wasn't currently worth implementing Brotli widely but will continue to experiment with it.

KurvaKde1334 11y ago
There's a lot of stuff greater then 64k nowadays ;)
- mwpmaybe 11y ago
  Especially those pesky XML and JSON payloads!
- clarinois 11y ago
  Exactly :) If Cloudflare decides it’s worth several man-days to offer this feature to their clients who deliver larger files than 64k, feel free to reach out to us, we’ll be happy to share our experience :) devs@cdn77.com
  - jgrahamc 11y ago
    Thanks. We actually wrote and open sourced our nginx modifications for choosing between Brotli and gzip. https://github.com/cloudflare/ngx_brotli_module https://github.com/cloudflare/ngx_brotli_module
  - deleted 11y ago
    [deleted]
- acqq 11y ago
  From the Cloduflare's tests: "Most files are smaller than 64KB, and if we look only at those files then Brotli 4 is actually 1.48X slower than zlib level 8!" And the faster zlib levels (1-7) are even faster! See the table. I especially like zlib 1.

bhouston 11y ago

I find that Brotli is over sold for the case of compressing generic binary files. The claims that it is comparable to xz/lzma for generic binary data are not accurate. In my real-world tests of compressing 3D data it far underperformed xz/lzma although it was still better than gzip: https://github.com/google/brotli/issues/165 https://github.com/google/brotli/issues/165

kevingadd 11y ago
It's fairly competitive with LZHAM, at least, even if it's way slower to compress. You will get much better results out of Brotli if you restructure your data to be more compressible, and that will also improve your lzma and gzip (especially gzip) compression ratios, to a tremendous degree. Have you done any of this? If not, ping me, and I can explain some techniques to apply.
- detaro 11y ago
  >You will get much better results out of Brotli if you restructure your data to be more compressible, and that will also improve your lzma and gzip (especially gzip) compression ratios, to a tremendous degree. This sounds interesting, I'd like to read some examples/links/explanations.
  - EdHominem 11y ago
    I imagine a lot of it is segregating your content. All strings in one file, etc. It'd be an interesting test to take our eight-language set of localization strings and compress them in UI order and language order and see if there's much of a difference. (UI order is all languages for one dialog element, then all eight for the next, etc. Language order is all the English first, then ...) I'd definitely like to hear Kevingadd's tips though.

sremani 11y ago

Is it true that Brotli wanted the .bro file extension and moved away from it, because it was deemed offensive?

return0 11y ago
Interesting. Is it also middle-out?
Bud 11y ago
Yes, that's confirmed. "In late September, Google released a compression algorithm called Brotli and gave files it makes the extension “.bro”. But last week the extension was changed to “.br”. The reason for the change is threads like this one, in which posters suggest that “'bro' has a gender problem” and “comes of[f] misogynistic and unprofessional due to the world it lives in.” http://www.theregister.co.uk/2015/10/11/googles_bro_file_format_changed_to_br_after_gender_politics_worries/ http://www.theregister.co.uk/2015/10/11/googles_bro_file_for...
- ant6n 11y ago
  I would've preferred .brot
  - oftenwrong 11y ago
    or just ".brotli" modern filesystems can handle a few extra characters
    - chocolatebunny 11y ago
      but we don't want to type them. when was the last time you typed chdir instead of cd?
      - ctrl-j 11y ago
        when was the last time you were in a shell that didn't have tab completion? .br<tab> is the same number of characters as .bro

KurvaKde1334 11y ago

Official comparison by google https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf https://cran.r-project.org/web/packages/brotli/vignettes/bro...

superiphone77 11y ago

Heh, Brotli + http2, it looks so Fast: http://www.http2demo.io http://www.http2demo.io

grandalf 11y ago

Like Facebook's Dragon, Brotli is an algorithm that is optimized for typical usage patterns. Similarly, an entire IOS device could be fabricated as a single ASIC, and (for example) uikit could be fabricated as part of that ASIC. There is always a tradeoff between generic optimization and usage-specific optimization which comes at the expense of flexibility. Google can do a statistical analysis of all the data it serves compressed with gzip, and determine exactly the characteristics of a compression algorithm that would save the most money. These are small, evolutionary optimizations that save tons of money by incrementally increasing efficiency in a large system.

frewsxcv 11y ago

If anyone wants to check out a Rust implementation: https://github.com/ende76/brotli-rs https://github.com/ende76/brotli-rs It's currently in use in Servo

ape4 11y ago

I see its dictionary is optimized for web content. Seems a bit like cheating ;) https://en.wikipedia.org/wiki/Brotli https://en.wikipedia.org/wiki/Brotli Unlike most general purpose compression algorithms, Brotli uses a pre-defined 120 kilobyte dictionary. The dictionary contains over 13000 common words, phrases and other substrings derived from a large corpus of text and HTML documents.[6][7] A pre-defined algorithm can give a compression density boost for short data files.

return0 11y ago
How is it cheating if it was developed specifically for that?
- Lagged2Death 11y ago
  "Cheating" might be a little strong, but "not actually comparable to general purpose tools like Gzip" seems justifiable.
  - kami8845 11y ago
    If we are talking about web content then how exactly is it not comparable?
donatj 11y ago
Huh, yeah that does feel like cheating. Even having some sort of sheared dictionary feels better than a hard-coded one. I wonder how well it performs on things that aren't text based?
- efuquen 11y ago
  Since when has being designed and optimized for a specific use case ever been "cheating" in software development?
- IncRnd 11y ago
  Static dictionaries are used frequently in data communications to "break the bounds" of Shannon's theorem. See static Huffman coding in fax machines.

charlesju 11y ago

Is this a Silicon Valley TV show joke or is this real? Not sarcasm, real question.

shawabawa3 11y ago
It's real
detaro 11y ago
What made you wonder about it being a joke?

cromwellian 11y ago

I think decompression speed is more important. I'm willing to burn compression time precompressing resources. For more resources, especially JS and images, you can achieve precompression. Or, if you compress on the fly, you can cache the result.

GigabyteCoin 11y ago

Title should be changed to "...25% improvement over Gzip" re: "Brotli should bring 25% reduction in data size compared to Gzip for the most common assets like Javascript and CSS files. For HTML, Brotli promises up to 40% difference (with median around 25%)."

Joky 11y ago

So since this is a "dictionary" based algorithm, what about specializing the dictionary per content (content/type=javascript would have a different dictionary than html for instance). Also to embed Brotli for my custom application format, what should I expect by specializing the dictionary for my use case?

SureshG 11y ago

Do we have any implementation for Brotli encoder/decoder on JVM or JNI/JNA is the best option available right now for using Brotli on jvm apps?