9 ms·

Brotli compression for the web

88 points by ssttoo 11y ago

diziet 11y ago
The support is really only on firefox: http://caniuse.com/#search=brotli http://caniuse.com/#search=brotli Chromium is still milling on this: https://code.google.com/p/chromium/issues/detail?id=452335 https://code.google.com/p/chromium/issues/detail?id=452335 But there seems to be some progress: https://chromium.googlesource.com/chromium/src.git/+log/lkcr/third_party/brotli https://chromium.googlesource.com/chromium/src.git/+log/lkcr...
- nailer 11y ago
  That's fine. It's quite possible they could sort out Chrome and most of the world's browsers would have the feature by the end of the year. We live in exciting times :-)
Sami_Lehtinen 11y ago
More detailed comparison: https://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf https://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf
- colanderman 11y ago
  Thank you. That is loads more informative than the linked blog post, which does not even bother to compare to LZMA.
  - ericlaw 11y ago
    It doesn't bother to compare LZMA because no major browser is planning to add LZMA.
- mavam 11y ago
  And one more comparative benchmark featuring Brotli: https://github.com/mavam/compbench https://github.com/mavam/compbench
Dylan16807 11y ago
I still don't understand why it's 'br' in the header and not 'brotli'
- sschueller 11y ago
  Maybe because the actual name is "Brötli" and "Zöplfi" coming from the Swiss German naming of small breads. Umlauts are usually converted like so: "ö" to "oe" but not always. So there may be confusion as to if it's "brotli" or "broetli". Then again, this is just some wild speculation. :)
  - kuschku 11y ago
    Actually, they decided the header would be .bro During a lot of overplayed drama it was decided to rename it to .br instead.
    - meowface 11y ago
      I feel like ".brot" would've been a reasonable compromise compared to ".br". (Though I also still think absolutely nothing was wrong with ".bro".)
      - anon4 11y ago
        That's because you're a man.
        meowface 11y ago
        I am a man, but I don't think that's why I think it. I'd be perfectly fine with a ".girl" or ".chick" or ".fem" or ".sis" extension, or any similar variation, if it happened to be a valid acronym for some technology. Unless the issue is that "bro" has some other kind of objectively offensive connotations, which I completely disagree with. Many women (and some men) use "sis" (or, much more rarely, "dudette"), in the exact same way as "bro". (And for the record, I didn't downvote you.)
kenOfYugen 11y ago
Previous discussion from 3 months ago: https://news.ycombinator.com/item?id=10257305 https://news.ycombinator.com/item?id=10257305
kevingadd 11y ago
Brotli is an awesome codec, but in a lot of my tests it was not really compelling versus LZHAM in terms of compression speed or compression ratio. In some test cases it edges it out on speed or ratio, but on others it's a tie if not worse. Sadly, in my testing I also hit at least one crash bug in the Brotli compressor, which makes it feel like an immature codec. On the bright side, the bug was fixed extremely quickly. I hope integration in browsers will help push the codec forward and encourage the developers of other codecs to try harder so that we get a really heated competition :-)
- ericlaw 11y ago
  > " it was not really compelling versus LZHAM " For those of us building things on the web, things supported by browsers are far more compelling than those that are not.
amelius 11y ago
I'm hoping for a compression algorithm which allows for separated codebooks, so e.g. one can compactly send updates when previous versions have already been transferred.
- robryk 11y ago
  What do you mean by a codebook? Something like a static dictionary?
  - PeCaN 11y ago
    For LZ algorithms, the dictionary made when compressing the first version would be reused for subsequent "versions" of the same document. So sort of a static dictionary, but specialized for what you're sending. That or maybe he means the Huffman codes, in which case you'd do the same thing but s/LZ/Huffman/ s/dictionary/tree/g. I tend to favor algorithms like LZF and LZ4 that don't have a separate dictionary for compressing text though, since there's just about no performance loss on either end and compression is "good enough".
    - robryk 11y ago
      > For LZ algorithms, the dictionary made when compressing the first version would be reused for subsequent "versions" of the same document. So sort of a static dictionary, but specialized for what you're sending. Ah, so you mean something like what SDCH does. This works well for files that are smaller than the window size and has rewards rapidly diminishing with filesize past that.
- twotwotwo 11y ago
  Chrome has something like this called SDCH, but it hasn't seen a ton of adoption outside Google: https://en.wikipedia.org/wiki/SDCH https://en.wikipedia.org/wiki/SDCH https://engineering.linkedin.com/shared-dictionary-compression-http-linkedin https://engineering.linkedin.com/shared-dictionary-compressi... https://aprescott.com/posts/sdch https://aprescott.com/posts/sdch Guessing part of the hurdle is that you can't simply drop it in like gzip; you have to generate a dictionary and serve it and refresh it now and then (the LinkedIn post explains). And a CDN can't just cache "the SDCH'd version" of a resource; ideally it'd cache one version for each recently used dict. The other, slightly happier "problem," is that other tuning can make the additional gains from SDCH less dramatic. I looked, and the Google homepage has a blob of JS that's 40KB gzip+sdch'd--but it's ~100KB gzip -9'd, and probably costs me 0KB most of the time because it's cacheable and its Last-Modified is about a month ago. CloudFlare's Railgun proxy does use delta coding to good effect, though in a much different context (origin server to edge location). I can imagine other variations, like priming Brotli's (de)compressor's history buffer with the first bytes of the old version of a cacheable resource in an If-None-Match request, if both sides have the old version handy and advertise the encoding. Sounds like what you're saying. Like SDCH, that still isn't drop-in; the server has to remember old versions of things. And it only applies to cached resources. Anyhow, interesting space.
nailer 11y ago
Anyone know if there's something planned for haproxy?
deleted 11y ago
[deleted]
powturbo 11y ago
You can make your own tests and your data with TurboBench: https://github.com/powturbo/TurboBench https://github.com/powturbo/TurboBench

88 points by ssttoo 11y ago

The support is really only on firefox: http://caniuse.com/#search=brotli http://caniuse.com/#search=brotli Chromium is still milling on this: https://code.google.com/p/chromium/issues/detail?id=452335 https://code.google.com/p/chromium/issues/detail?id=452335 But there seems to be some progress: https://chromium.googlesource.com/chromium/src.git/+log/lkcr/third_party/brotli https://chromium.googlesource.com/chromium/src.git/+log/lkcr...

nailer 11y ago
That's fine. It's quite possible they could sort out Chrome and most of the world's browsers would have the feature by the end of the year. We live in exciting times :-)

Sami_Lehtinen 11y ago

More detailed comparison: https://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf https://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf

colanderman 11y ago
Thank you. That is loads more informative than the linked blog post, which does not even bother to compare to LZMA.
- ericlaw 11y ago
  It doesn't bother to compare LZMA because no major browser is planning to add LZMA.
mavam 11y ago
And one more comparative benchmark featuring Brotli: https://github.com/mavam/compbench https://github.com/mavam/compbench

Dylan16807 11y ago

I still don't understand why it's 'br' in the header and not 'brotli'

sschueller 11y ago
Maybe because the actual name is "Brötli" and "Zöplfi" coming from the Swiss German naming of small breads. Umlauts are usually converted like so: "ö" to "oe" but not always. So there may be confusion as to if it's "brotli" or "broetli". Then again, this is just some wild speculation. :)
- kuschku 11y ago
  Actually, they decided the header would be .bro During a lot of overplayed drama it was decided to rename it to .br instead.
  - meowface 11y ago
    I feel like ".brot" would've been a reasonable compromise compared to ".br". (Though I also still think absolutely nothing was wrong with ".bro".)
    - anon4 11y ago
      That's because you're a man.
      - meowface 11y ago
        I am a man, but I don't think that's why I think it. I'd be perfectly fine with a ".girl" or ".chick" or ".fem" or ".sis" extension, or any similar variation, if it happened to be a valid acronym for some technology. Unless the issue is that "bro" has some other kind of objectively offensive connotations, which I completely disagree with. Many women (and some men) use "sis" (or, much more rarely, "dudette"), in the exact same way as "bro". (And for the record, I didn't downvote you.)

kenOfYugen 11y ago

Previous discussion from 3 months ago: https://news.ycombinator.com/item?id=10257305 https://news.ycombinator.com/item?id=10257305

kevingadd 11y ago

Brotli is an awesome codec, but in a lot of my tests it was not really compelling versus LZHAM in terms of compression speed or compression ratio. In some test cases it edges it out on speed or ratio, but on others it's a tie if not worse. Sadly, in my testing I also hit at least one crash bug in the Brotli compressor, which makes it feel like an immature codec. On the bright side, the bug was fixed extremely quickly. I hope integration in browsers will help push the codec forward and encourage the developers of other codecs to try harder so that we get a really heated competition :-)

ericlaw 11y ago
> " it was not really compelling versus LZHAM " For those of us building things on the web, things supported by browsers are far more compelling than those that are not.

amelius 11y ago

I'm hoping for a compression algorithm which allows for separated codebooks, so e.g. one can compactly send updates when previous versions have already been transferred.

robryk 11y ago
What do you mean by a codebook? Something like a static dictionary?
- PeCaN 11y ago
  For LZ algorithms, the dictionary made when compressing the first version would be reused for subsequent "versions" of the same document. So sort of a static dictionary, but specialized for what you're sending. That or maybe he means the Huffman codes, in which case you'd do the same thing but s/LZ/Huffman/ s/dictionary/tree/g. I tend to favor algorithms like LZF and LZ4 that don't have a separate dictionary for compressing text though, since there's just about no performance loss on either end and compression is "good enough".
  - robryk 11y ago
    > For LZ algorithms, the dictionary made when compressing the first version would be reused for subsequent "versions" of the same document. So sort of a static dictionary, but specialized for what you're sending. Ah, so you mean something like what SDCH does. This works well for files that are smaller than the window size and has rewards rapidly diminishing with filesize past that.
twotwotwo 11y ago
Chrome has something like this called SDCH, but it hasn't seen a ton of adoption outside Google: https://en.wikipedia.org/wiki/SDCH https://en.wikipedia.org/wiki/SDCH https://engineering.linkedin.com/shared-dictionary-compression-http-linkedin https://engineering.linkedin.com/shared-dictionary-compressi... https://aprescott.com/posts/sdch https://aprescott.com/posts/sdch Guessing part of the hurdle is that you can't simply drop it in like gzip; you have to generate a dictionary and serve it and refresh it now and then (the LinkedIn post explains). And a CDN can't just cache "the SDCH'd version" of a resource; ideally it'd cache one version for each recently used dict. The other, slightly happier "problem," is that other tuning can make the additional gains from SDCH less dramatic. I looked, and the Google homepage has a blob of JS that's 40KB gzip+sdch'd--but it's ~100KB gzip -9'd, and probably costs me 0KB most of the time because it's cacheable and its Last-Modified is about a month ago. CloudFlare's Railgun proxy does use delta coding to good effect, though in a much different context (origin server to edge location). I can imagine other variations, like priming Brotli's (de)compressor's history buffer with the first bytes of the old version of a cacheable resource in an If-None-Match request, if both sides have the old version handy and advertise the encoding. Sounds like what you're saying. Like SDCH, that still isn't drop-in; the server has to remember old versions of things. And it only applies to cached resources. Anyhow, interesting space.

nailer 11y ago

Anyone know if there's something planned for haproxy?

deleted 11y ago

[deleted]

powturbo 11y ago

You can make your own tests and your data with TurboBench: https://github.com/powturbo/TurboBench https://github.com/powturbo/TurboBench