5 ms·

I never really understood with Google keeps pushing Brotli. LZ4 and ZStd both predate it by a couple of years, and seem to offer superior performance overall. I

• joelthelion 3y ago
  Patents, maybe?
• oynqr 3y ago
  Brotli usually compresses plain text better than ZSTD.
  • out_of_protocol 3y ago
    Both with dictionaries? Both without? Any links to benchmarks?
    • oynqr 3y ago
      Brotli comes with a dictionary by default, which is optimized for plaintext use cases. Once you give ZSTD a similar dictionary, its ratios are going to be similar/better. I won't provide random links, because those can't replace testing on your own dataset.
• __alexs 3y ago
  Zstd no longer comes with a patent grant which I guess makes it kind of hard to actually use for some people.
  • jacobgorm 3y ago
    Zstd is already used in the Linux kernel, and I would assume that any patents covering Lempel-Ziv and Huffman coding would have expired long ago. The situation around ANS also seems to have been resolved, see https://en.wikipedia.org/wiki/Asymmetric_numeral_systems#Tabled_variant_(tANS) https://en.wikipedia.org/wiki/Asymmetric_numeral_systems#Tab... .
    • abdullahhhh 3y ago
      [flagged]
    • __alexs 3y ago
      The Linux kernel is stuffed full of potentially patent encumbered code.
  • secondcoming 3y ago
    huh, isn’t zstd the compression used for debian packages?
• lifthrasiir 3y ago
  They were designed more or less independently from each other, and your statement that Zstandard predates Brotli is also wrong (both around 2015). They are both designed to be much more efficient than DEFLATE in both time and size axes, but otherwise have little similarities. Zstandard is notable in its use of ANS, while Brotli is probably the first mainstream format that prominently uses a 2nd-order context model which was previously thought to be too slow. And Brotli is designed for web contents in mind, which made it excellent for small inputs with no prior. Zstandard has no such treatments but have a better performance when I/O can handle it, which makes it excellent for server jobs.
  • jacobgorm 3y ago
    ZStd started out as ZHuff, which was first announced in 2008; see https://encode.su/threads/521-Zhuff-fast-compression https://encode.su/threads/521-Zhuff-fast-compression and http://fastcompression.blogspot.com/p/zhuff.html http://fastcompression.blogspot.com/p/zhuff.html I've been using LZ4 in commercial software since 2011, so unless I am some kind of luminary Google's compression experts must have been aware of both LZ4 and ZHuff when they started working on Brotli.
    • lifthrasiir 3y ago
      By that reckoning I can claim that the origin of Brotli also dates back to 2012 or earlier. :-) Indeed, Brotli authors were also reponsible for the VP8 Lossless codec (2012), Pik (2017) and eventually JPEG XL (2019, in part). The entropy coding scheme is substantially similar among those codecs for this reason. But really, the large amount of differences between two formats rather suggest that they were designed for different uses in mind, which I believe relate to the maximum possible I/O. Formats designed for the web don't have to have an extremely fast decoder because I/O is the bottleneck: 100 MB/s should be enough, and the earned time can be invested to higher compression ratio which affects the decoder performance.
      • adgjlsfhk1 3y ago
        otoh, they achieve pretty similar compression ratios and faster decode saves battery life and will let you get on to running the JavaScript earlier
• jauntywundrkind 3y ago
  Brotli has two major advantages. First it encodes to the same well known gzip format, so rolling it out didn't require new clients. Second, it didn't require dictionary support. Where-as if you look at Mozilla's standards-positions for zstd, they defer because it requires a compression dictionary to encode & Moz didn't want to have to "standardize" one specific dictionary. https://github.com/mozilla/standards-positions/issues/105 https://github.com/mozilla/standards-positions/issues/105
  • pmarreck 3y ago
    > Moz didn't want to have to "standardize" one specific dictionary You could have multiple sources of dictionary identified by something like a hash, and cached in the browser. For example you could have a general-purpose HTML dictionary, a general-purpose CSS dictionary and a general-purpose JS dictionary, have canonical/official versions of these hosted somewhere (say, on Github) and distributed wherever, and update them on a semi regular basis (at the cost of re-compression of the associated data); and in turn these could be overrided on perhaps a site by site basis with site-specific versions of these. You could basically associate an overridable but defaulted dictionary specification by MIME type (or none at all in the case of already-compressed data). Every file would have to be sent with the hash of the dictionary that was used to start compressing it, which would then be retrieved and cached somehow. I could see this resulting in some fairly significant byte savings, especially for retrieval of many smaller files. I bet this would also be useful potentially for email clients/servers.
    • jauntywundrkind 3y ago
      > You could have multiple sources of dictionary identified by something like a hash, and cached in the browser. The browser actually has a very advanced implementation of something like this! They're called urls. The work here is to make it possible to identify dictionaries & retrieve & cache them in the browsers. Mozilla I think smartly avoided creating a spec that requires some variety of centralized governance of what dictionaries to use, like you describe. Whatever the intent of such a community, it creates biases & will have a hard time figuring out what sets of dictionaries are worth creating, for what intents or code bases, and adapting or not as it discovers underserved code bases.
• pmarreck 3y ago
  The Zstd guy is employed by Facebook, so I'd suspect it's at least in part a NIH thing.