5 ms·
When can we bring this to the web? Zstd aka RFC8478[1] is so good. That it can continue to improve at all feels almost unbelievable, but @Cyan4973 &al continue
by rektide 6y ago
When can we bring this to the web? Zstd aka RFC8478[1] is so good. That it can continue to improve at all feels almost unbelievable, but @Cyan4973 &al continue to make it faster, somehow.
Especially on mobile, with large assets, I feel like zstd's lightning fast decompression time could be a huge win. It used to be that Brotli was the obvious choice for achieving high compression, but it doesn't feel so clear to me now. Here's one random run-off between the two[2].
The other obvious use case is if there is large-ish dynamic-ish data, where the cost of doing Brotli compression each time might be too high.
[1] https://datatracker.ietf.org/doc/html/rfc8478 https://datatracker.ietf.org/doc/html/rfc8478
[2] https://peazip.github.io/fast-compression-benchmark-brotli-zstandard.html https://peazip.github.io/fast-compression-benchmark-brotli-z...
- meltedcapacitor 6y agoBrotli is such an ugly hack (hardcoded dictionary with a snapshot of the world as it looked from Mountain View on some random day...), the quicker it dies the better.
- nvllsvm 6y agoIt's also impossible to identify whether an arbitrary file is compressed with brotli. It lacks a magic number. https://github.com/google/brotli/issues/298 https://github.com/google/brotli/issues/298
- duskwuff 6y agoThe contents of that hardcoded dictionary are really weird, too. It includes a lot of bizarrely specific entries like: in popular culture Holy Roman Emperor It is important to examples include the have speculated that aria-hidden="true">·< (new Date).getTime()} :url" content="http:// Many of the English phrases (like "in popular culture"!) are highly characteristic of content from the English Wikipedia. I've speculated before that this may be the result of the use of a compression benchmark which included (or consisted entirely of) content from that site, such as https://cs.fit.edu/~mmahoney/compression/textdata.html https://cs.fit.edu/~mmahoney/compression/textdata.html If you're curious, here's a full dump of the dictionary: https://gist.github.com/duskwuff/8a75e1b5e5a06d768336c8c7c370f0f3 https://gist.github.com/duskwuff/8a75e1b5e5a06d768336c8c7c37...
- hifly 6y agohowever the idea seems sound? I can't help wondering why Google who can fund GP-3, couldn't come up with a better initial dictionary? especially for small responses it is a big win
- JyrkiAlakuijala 6y agoStatic dictionary is not why Brotli reaches excellent compression density. Much of it comes from the more advanced context modeling and in general more dynamic entropy modeling. Brotli wins over Zstd on web compression even when you fill the static dictionary with zeros. No Mountain View based engineers participated in the development of Brotli, it was built in Zurich. The static dictionary was distilled from a language corpus originally having 40 languages. To reduce the binary size while keeping effectiveness, I reduced it to six: English, Spanish, Chinese, Russian, Hindi, Arabic. What got into the dictionary and in which order it was (both words and transforms) were chosen by how much entropy they reduced in a relatively large test corpus including diverse material (for example over 110 natural languages). Brotli was originally designed as a faster to decode replacement of LZMA for fonts to enable WOFF2. LZMA was too slow for that use. WOFF2 is just geometry, no natural languages there. There, W3C observed that Brotli performed similarly to LZMA in density but was significantly (~5x) faster to decode -- and enabled the WOFF2 launch. Unlike SDCH, Brotli's performance does not degrade when the static dictionary ages. This is because Brotli only uses invalid LZ77 commands to codify static dictionary entries. If you don't use them, you are not paying for them. Just like Zstd, users can bring their own dictionaries when they prefer that -- a functionality that was in Brotli before it was introduced in Zstd. Unlike ZStd, Brotli degrades less for exotic languages heavy on utf-8 (Korean, Vietnamese, Chinese, Japanese, etc.), and especially so for mixed data such as HTML-commands mixed with the above utf-8 sources. Unlike LZMA, Brotli's context modeling works for very short data, too -- It becomes about 0.6 % worse on gigabyte corpora, but can be 15+ % better on shortest documents. Unlike Zstd, Brotli offers the data in a streamable order, i.e., hides less data during the transfer. The user can decode much more out of a brotli stream than a respective fraction of zstd stream (shadowed amount is in tens of bits vs. tens of kilobytes in zstd). This is because brotli does not reshuffle the data within the blocks for less cpu-work at decoding time. Any shadowing of data will make other dependant resources loads start later or dependant processing (Javascript or HTML parsing) to happen later. Most of Brotli's advances come from other algorithms than LZ77. The LZ77 part of ZStd and Brotli are essentially the same. The LZ77 algorithm is proven optimal when the data is infinitely long as well as the sliding window -- this means that the longer the data, the less difference you see. If you benchmark Brotli vs. Zstd with real life data (like HTML pages of 75 kB) you see a different performance behaviour than if you compare them using a 100 MB or 10 GB file. There, most of Brotli's benefits disappear. In a gigabyte multi-corpus comparison, Brotli still compresses ~5 % more than Zstd. See: https://github.com/google/brotli/issues/642 https://github.com/google/brotli/issues/642 -- aggregated here: https://encode.su/threads/2947-large-window-brotli-results-available-for-a-several-corpora?p=56651&viewfull=1#post56651 https://encode.su/threads/2947-large-window-brotli-results-a... Zstd's encoder has seen more work during the years and some features (like the long maching) is missing from Brotli. Reaching parity in the encoder could reduce the gap. An interesting option would be to have a single encoder for both. Compiling is also difficult: One continual topic has been performance degradations when moving from GCC/CLANG to MSVC. Brotli was never optimized for MSVC and it seems something is going badly wrong there. Also, in the past several benchmarks were comparing a non-optimized build of Brotli vs a release build of Zstd. This was because until summer 2020 Brotli compiled without options produced a non-optimized build -- you'd need to configure release specifically to get it right.
- ac29 6y agoCaddy supports Zstd encoding: https://caddyserver.com/docs/caddyfile/directives/encode https://caddyserver.com/docs/caddyfile/directives/encode On the client end, curl does: https://curl.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html https://curl.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html
- chrismorgan 6y agoYou cite RFC 8478 as though it being an RFC gives it weight, but always remember when looking at IETF RFCs to check the Category, which is Informational in this case, not Standards Track. > Despite use of the word "standard" as part of its name, readers are advised that this document is not an Internet Standards Track specification; it is being published for informational purposes only. This is an IETF-stream document, which I believe (though I’m not certain) implies that it’s at least a little more than just Facebook saying “here, can we publish this?”, but it hasn’t come through any working group, so it hasn’t been subject to the full IETF experience which is so helpful at improving things by collaboration. (It’s also obsoleted by RFC 8878, to which the same caveats apply.)
- vanderZwan 6y ago> It’s also obsoleted by RFC 8878, to which the same caveats apply. Tangent: I decided to sit down and read that RFC yesterday. So I downloaded the PDF to my e-reader (because it doesn't support mono fonts for the plain text option). Turns out my Kobo H20 for some reason can't render the "fi" and "ff" ligatures, which made it a real struggle to read at times ("compressed les", "Human Encoding Trees"). Does anyone have any advice for how to turn plaintext RFCs into minimal PDFs with an embedded mono font? https://www.rfc-editor.org/info/rfc8878 https://www.rfc-editor.org/info/rfc8878
- rini17 6y agoPrint as PDF from a browser or wordprocessor? In LibreOffice Writer you have the option to embed the font.
- vanderZwan 6y ago... sometimes the simplest solutions are the best. I feel so dumb, thanks! I could probably even set the page size to the right dimensions in LibreOffice Writer