5 ms·
I see its dictionary is optimized for web content. Seems a bit like cheating ;) https://en.wikipedia.org/wiki/Brotli https://en.wikipedia.org/wiki/Brotli Unlike
by ape4 11y ago
I see its dictionary is optimized for web content.
Seems a bit like cheating ;)
https://en.wikipedia.org/wiki/Brotli https://en.wikipedia.org/wiki/Brotli
Unlike most general purpose compression algorithms, Brotli uses a pre-defined 120 kilobyte dictionary. The dictionary contains over 13000 common words, phrases and other substrings derived from a large corpus of text and HTML documents.[6][7] A pre-defined algorithm can give a compression density boost for short data files.
- return0 11y agoHow is it cheating if it was developed specifically for that?
- Lagged2Death 11y ago"Cheating" might be a little strong, but "not actually comparable to general purpose tools like Gzip" seems justifiable.
- kami8845 11y agoIf we are talking about web content then how exactly is it not comparable?
- donatj 11y agoHuh, yeah that does feel like cheating. Even having some sort of sheared dictionary feels better than a hard-coded one. I wonder how well it performs on things that aren't text based?
- efuquen 11y agoSince when has being designed and optimized for a specific use case ever been "cheating" in software development?
- IncRnd 11y agoStatic dictionaries are used frequently in data communications to "break the bounds" of Shannon's theorem. See static Huffman coding in fax machines.
- deleted 11y ago[deleted]
- donatj 11y agoWell unlike fax machines though are protocol shifts over time. New tags and conventions are invented. A known shared dictionary seems a better method than a hard coded static dictionary.
- NelsonMinar 11y agoWhat's wrong with cheating?
- ape4 11y agoWhat if a new keyword is added to javascript / html.
- jzymbaluk 11y agoThen I imagine that keyword would be added to the static dictionary.
- Dylan16807 11y agoThen you lose 6 bytes off of optimal.
- kevingadd 11y agoIt leads people to believe this will be a compression improvement for all their content, but after they do the work to integrate it they may discover that it's only a slight improvement because their content isn't identical to the content its static dictionary was designed for. It's at least a very good codec, though, so it's still a win for other data. Just smaller than you might expect.
- xnyhps 11y agoAlso note that the dictionary is strongly biased towards English. Sure, there is some Russian, Chinese, Arabic (and probably some other scripts in there which I don't recognize), but there seems to be more English words in there than all those others combined. If you're compressing small documents in any other language than English, it might not be worth it to use Brotli. Edit: They wrote this about it in http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf : > Unlike other algorithms compared here, brotli includes a static dictionary. It contains 13’504 words or syllables of English, Spanish, Chinese, Hindi, Russian and Arabic, as well as common phrases used in machine readable languages, particularly HTML and JavaScript. The total size of the static dictionary is 122’784 bytes. The static dictionary is extended by a mechanism of transforms that slightly change the words in the dictionary. A total of 1’633’984 sequences, although not all of them unique, can be constructed by using the 121 transforms. To reduce the amount of bias the static dictionary gives to the results, we used a multilingual web corpus of 93 different languages where only 122 of the 1285 documents (9.5 %) are in languages supported by our static dictionary.