3 ms·
The Request For Position on Mozilla Zstd Support (2018) has a ton of interesting discussion on dictionaries. https://github.com/mozilla/standards-positions/issu
by jauntywundrkind 3y ago
The Request For Position on Mozilla Zstd Support (2018) has a ton of interesting discussion on dictionaries. https://github.com/mozilla/standards-positions/issues/105 https://github.com/mozilla/standards-positions/issues/105
The original proposal for Zstd was to use a predefined stastically generated dictionary. Mozilla rejected the proposal for that.
But there's a lot of great discussion on what Zstd can do, whic.h is astoundingly flexible & powerful. There's discussion on dynamic adjustment if cinpression ratios. And discussion around shared dictionaries and their privacy implications. That Mozilla turned around & started supporting Zstd & has stamped a positive indicator, worth prototyping on shared dictionaries is a good initial stamp of approval to see! https://github.com/mozilla/standards-positions/issues/771 https://github.com/mozilla/standards-positions/issues/771
One of my main questions after reading this promising update is: how do pick what to include when generating custom dictionaries? Another comment mentions that brotli has a standard dictionary it uses, and that's some kind of possible starting place. But it feels like tools to build one's custom dictionary would be ideal.
- patrickmeenan 3y agoThe brotli repo on github has a dictionary generator: https://github.com/google/brotli/blob/master/research/dictionary_generator.cc https://github.com/google/brotli/blob/master/research/dictio... I have a hosted version of it on https://use-as-dictionary.com/ https://use-as-dictionary.com/ to make it easier to experiment with.