3 ms·

can't zstandard use standardized dictionaries as well?

• cyann 4y ago
  Yes, sort of: "Zstd offers a training mode, which can be used to tune the algorithm for a selected type of data. Training Zstandard is achieved by providing it with a few samples (one file per sample). The result of this training is stored in a file called "dictionary", which must be loaded before compression and decompression. Using this dictionary, the compression ratio achievable on small data improves dramatically." https://github.com/facebook/zstd#the-case-for-small-data-compression https://github.com/facebook/zstd#the-case-for-small-data-com...
• PoignardAzur 4y ago
  According to the Bugzilla thread, Facebook won't bother going through the process of publishing standardized dictionaries.
  • unsigner 4y ago
    I wouldn't go to the trouble of determining the privacy implications of standartized dictionaries built from user data.
  • wongarsu 4y ago
    But surely anyone could publish dictionaries? Like let's say Mozilla could publish a dictionary trained on HTTP headers, HTML and Javascript, and then accept traffic compressed with it under a "Accept-Encoding: zstd/mozilla" header