3 ms·

Every piece of information or file that is compressed sends a dictionary along with it. In the case of, say, many HTML or CSS files, this dictionary data is lik

Every piece of information or file that is compressed sends a dictionary along with it. In the case of, say, many HTML or CSS files, this dictionary data is likely nearly completely redundant. There's almost no added complexity since zstd already handles separate compression dictionaries quite well.

pornel 1y ago
The standard compressed formats don't literally contain a dictionary. The decompressed data becomes its own dictionary while its being decompressed. This makes the first occurrence of any pattern less efficiently compressed (but usually it's still compressed thanks to entropy coding), and then it becomes cheap to repeat. Brotli has a default dictionary with bits of HTML and scripts. This is built in into the decompressor, and not sent with the files. The decompression dictionaries aren't magic. They're basically a prefix for decompressed files, so that a first occurrence of some pattern can be referenced from the dictionary instead of built from scratch. This helps only with the first occurrences of data near the start of the file, and for all the later repetitions the dictionary becomes irrelevant. The dictionary needs to be downloaded too, and you're not going to have dictionaries all the way down, so you pay the cost of decompressing the data without a dictionary whether it's a dictionary + dictionary-using-file, or just the full file itself.
- yorwba 1y ago
  > The dictionary needs to be downloaded too Which is why the idea is to use a previous version of the same file, which you already have cached from a prior visit to the site. You pay the cost of decompressing without a dictionary, but only on the first visit. Basically it's a way to restore the benefits of caching for files that change often, but only a little bit each time.
- zvr 1y ago
  Of course, the Brotli default (built-in) dictionary is infamous for containing such strings like "Holy Roman Emperor", "Confederate States", "Dominican Republic", etc., due to the way it was created. One can see the whole dictionary in https://gist.github.com/duskwuff/8a75e1b5e5a06d768336c8c7c370f0f3 https://gist.github.com/duskwuff/8a75e1b5e5a06d768336c8c7c37.... Having a dictionary created by actual content to be compressed will end up with a very different dictionary.
- pmarreck 1y ago
  > The dictionary needs to be downloaded too, and you're not going to have dictionaries all the way down We already have a way to manage this: Standardizing and versioning dictionaries for various media types (also with a checksum), and then just caching them locally forever, since they should be immutable by design. To prevent an overgrowth of dictionaries with small differences, we could require each one to be an RFC.

pornel 1y ago

The standard compressed formats don't literally contain a dictionary. The decompressed data becomes its own dictionary while its being decompressed. This makes the first occurrence of any pattern less efficiently compressed (but usually it's still compressed thanks to entropy coding), and then it becomes cheap to repeat. Brotli has a default dictionary with bits of HTML and scripts. This is built in into the decompressor, and not sent with the files. The decompression dictionaries aren't magic. They're basically a prefix for decompressed files, so that a first occurrence of some pattern can be referenced from the dictionary instead of built from scratch. This helps only with the first occurrences of data near the start of the file, and for all the later repetitions the dictionary becomes irrelevant. The dictionary needs to be downloaded too, and you're not going to have dictionaries all the way down, so you pay the cost of decompressing the data without a dictionary whether it's a dictionary + dictionary-using-file, or just the full file itself.

yorwba 1y ago
> The dictionary needs to be downloaded too Which is why the idea is to use a previous version of the same file, which you already have cached from a prior visit to the site. You pay the cost of decompressing without a dictionary, but only on the first visit. Basically it's a way to restore the benefits of caching for files that change often, but only a little bit each time.
zvr 1y ago
Of course, the Brotli default (built-in) dictionary is infamous for containing such strings like "Holy Roman Emperor", "Confederate States", "Dominican Republic", etc., due to the way it was created. One can see the whole dictionary in https://gist.github.com/duskwuff/8a75e1b5e5a06d768336c8c7c370f0f3 https://gist.github.com/duskwuff/8a75e1b5e5a06d768336c8c7c37.... Having a dictionary created by actual content to be compressed will end up with a very different dictionary.
pmarreck 1y ago
> The dictionary needs to be downloaded too, and you're not going to have dictionaries all the way down We already have a way to manage this: Standardizing and versioning dictionaries for various media types (also with a checksum), and then just caching them locally forever, since they should be immutable by design. To prevent an overgrowth of dictionaries with small differences, we could require each one to be an RFC.