4 ms·

Zstandard is great indeed. Performance is excellent, but in my experience its killer feature is the user definable dictionary. My search engine software stores

Zstandard is great indeed. Performance is excellent, but in my experience its killer feature is the user definable dictionary. My search engine software stores its documents in an archive to which I need random access. Lots of small documents, random access: bad for compression. Use of zstd with a tuned dictionary improved compression fivefold(!) when compared to the older zlib solution.

walrus01 6y ago
> but in my experience its killer feature is the user definable dictionary. If I had to guess, since this originated from within Facebook, they're using the feature for user definable dictionary to maintain a large corpus of sample, representative user uploaded content. Which would accomplish the purpose of tuning its performance for their own specific internal usage.
- Mokosha 6y ago
  This didn't originate from within Facebook. It is an evolution of the work done in late 2013/early 2014 around FSE[1]. Facebook later hired Yann to support his work around this (pretty amazing!) tool. The ability to define your own dictionary is useful beyond simply large indexable datasets. Anything application-specific can usually benefit from user-defined dictionaries. [1] http://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-breed-of.html http://fastcompression.blogspot.com/2013/12/finite-state-ent...
- kieckerjan 6y ago
  The dictionary helps with the boilerplate stuff that is shared between records but not repeated inside them. So it works like a charm for relatively small fragments of structured data like e.g. HTML pages, XML-docs or SQL statements: lots of "SELECT * FROM" fragments that can be replaced by a single token. I suspect it works even better for this kind of data than for natural language data, which tends to be skewed in a Zipf-like way.
staticassertion 6y ago
I've wanted to use zstandard with a dictionary, but how do people manage their dictionaries? What if I want to update a dictionary, will that break old compressed data? Do I need to 'tag' the data so I know which dictionary to fetch?
- kieckerjan 6y ago
  It is not the responsibility of a compressor to manage (or even store) your data. So yes, you have to come up with some sort of scheme yourself. But the overhead of a version number or something is usually negligible compared to the gains.
  - staticassertion 6y ago
    I'm a lot less concerned with adding a version number and much more concerned with the additional complexity of having to fetch a dictionary.
- ctur 6y ago
  We documented some of this in our Engineering blog posts on Zstandard. Check this one out and search for "Managed Compression" for the system we built. https://engineering.fb.com/2018/12/19/core-data/zstandard/ https://engineering.fb.com/2018/12/19/core-data/zstandard/
- kelnos 6y ago
  My first thought would be to store the dictionary in a file with the same name as the archive, but with '.dict' or something like that appended. If you're concerned about wasting space by duplicating the dictionary for many many archives, use a single file and hard links instead.

walrus01 6y ago

> but in my experience its killer feature is the user definable dictionary. If I had to guess, since this originated from within Facebook, they're using the feature for user definable dictionary to maintain a large corpus of sample, representative user uploaded content. Which would accomplish the purpose of tuning its performance for their own specific internal usage.

Mokosha 6y ago
This didn't originate from within Facebook. It is an evolution of the work done in late 2013/early 2014 around FSE[1]. Facebook later hired Yann to support his work around this (pretty amazing!) tool. The ability to define your own dictionary is useful beyond simply large indexable datasets. Anything application-specific can usually benefit from user-defined dictionaries. [1] http://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-breed-of.html http://fastcompression.blogspot.com/2013/12/finite-state-ent...
kieckerjan 6y ago
The dictionary helps with the boilerplate stuff that is shared between records but not repeated inside them. So it works like a charm for relatively small fragments of structured data like e.g. HTML pages, XML-docs or SQL statements: lots of "SELECT * FROM" fragments that can be replaced by a single token. I suspect it works even better for this kind of data than for natural language data, which tends to be skewed in a Zipf-like way.

staticassertion 6y ago

I've wanted to use zstandard with a dictionary, but how do people manage their dictionaries? What if I want to update a dictionary, will that break old compressed data? Do I need to 'tag' the data so I know which dictionary to fetch?

kieckerjan 6y ago
It is not the responsibility of a compressor to manage (or even store) your data. So yes, you have to come up with some sort of scheme yourself. But the overhead of a version number or something is usually negligible compared to the gains.
- staticassertion 6y ago
  I'm a lot less concerned with adding a version number and much more concerned with the additional complexity of having to fetch a dictionary.
ctur 6y ago
We documented some of this in our Engineering blog posts on Zstandard. Check this one out and search for "Managed Compression" for the system we built. https://engineering.fb.com/2018/12/19/core-data/zstandard/ https://engineering.fb.com/2018/12/19/core-data/zstandard/
kelnos 6y ago
My first thought would be to store the dictionary in a file with the same name as the archive, but with '.dict' or something like that appended. If you're concerned about wasting space by duplicating the dictionary for many many archives, use a single file and hard links instead.