3 ms·
I've wanted to use zstandard with a dictionary, but how do people manage their dictionaries? What if I want to update a dictionary, will that break old compress
by staticassertion 6y ago
I've wanted to use zstandard with a dictionary, but how do people manage their dictionaries? What if I want to update a dictionary, will that break old compressed data? Do I need to 'tag' the data so I know which dictionary to fetch?
- kieckerjan 6y agoIt is not the responsibility of a compressor to manage (or even store) your data. So yes, you have to come up with some sort of scheme yourself. But the overhead of a version number or something is usually negligible compared to the gains.
- staticassertion 6y agoI'm a lot less concerned with adding a version number and much more concerned with the additional complexity of having to fetch a dictionary.
- ctur 6y agoWe documented some of this in our Engineering blog posts on Zstandard. Check this one out and search for "Managed Compression" for the system we built. https://engineering.fb.com/2018/12/19/core-data/zstandard/ https://engineering.fb.com/2018/12/19/core-data/zstandard/
- kelnos 6y agoMy first thought would be to store the dictionary in a file with the same name as the archive, but with '.dict' or something like that appended. If you're concerned about wasting space by duplicating the dictionary for many many archives, use a single file and hard links instead.