3 ms·

I wonder how well a zstd dictionary will work against these kinds of data sets.

• nullc 4y ago
  They're big enough that it shouldn't help. Dictionaries are useful when you're compressing tiny files such that the overhead of the compressor learning the statistics from the input is significant. Basically you hope to save a fraction of the size of the dictionary from each file. From a 2GB file that's negligible. From a 1000 byte file it could be gigantic.
• croemer 4y ago
  Using a custom dictionary works well if you want to compress single consensus sequences (30kbp), say before putting them into an sqlite db. Then you can get each sequence down to ~200 bytes. If you compress a large dataset, there's no point in using a custom dictionary, the auto-embedded one works just fine.