2 ms·
You need what's called "solid compression", meaning that the files are concatenated together before compression. Because otherwise compression restarts from zer
by 323 4y ago
You need what's called "solid compression", meaning that the files are concatenated together before compression. Because otherwise compression restarts from zero for each new file - https://en.wikipedia.org/wiki/Solid_compression https://en.wikipedia.org/wiki/Solid_compression
Some compressors have options for solid compression, or you can use tar to first concatenate the files. For both, you need to sort the files first, so that they are compressed in chronological order, otherwise the common information will drop out of the dictionary.
And how big is one file and how much RAM you have? Because you might need increasing the "dictionary size" of the compression algorithm, both 7zip and zstd support multi-gigabyte sizes.
- guipsp 4y agoIn addition to this, you can pretrain a dictionary on sample data, and then use that when compressing files individually
- cogman10 4y agozstd supports using a dict across compression actions, which is what you'd want for this.
- donatj 4y agoThe files start at around 700mb and approach 1.8gb by the end of the series. I've got 64gb of ram and 10 cores to work with, but it seems single core compression might be in order?
- 323 4y ago7zip can use 2 cores (maybe more now), zstd can use all of them, but zstd doesn't support solid compression - you need to use tar with it.