4 ms·
So we've got a system that backs up to an XML file nightly. I noticed a couple weeks ago the bucket was using a not-insignificant amount of disk space. These X
by donatj 4y ago
So we've got a system that backs up to an XML file nightly. I noticed a couple weeks ago the bucket was using a not-insignificant amount of disk space.
These XML files are full backups rather than deltas - so each one contains the full previous file plus the new additional data.
My assumption was if I grabbed just a swath of the old ones, say a years worth and compressed them together I would get really decent savings.
I was pretty disappointed - gzip knocked 10 gb off of about 100 gb of data. I started doing some research and found people saying 7zip and it's sliding dictionary size options were the answer. After multiple tries, each run of 7zip taking multiple days I was able to get it down to about 70 gigabytes from 100. Better then gzip but frankly nowhere near what I would expect.
Does there exist a compression that could better handle this sort of expanding documents?
- lazide 4y agoThat doesn’t make much sense, window size or not. Even base64 encoded random data would be about that bad. Is the XML wrapping a bunch of other random data or something?
- cdavid 4y agobase64 random data would be that bad only because of base64. Random data does not compress at all on average. As an example, I have zstd enabled on some zfs pool. The client-side encrypted time machine backups does not even compress 1 %, as expected.
- lazide 4y agoYes, that’s why I said that? You can’t embed binary data in xml raw, so a common pattern is embedding it as base64, or similar type of wrapping. Technically, it’s possible to escape it in other ways, but it’s error prone. Either way, XML which has text strings or whatever typical XML document data that ISN’T something like base64 encoded random data should compress dramatically better than what the poster was talking about. So, what the hell is in your XML anyway?
- 323 4y agoYou need what's called "solid compression", meaning that the files are concatenated together before compression. Because otherwise compression restarts from zero for each new file - https://en.wikipedia.org/wiki/Solid_compression https://en.wikipedia.org/wiki/Solid_compression Some compressors have options for solid compression, or you can use tar to first concatenate the files. For both, you need to sort the files first, so that they are compressed in chronological order, otherwise the common information will drop out of the dictionary. And how big is one file and how much RAM you have? Because you might need increasing the "dictionary size" of the compression algorithm, both 7zip and zstd support multi-gigabyte sizes.
- guipsp 4y agoIn addition to this, you can pretrain a dictionary on sample data, and then use that when compressing files individually
- cogman10 4y agozstd supports using a dict across compression actions, which is what you'd want for this.
- donatj 4y agoThe files start at around 700mb and approach 1.8gb by the end of the series. I've got 64gb of ram and 10 cores to work with, but it seems single core compression might be in order?
- 323 4y ago7zip can use 2 cores (maybe more now), zstd can use all of them, but zstd doesn't support solid compression - you need to use tar with it.
- mkl 4y ago> Does there exist a compression that could better handle this sort of expanding documents? Yes: storing deltas. Git might work well. A backup system that does subfile deduplication may work too (e.g. Restic, Borg).
- bombcar 4y agoThat would be my idea, store the files in git and then backup the whole git repository.
- pavel_lishin 4y agoThey mention that the files start at 700mb: https://news.ycombinator.com/item?id=33646345 https://news.ycombinator.com/item?id=33646345 I was under the impression that git didn't do well with large files.
- glogla 4y agoYeah. We used to store PowerDesigner files in git a while back, which are XMLs usually smaller than 100 MB, and that was pretty much a disaster of hour taking clones already. I can't imagine 700 MB working.
- bombcar 4y agoGood point - I wonder if the XML really is "diffable" or if it's basically encoded binary data and has to be stored as a blob anyway.
- trillic 4y agolrzip Long Range ZIP or LZMA RZIP https://github.com/ckolivas/lrzip https://github.com/ckolivas/lrzip "A compression utility that excels at compressing large files (usually > 10-50 MB). Larger files and/or more free RAM means that the utility will be able to more effectively compress your files (ie: faster / smaller size), especially if the filesize(s) exceed 100 MB. You can either choose to optimise for speed (fast compression / decompression) or size, but not both."