3 ms·
> Does there exist a compression that could better handle this sort of expanding documents? Yes: storing deltas. Git might work well. A backup system that do
by mkl 4y ago
> Does there exist a compression that could better handle this sort of expanding documents?
Yes: storing deltas. Git might work well. A backup system that does subfile deduplication may work too (e.g. Restic, Borg).
- bombcar 4y agoThat would be my idea, store the files in git and then backup the whole git repository.
- pavel_lishin 4y agoThey mention that the files start at 700mb: https://news.ycombinator.com/item?id=33646345 https://news.ycombinator.com/item?id=33646345 I was under the impression that git didn't do well with large files.
- glogla 4y agoYeah. We used to store PowerDesigner files in git a while back, which are XMLs usually smaller than 100 MB, and that was pretty much a disaster of hour taking clones already. I can't imagine 700 MB working.
- bombcar 4y agoGood point - I wonder if the XML really is "diffable" or if it's basically encoded binary data and has to be stored as a blob anyway.
- viraptor 4y agoTo expand on the dedup a bit, Borg uses content aware chunking, so any large enough repeats will be fully deduplicated without relying on any specific block size/alignment. This is amazing for "slightly changing files".