4 ms·
This is really neat. Feature request: Add a "library" option to give mkdwarfs a list of files that should be loaded into the dedup mechanism first, but not sto
by mnw21cam 3y ago
This is really neat.
Feature request: Add a "library" option to give mkdwarfs a list of files that should be loaded into the dedup mechanism first, but not stored, allowing the image to be even smaller if the contents of the file can be retrieved from that library instead. Bonus points if you can specify a dwarfs image as a library and have it sensibly use the files contained in it.
Then you have the basis for a deduplicating incremental backup system. Currently, I have a system I wrote that will take a single file and a list of library files and produce a compressed deduplicated file that can re-create that single file using the library, which is great if you use tar to create that single file, but a little unwieldy when coming to decompress and restore everything. The bonus of making it a proper mountable filesystem instead is that then it's a proper mountable filesystem and retrieving single files is a doddle.
My use case is that I have students, and I have given them coursework, which involves them logging in to a Linux machine and hacking away. I want to store regular snapshots of their work so that I can keep a backup for their sake but also so I can see a progression of development to try to work out if they are cheating (yes, I have had to deal with this), but I don't want to store 100 copies of the same fairly large files.
- nextaccountic 3y agoborg and restic/rustic are alternatives that do this multi layered backup thing. they can be mounted as a filesystem as well (the rustic reimplementation of restic however doesn't support this yet) https://www.borgbackup.org/ https://www.borgbackup.org/ https://restic.net/ https://restic.net/ https://github.com/rustic-rs/rustic https://github.com/rustic-rs/rustic
- jszymborski 3y agoCan't say enough nice things about Borg and it's dedup. Saves me a lot of money in cloud storage.
- chuzz 3y agowould this be doable with overlayFS using a tmpfs to prepare a snapshot?
- ktm5j 3y agoI think Kopia would be great for your use case https://kopia.io/ https://kopia.io/ It has a great system to snapshot files but only store data if it's changed. I use it in an environment where I can't use something like zfs to snapshot data because I don't have the ability to make decisions about what filesystem we're using. It's been amazing, love it so much!