3 ms·

Problem with Snebu: it only does file-level deduplication I think which means storing something like virtual machine snapshots or images will not deduplicate we

• derekp7 12y ago
  That is correct -- at this time it is only doing file level dedup, due to overhead and complexity of block level (or better yet, variable block level) deduplication. The file level dedup is done by computing the sha1 checksum of the file, and using that as the name to store the file contents under. I've got a solution in the works specifically for KVM images, that hopefully I'll be able to finish up the next time I get a week off of work. The way I'm planning on handling that (at least for LibVirt based VMs) is to use libguestfs to create an equivalent to the `find` command (with all the -printf options that it supports), and to generate tar file output of selected files from the VM. (Snebu is designed around only requiring find/tar on the client side -- so anything that can produce this output will work). Although I don't like that libguestfs actually fires up a VM in order to work with the image files -- I may work on an approach to read the qcow2 file images directly. Again, looking forward to a vacation this spring so I can pound out that module.