6 ms·
I run rdfind[1] as a cronjob to replace duplicates with hardlinks. Works fine! https://github.com/pauldreik/rdfind https://github.com/pauldreik/rdfind
by tiagod 2y ago
I run rdfind[1] as a cronjob to replace duplicates with hardlinks. Works fine!
https://github.com/pauldreik/rdfind https://github.com/pauldreik/rdfind
- Wowfunhappy 2y agoBut then you have to be careful not to remove the one which happens to be the "original" or the hardlinks will break, right?
- Dylan16807 2y agoNo, pointing to an original is how soft links work. Hard links are all equivalent. A file has any number of hard links, and at least in theory you can't distinguish between them. The risk with hardlinks is that you might alter the file. Reflinks remove that risk, and also perform very well.
- Wowfunhappy 2y agoThank you, I was unaware of this. However, the fact that editing one copy edits all of them still makes this a non-solution for me at least. I'd also strongly prefer deduping at the block level vs file level.
- mdaniel 2y agoI would suspect a call to $(chmod a-w) would fix that, or at least serve as a very fine reminder that there's something special about them
- AndrewDavis 2y agoSo this is great, if you're just looking to deduplicate read only files. Less so if you intend to write to them. Write to one and they're both updated. Anyway. Offline/lazy dedup (not in the zfs dedup sense) is something that could be done in userspace, at the file level on any filesystem that supports reflinks. When a tool like rdfind finds a duplicate, instead of replacing with a hardlink, create a copy of the file with `copy_file_range(2)` and let the filesystem create a reflink to it. Now you've got space savings and they're two separate files so if one is written to the other remains the same.
- spockz 2y agoHow would this work if I have snapshots? Wouldn’t then the version of the file I just replaced still be in use there? But maybe I also need to store the copy again if I make another snapshot because the “original “ file isn’t part of the snapshot? So now I’m effectively storing more not less?
- magicalhippo 2y agoAFAIK, yes. Blocks are reference counted, so if the duplicate file is in a snapshot then the blocks would be referenced by the snapshot and hence not be eligible for deallocation. Only once the reference count falls to zero would the block be freed. This is par for the course with ZFS though. If you delete a non-duplicated file you don't get the space back until any snapshots referencing the file are deleted.
- spockz 2y agoYes that snapshots incur a cost I know. But I’m wondering whether now the action of deduplicating actually created an extra copy instead of saving’one.
- magicalhippo 2y agoI don't fully understand the scenario you mentioned. Could you perhaps explain in a bit more detail?
- DannyBee 2y agocopy_file_range already works on zfs, but it doesn't guarantee anything interesting. Basically all dupe tools that are modern use fideduprange, which is meant to tell the FS which things should be sharing data, and let it take care of the rest. (BTRFS, bcachefs, etc support this ioctl, and zfs will soon too) Unlike copy_file_range, it is meant for exactly this use case, and will tell you how many bytes were dedup'd, etc.
- sureglymop 2y agoQuite cool, though it's not as storage saving as deduplicating at e.g. N byte blocks, at block level.