3 ms·

this more like a tool suggestion request, because I haven't been able to find a solution in Google. I'm looking for a more advanced duplicate files finder in L

by staz 5y ago

this more like a tool suggestion request, because I haven't been able to find a solution in Google. I'm looking for a more advanced duplicate files finder in Linux, specially one that can handle folders. Most tool just return a list of duplicate files, but what I need is to also know is if whole folders are duplicate, or subset of others and have it presented in an elegant way t o resolve conflict. ofc everything is in a big folder and everything is a mess, bit of a hoarding collection of files that got accumulated over the years. I could probably code some stupid script myself, if I ever got the time (spoiler I probably won't) but I have no idea on how to present the result elegantly. So it would be nice if such tool already existed.

minhmeoke 5y ago
https://meldmerge.org/ https://meldmerge.org/ might be what you are looking for. But if there are tools which can find duplicated files and directories which are named differently, I'd be interested in learning more about these as well.
- staz 5y ago
  meld can compare two directory if they have the same structure and name for files but can't really tell if I inside my foot folder, I have a folder root/X which is also present as root/category/b/X. (except all the file in it have been prepended by "foo" somehow) For tools which can find differently names files you have for examples: fslint (gui) or fdupes (cli)
  - minhmeoke 5y ago
    That's true, meld is designed for comparing directories which already have similar structures. It seems like what we're looking for is content-addressable storage [1]. The theory behind it appears to be based on Merkle trees and cryptographic signatures [2]. IPFS already has an implementation [3] of this, and there are other implementations (borgbackup, restic, zpaq) listed in that link and in the content-addressable storage article. Disclaimer: I haven't used any of these yet, just found them a few minutes ago. [1] https://en.wikipedia.org/wiki/Content-addressable_storage https://en.wikipedia.org/wiki/Content-addressable_storage [2] https://gist.github.com/mafintosh/464bb8f1451f22c9e5c5 https://gist.github.com/mafintosh/464bb8f1451f22c9e5c5 [3] https://discuss.ipfs.io/t/ipfs-and-file-deduplication/4674 https://discuss.ipfs.io/t/ipfs-and-file-deduplication/4674

by staz 5y ago

minhmeoke 5y ago

https://meldmerge.org/ https://meldmerge.org/ might be what you are looking for. But if there are tools which can find duplicated files and directories which are named differently, I'd be interested in learning more about these as well.

staz 5y ago
meld can compare two directory if they have the same structure and name for files but can't really tell if I inside my foot folder, I have a folder root/X which is also present as root/category/b/X. (except all the file in it have been prepended by "foo" somehow) For tools which can find differently names files you have for examples: fslint (gui) or fdupes (cli)
- minhmeoke 5y ago
  That's true, meld is designed for comparing directories which already have similar structures. It seems like what we're looking for is content-addressable storage [1]. The theory behind it appears to be based on Merkle trees and cryptographic signatures [2]. IPFS already has an implementation [3] of this, and there are other implementations (borgbackup, restic, zpaq) listed in that link and in the content-addressable storage article. Disclaimer: I haven't used any of these yet, just found them a few minutes ago. [1] https://en.wikipedia.org/wiki/Content-addressable_storage https://en.wikipedia.org/wiki/Content-addressable_storage [2] https://gist.github.com/mafintosh/464bb8f1451f22c9e5c5 https://gist.github.com/mafintosh/464bb8f1451f22c9e5c5 [3] https://discuss.ipfs.io/t/ipfs-and-file-deduplication/4674 https://discuss.ipfs.io/t/ipfs-and-file-deduplication/4674