6 ms·
Fclones – an efficient duplicate file finder and remover
- ZeroGravitas 5y agoCoincidentally I was looking through the list of apps that do this the other day, I think I looked at all the other ones they list as competitors. My particular use case involved images that might be very near duplicates (screenshots of the same web page), which some of the tools cover, though it feels like a slightly seperate task from the exact bit duplicates, so not all do it. One interesting one I found that wasn't listed in the Readme (edit, it was there, I just didn't spot it on first read through) was: https://github.com/kornelski/dupe-krill https://github.com/kornelski/dupe-krill Which had some notes about their use of BTreehashes to progressively compare files. Not sure how much difference it makes in practice but sounded elegant.
- mastax 5y agoI just used fclones the other day on a 20TB archive of mostly videos at work. Saved 1.8TB by hard linking the duplicates. I like how fclones makes it easy to have a two stage process where you scan for duplicates and populate a text file, that you can then pipe in to take action. You can inspect the paths to make sure it's okay and decide what to do.
- pavel_lishin 5y agoThe last time I needed to do this - to gather a bunch of files from old hard drives, and organize them so I could back them up - I ended up using fdupes. This looks very cool, and would probably have been a bit easier.
- metadat 5y ago(Fclones is written in Rust!) How does it compare to fdupes in terms of performance?
- pkolaczk 5y agoAuthor of fclones here. See the Benchmarks section at the end of the README. Tl;dr - it is many times faster. The major advantages are due to: - parallel scanning of files on SSD - accessing files in the order they are physically stored on some Linux filesystems that support it, and by inode IDs on other (less efficient but still a lot better) - makes a huge difference on spinning drives - selecting the optimal access strategy based on the type of the drive (SSD vs HDD) - optionally using a hash cache
- BeefWellington 5y agoIf I'm reading this right, you're ultimately calling out to the shell or command prompt to move, delete, and symlink/hardlink files. Is there a reason to take this approach rather than use language built-ins or platform functions?
- pkolaczk 5y agoNo, it does not call out to the shell. It uses the built-in functions.
- BeefWellington 5y agoAh, I misread the Dedup function to_shell_str. Apologies.
- torstenvl 5y agoThis looks awesome, I've been looking at something to do this, specifically to hard link duplicates across parallel directory structures.