7 ms·

As far as I know, the standard tool for this is rdfind. This new tool claims to be "blazingly fast", so it should provide something to show it. Ideally a compar

As far as I know, the standard tool for this is rdfind. This new tool claims to be "blazingly fast", so it should provide something to show it. Ideally a comparison with rdfind, but even a basic benchmark would make it less dubious. https://github.com/pauldreik/rdfind https://github.com/pauldreik/rdfind But the main problem is not the suspicious performance, it's the lack of explanation. The tool is supposed to "find duplicate files (photos, videos, music, documents)". Does it mean it is restricted to some file types? Does it find identical photos with different metadata to be duplicates? Compare this with rdfind which clearly describes what it does, provides a summary of its algorithm, and even mentions alternatives. Overall, it may be a fine toy/hobby project (3 commits only, 3 months ago), I didn't read the code (except for finding the command-line options). I don't get why it got so much attention.

justinsaccount 5y ago
Yeah, this tool does not appear to be very good, especially compared to established alternatives. It initially groups files that have the "same extension and same size", so you're out of luck if you have two copies named foo.jpg and foo.jpeg. Then, it cheats by computing a crc32 (!) of the beginning, middle, and end bytes of the file and groups together files that have the same crc32. So, it'll mostly work, but miss a lot of duplicates, and potentially flag different files as duplicates.
- brundolf 5y ago
  Sounds like Apple's new photo scanning system
- pdimitar 5y ago
  What other alternative tools do you know about? I'm curious and want to create a collection of such tools.
  - vlovich123 5y ago
    See fclones mentioned elsewhere in this thread. Seems like a better & faster tool from a quick glance.
artemisart 5y ago
See also fclones (focuses on performance, has benchmarks https://github.com/pkolaczk/fclones https://github.com/pkolaczk/fclones). I didn't know about rdfind but thought the standard was fdupes https://github.com/adrianlopezroche/fdupes https://github.com/adrianlopezroche/fdupes, which is as fast (or slow) as rdfind according to fclones (and fclones is much faster).
- justinsaccount 5y ago
  afaik fdupes is super slow because it checksums entire files in order to find duplicates. This causes a ton of unnecessary IO if you have a lot of size collisions. The efficient way to do things is to just read files in parallel and break once they diverge. Basically how `cmp` works.
  - pvaldes 5y ago
    Rdfind is the logical evolution of fdupes. Not only faster but also more clever. This does not mean that fdupes is a bad tool at all, but rdfind can do things that fdupes can't. Example. For fileX.txt being X=1 to 10000 you have three copies of each archive in: mydir/file-X.txt mydir/subdir/file-X.txt and mydir/subdir/copy/file-X.txt. Fdupes would delete random files in mydir/, mydir/subdir/ and mydir/subdir/copy/. You would end with the remaining files scattered by all the directory tree. A mess with three incomplete copies. Rdfind correctly guess that what most people would want is to remove entirely all files in two of the directories and keep one copy (files and tree-dir) intact. So it wipes the inner subdirs in a predictable way and keeps the outer dir intact. This is a terrific feature able to disentangle one tree directory cloned and nested into the original copy without distroying it, like in this case: a/b/c/d/files00X.txt a/b/a/b/c/d/files00X.txt
    - pkolaczk 5y ago
      Instead of trying to be clever, fclones gives the user a choice. It can select files to remove by their nesting level, creation/update/access time or glob expression (include/exclude). It also allows the user to modify the list of found files before deleting.
code_biologist 5y ago
I've found rmlint to be another very good tool in this space: https://github.com/sahib/rmlint https://github.com/sahib/rmlint
diskzero 5y ago
I think we need a lookup table of marketing speech to real-world performance metrics. Blazingly fast has been showing up a lot lately. The cynical side of me wants to know what features and safety checks a "blazingly fast" tool has not implemented that the older "glacially slow" tool it is replacing ended up implementing after all the edge conditions were uncovered.
nieve 5y ago
Do you know of any tool that does a good job of finding files that differ only in their metadata or even better can use a perceptual hash to find possible matches? Geeqie's find duplicates seems to do the latter, but afaict you can't run that function from the command line.

justinsaccount 5y ago

Yeah, this tool does not appear to be very good, especially compared to established alternatives. It initially groups files that have the "same extension and same size", so you're out of luck if you have two copies named foo.jpg and foo.jpeg. Then, it cheats by computing a crc32 (!) of the beginning, middle, and end bytes of the file and groups together files that have the same crc32. So, it'll mostly work, but miss a lot of duplicates, and potentially flag different files as duplicates.

brundolf 5y ago
Sounds like Apple's new photo scanning system
pdimitar 5y ago
What other alternative tools do you know about? I'm curious and want to create a collection of such tools.
- vlovich123 5y ago
  See fclones mentioned elsewhere in this thread. Seems like a better & faster tool from a quick glance.

artemisart 5y ago

See also fclones (focuses on performance, has benchmarks https://github.com/pkolaczk/fclones https://github.com/pkolaczk/fclones). I didn't know about rdfind but thought the standard was fdupes https://github.com/adrianlopezroche/fdupes https://github.com/adrianlopezroche/fdupes, which is as fast (or slow) as rdfind according to fclones (and fclones is much faster).

justinsaccount 5y ago
afaik fdupes is super slow because it checksums entire files in order to find duplicates. This causes a ton of unnecessary IO if you have a lot of size collisions. The efficient way to do things is to just read files in parallel and break once they diverge. Basically how `cmp` works.
- pvaldes 5y ago
  Rdfind is the logical evolution of fdupes. Not only faster but also more clever. This does not mean that fdupes is a bad tool at all, but rdfind can do things that fdupes can't. Example. For fileX.txt being X=1 to 10000 you have three copies of each archive in: mydir/file-X.txt mydir/subdir/file-X.txt and mydir/subdir/copy/file-X.txt. Fdupes would delete random files in mydir/, mydir/subdir/ and mydir/subdir/copy/. You would end with the remaining files scattered by all the directory tree. A mess with three incomplete copies. Rdfind correctly guess that what most people would want is to remove entirely all files in two of the directories and keep one copy (files and tree-dir) intact. So it wipes the inner subdirs in a predictable way and keeps the outer dir intact. This is a terrific feature able to disentangle one tree directory cloned and nested into the original copy without distroying it, like in this case: a/b/c/d/files00X.txt a/b/a/b/c/d/files00X.txt
  - pkolaczk 5y ago
    Instead of trying to be clever, fclones gives the user a choice. It can select files to remove by their nesting level, creation/update/access time or glob expression (include/exclude). It also allows the user to modify the list of found files before deleting.

code_biologist 5y ago

I've found rmlint to be another very good tool in this space: https://github.com/sahib/rmlint https://github.com/sahib/rmlint

diskzero 5y ago

I think we need a lookup table of marketing speech to real-world performance metrics. Blazingly fast has been showing up a lot lately. The cynical side of me wants to know what features and safety checks a "blazingly fast" tool has not implemented that the older "glacially slow" tool it is replacing ended up implementing after all the edge conditions were uncovered.

nieve 5y ago

Do you know of any tool that does a good job of finding files that differ only in their metadata or even better can use a perceptual hash to find possible matches? Geeqie's find duplicates seems to do the latter, but afaict you can't run that function from the command line.