3 ms·

Off topic, any tool to deduplicate files across different external Hard disks? Over the years I made multiple copies of my laptop HDD to different external HDD

• nikisweeting 2y ago
  How would you want the duplicates resolved? Just reported in some interface or would you want the duplicates deleted off some machines automatically? There are a few different ways you could solve it but it depends on what final outcome you need.
  • girishso 2y ago
    Just reporting in some plain text format so I can manually delete the duplicates, or create some script to delete. I can't have like 10 external HDDs attached at the same time, so the tool needs to dump details (hashes?) somewhere on Mac HDD, and compare against those to find the duplicates.
    • nikisweeting 2y ago
      Here you go: cd /path/to/drive find . -type f -exec sha256sum {} + | sed -E 's/^([^ ]+) \./\1,/' >> ~/all_hashes.txt Run that for each drive, then when you're done run: sort ~/all_hashes.txt > ~/sorted_hashes.txt awk -F, 'NR==1{print;next} {print $0 | "sort | uniq -w64 -D"}' ~/sorted_hashes.txt > ~/non_unique_hashes.txt The output in ~/non_unique_hashes.txt will contain only the non-unique hashes that appear on more than one path.
• UltraSane 2y ago
  dupeGuru works pretty well.