4 ms·
Does anybody know of a good deduplication script / tool that runs on every filesystem and can be used as a nightly cronjob?
by softwarelimits 10y ago
Does anybody know of a good deduplication script / tool that runs on every filesystem and can be used as a nightly cronjob?
- kruhft 10y agoHere's a quick one if your filesystem supports hard links: #!/bin/ksh typeset -A hashes find . -type f -exec shasum -a 512 {} \; | while read hash filename; do if [[ -z ${hashes[$hash]} ]]; then hashes[$hash]="$filename" else echo ln -f "'${hashes[$hash]}'" "'$filename'" fi done Pipe to /bin/sh if you're happy with the results.
- rsanders 10y agoGood answer, but unless you mark the files read-only/immutable, this ought to have big red flags all over it. Userland just can't do this safely in the general case.
- kruhft 10y agoGood point. This is for data that's not in use.
- khc 10y agoDo you actually use this? Seems at the minimum you want to use -xdev with find and do something better than ln -f.
- kruhft 10y agoYes, I forgot to put in -xdev, but what's wrong with ln -f?
- nabla9 10y agofdupes -rL /dir see also jdupes fork of fdupes https://github.com/jbruchon/jdupes https://github.com/jbruchon/jdupes https://github.com/adrianlopezroche/fdupes https://github.com/adrianlopezroche/fdupes