4 ms·

I made a command line utility called `dedup` a while back to do the same thing. It has a dry-run mode, will “intelligently” choose the best clone source, unders

I made a command line utility called `dedup` a while back to do the same thing. It has a dry-run mode, will “intelligently” choose the best clone source, understands hard links and other clones, preserves metadata, deals with HFS compressed files properly. It hasn’t destroyed any of my own data, but like any file system tool, use at your own risk. 0 - https://github.com/ttkb-oss/dedup https://github.com/ttkb-oss/dedup

LVB 2y ago
Just tried it, and it works well! I didn't realize the potential of this technique until I saw just how many dupes there were of certain types of files, especially in node_modules. It wasn't uncommon to see it replace 50 copies of some js file with one, and that was just in a specific subdirectory. I see it is "pre-release" and sort of low GH stars (== usage?), so I'm curious about the stability since this type of tool is relatively scary if buggy.
- jonhohle 2y ago
  I use it on my family photos, work documents, etc. and have not run into an issue that I haven’t added a test for. I didn’t commercialize it because I didn’t know what I didn’t know, but the utility does try to fail quickly if any of the file system operations fail (cloning, metadata duplication, atomic swaps, etc.). Whenever using it on something sensitive that I can’t back up first for whatever reason, I make checksum files and compare them afterwards. I’ve done this many times on hundreds of GB and haven’t seen corruption. Caveat emptor. There is one huge caveat I should add to the README - block corruption happens. Having a second copy of a file is a crude form of backup. Cloning causes all instances to use the same block, so if that one instance is corrupted, all clones are. That’s fine for software projects with generated files that can be rebuilt or checked out again, but introduces some risk for files that may not otherwise be replaceable. I keep multiple backups of all that stuff in hardware other than where I’m deduping, so I dedup with abandon. I’m a nobody with no audience. Maybe some attention here will get some users.
- ncann 2y ago
  Somewhat unrelated but I believe the dupe issue with node_modules is the main reason to use pnpn instead of npm - pnpm just uses a single global package repo on your machine and creates links inside node_modules as needed.
actinium226 2y ago
Wow, that's some excellent documentation. I was also really impressed that `make` ran basically instantly.
- jonhohle 2y ago
  Thanks! I love the documentation from FreeBSD and OpenBSD. Only having target one platform and only system libraries makes building simple.
jonhohle 2y ago
Replying to myself now that I've had a chance to try the scan, but not the deduplication. I work with disc images, program binaries, intermediate representations in a workspace that's 7.6G. A few notes: * By default it doesn't scan everything. It ignores all files but those in an allow list. The way the allow list is structured, it seems like Hyperspace needs to understand the content of a file. As an end user, I have no idea what the difference between a Text file and a Source Code file would be or how Hyperspace would know. Hyperscan only found 360MB to dedup. Allowing all files increased that to 842MB. * It doesn't scan files smaller than 100 KB by default. Disabling the size limit along with allowing all files increased that to 1.1GB * With all files and no size limit it scanned 67,309 of 68,874 files. `dedup` scans 67,426. * It says 29,522 files are eligible. Eligible means they can be deduped. `dedup` only fines 29,447. There are 76 already deduped files, which is an off-by-one, so I'm not sure what the difference is. * Scanning files in Hyperspace took around 50s vs `dedup` at 14s * It seems to scan the file system, then do a duplication calculation, then do the deduplication. I'm not sure why the first shouldn't be done together. I chose to queue any filesystem metadata as it was scanned and in parallel start calculating duplicates. The vast majority of the time files can be mismatched by size, which is available from `fts_read` "for free" while traversing the directory. * Hyperspace found 1.1GB to save, `dedup` finds 1.04GB and 882MB already saved (from previous deduping) * I'm not going to buy Hyperspace at this time, so I don't know how long it takes to dedup or if it preserves metadata or deals with strange files. `dedup` took 31s to scan and deduplicate. * After deduping with `dedup`, Hyperscan thinks there are still 2 files that can be deduped. * Hyperspace seems to understand it can't dedup files with multiple hard links, empty files, and some of the other things `dedup` also checks for. * I can't test ACLs or any other attribute preservation like that without paying. `strings` suggests those are handled. HFS Compression is a tricky edge case, but I haven't tested how Hyperspace's scan deals with those.
- karparov 2y ago
  I'm a little surprised that folks here are investing so much time into this app. It's closed source, only available for a non-obious amount, time-limited or subscription-based and lots of details of how it works are missing. With a FOSS project this would have been expected, but with a ShareWare-style model? Idk..
gurjeet 2y ago
Thank you for creating and sharing this utility. I ran it over my Postgres development directories that have almost identical files. It saved me about 1.7GB. The project doesn't have any license associated with it. If you don't mind, can you please license this project with a license of your choice. As a gesture of thanks, I have attempted to improve the installation step slightly and have created this pull request: https://github.com/ttkb-oss/dedup/pull/6 https://github.com/ttkb-oss/dedup/pull/6
- jonhohle 2y ago
  Thanks for the PR. Individual files were licensed, but I’ve also added a LICENSE file as well.
Recursing 2y ago
See the comments on https://news.ycombinator.com/item?id=38113396 https://news.ycombinator.com/item?id=38113396 for a list of alternatives. I used https://github.com/sahib/rmlint https://github.com/sahib/rmlint in the past and can't complain.

LVB 2y ago

Just tried it, and it works well! I didn't realize the potential of this technique until I saw just how many dupes there were of certain types of files, especially in node_modules. It wasn't uncommon to see it replace 50 copies of some js file with one, and that was just in a specific subdirectory. I see it is "pre-release" and sort of low GH stars (== usage?), so I'm curious about the stability since this type of tool is relatively scary if buggy.

jonhohle 2y ago
I use it on my family photos, work documents, etc. and have not run into an issue that I haven’t added a test for. I didn’t commercialize it because I didn’t know what I didn’t know, but the utility does try to fail quickly if any of the file system operations fail (cloning, metadata duplication, atomic swaps, etc.). Whenever using it on something sensitive that I can’t back up first for whatever reason, I make checksum files and compare them afterwards. I’ve done this many times on hundreds of GB and haven’t seen corruption. Caveat emptor. There is one huge caveat I should add to the README - block corruption happens. Having a second copy of a file is a crude form of backup. Cloning causes all instances to use the same block, so if that one instance is corrupted, all clones are. That’s fine for software projects with generated files that can be rebuilt or checked out again, but introduces some risk for files that may not otherwise be replaceable. I keep multiple backups of all that stuff in hardware other than where I’m deduping, so I dedup with abandon. I’m a nobody with no audience. Maybe some attention here will get some users.
ncann 2y ago
Somewhat unrelated but I believe the dupe issue with node_modules is the main reason to use pnpn instead of npm - pnpm just uses a single global package repo on your machine and creates links inside node_modules as needed.

actinium226 2y ago

Wow, that's some excellent documentation. I was also really impressed that `make` ran basically instantly.

jonhohle 2y ago
Thanks! I love the documentation from FreeBSD and OpenBSD. Only having target one platform and only system libraries makes building simple.

jonhohle 2y ago

Replying to myself now that I've had a chance to try the scan, but not the deduplication. I work with disc images, program binaries, intermediate representations in a workspace that's 7.6G. A few notes: * By default it doesn't scan everything. It ignores all files but those in an allow list. The way the allow list is structured, it seems like Hyperspace needs to understand the content of a file. As an end user, I have no idea what the difference between a Text file and a Source Code file would be or how Hyperspace would know. Hyperscan only found 360MB to dedup. Allowing all files increased that to 842MB. * It doesn't scan files smaller than 100 KB by default. Disabling the size limit along with allowing all files increased that to 1.1GB * With all files and no size limit it scanned 67,309 of 68,874 files. `dedup` scans 67,426. * It says 29,522 files are eligible. Eligible means they can be deduped. `dedup` only fines 29,447. There are 76 already deduped files, which is an off-by-one, so I'm not sure what the difference is. * Scanning files in Hyperspace took around 50s vs `dedup` at 14s * It seems to scan the file system, then do a duplication calculation, then do the deduplication. I'm not sure why the first shouldn't be done together. I chose to queue any filesystem metadata as it was scanned and in parallel start calculating duplicates. The vast majority of the time files can be mismatched by size, which is available from `fts_read` "for free" while traversing the directory. * Hyperspace found 1.1GB to save, `dedup` finds 1.04GB and 882MB already saved (from previous deduping) * I'm not going to buy Hyperspace at this time, so I don't know how long it takes to dedup or if it preserves metadata or deals with strange files. `dedup` took 31s to scan and deduplicate. * After deduping with `dedup`, Hyperscan thinks there are still 2 files that can be deduped. * Hyperspace seems to understand it can't dedup files with multiple hard links, empty files, and some of the other things `dedup` also checks for. * I can't test ACLs or any other attribute preservation like that without paying. `strings` suggests those are handled. HFS Compression is a tricky edge case, but I haven't tested how Hyperspace's scan deals with those.

karparov 2y ago
I'm a little surprised that folks here are investing so much time into this app. It's closed source, only available for a non-obious amount, time-limited or subscription-based and lots of details of how it works are missing. With a FOSS project this would have been expected, but with a ShareWare-style model? Idk..

gurjeet 2y ago

Thank you for creating and sharing this utility. I ran it over my Postgres development directories that have almost identical files. It saved me about 1.7GB. The project doesn't have any license associated with it. If you don't mind, can you please license this project with a license of your choice. As a gesture of thanks, I have attempted to improve the installation step slightly and have created this pull request: https://github.com/ttkb-oss/dedup/pull/6 https://github.com/ttkb-oss/dedup/pull/6

jonhohle 2y ago
Thanks for the PR. Individual files were licensed, but I’ve also added a LICENSE file as well.

Recursing 2y ago

See the comments on https://news.ycombinator.com/item?id=38113396 https://news.ycombinator.com/item?id=38113396 for a list of alternatives. I used https://github.com/sahib/rmlint https://github.com/sahib/rmlint in the past and can't complain.