7 ms·
Should you remove duplicate files?
- iamnotsure 4y agoI would prefer not to.
- ggm 4y agoIf your fs implements block dedupe, no need. But, it's expensive to implement and from what I see no longer recommended on ZFS. If your fs implements hard links, no need inside one mount point. But you can't hard link cross mount. Copies can be good or bad. It depends.
- ajsnigrutin 4y agoDedup on FS works, but hard links are useful only if you (and everyone else using the PC) is aware of the hard link (and never forgets about that). Otherwise it's "I can just mess with this file, I have a copy in that other folder for backup".. until you notice it's not an actual copy.
- deleted 4y ago[deleted]
- matheusmoreira 4y agoIt's better to assume all files are potentially links that must be explicitly copied before any destructive operation. Software should be designed to always produce new copies by default and overwrite files only when explicitly requested.
- counttheforks 4y agoI'd like copy-on-write hardlinks, or deduplicated blocks, that are made automatically on a schedule as to not incur the constant overhead of dedup, but still reap some of the benefits for archival.
- Freaky 4y agoReflinks are basically CoW hardlinks — supported on Linux by btrfs, XFS, and I've seen some effort to bring it to ZFS. DragonFlyBSD's HAMMER filesystem supports on-demand/scheduled dedup using the hammer(8) command, with configurable memory and runtime limits.
- the8472 4y agohttps://manpages.debian.org/testing/duperemove/duperemove.8.en.html https://manpages.debian.org/testing/duperemove/duperemove.8....
- counttheforks 4y agoSweet, thanks.
- tatersolid 4y agoNTFS deduplication works this way on Windows server since 2012, but I don’t think any of the Linux NTFS drivers support NTFS dedupe correctly.
- soneil 4y agoThe behaviour is very different to hardlinks. The clone is a new file that uses the same blocks. When you modify the clone, writes are written to new blocks rather than the old ones. So if I have a disk image that takes 1,000 blocks, and clone it - I now have two images that total 1,000 blocks. And if I modify one block in one image, I now have two different images that total 1,001 blocks. With a hardlink the modification would affect both images. The way I visualise the difference, is that a directory entry points to a file entry, and a file entry points to a list of blocks. A hardlink is a new directory entry that points to an existing file entry. a linked clone is a new file entry that points to the same list of blocks.
- the8472 4y ago> But, it's expensive to implement and from what I see no longer recommended on ZFS. Btrfs (and more recently also XFS) supports on-demand, batch dedup which doesn't have any of the downsides of ZFS's always-active dedup. This is an implementation issue, not a categorical problem of block (or extent) dedup.
- matheusmoreira 4y agoBtrfs is like an almost perfect file system. If only it had proper stable support for software RAID. It's weird that it still doesn't have it, I thought enterprises cared about that.
- nicolaslem 4y agoIt has proper stable support for many kinds of software RAID: https://btrfs.readthedocs.io/en/latest/Status.html https://btrfs.readthedocs.io/en/latest/Status.html
- matheusmoreira 4y agoExcept RAID6 which is the one I want. From your link: https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid56-status-and-recommended-practices https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid5... Also, reading stuff like "mostly OK" in a file system doesn't inspire confidence.
- kevincox 4y agoThe ideal situation here would probably be to update any identical but not reflinked files to be reflinks. But if there is no API like you suggested then it may not be easy. Also ideally there would be an API to do this atomically please share the storage of these files assuming they are the same) If Time Machine is tracking file contents for deduplication it would be really cool if it did this to the source files whenever it noticed that they were identical.
- lizknope 4y agoThere are many programs out there that will find duplicates files. I use this one. It gives you the option of replacing the duplicates with either hard or symbolic links. https://github.com/qarmin/czkawka https://github.com/qarmin/czkawka
- ukuina 4y agoI have been looking for something like this, thanks for sharing! From the project: > This name is not as bad as it seems, because I was also thinking about using words like żółć, gżegżółka or żołądź, but I gave up on these ideas because they contained Polish characters, *which would cause difficulty in searching for the project.* Really dodged a bullet there.
- Eisenstein 4y agoRemember how it is pronounced and you can remember the spelling: Czkawka == Ch-kah-v-kah. Cz is 'ch' like Czech, and w is v. It sounds like a sneeze when you say it.
- bloaf 4y agoI've used fdupes and refund for exact duplicate finding. They work shockingly well on large libraries, I've managed to check millions of files consuming several TBs in a matter of hours.
- Szpadel 4y agoI use btrfs + bees (https://github.com/Zygo/bees https://github.com/Zygo/bees) for such cases
- binkHN 4y agoBack in the day I used to do this on file servers at the block level. Was valuable as storage was more limited, IT would often have duplicated large deployment images and users would often have duplicate copies of the same files. Multiple one user by ~1000 and you have significant space savings.
- vinaypai 4y ago"Finding and removing duplicate files has long been one of the mainstays of Mac housekeeping." What are Mac users doing that makes finding and removing duplicate files such a regular task? I don't use a Mac, but I can honestly say I've never given second thought to duplicate files. I'm sure there are some here and there but my hard drive isn't filling up with duplicates of large files.
- nopakos 4y agoSpace is precious! Soldered SSDs can't be replaced, so we are stuck with the size we could afford at the time we bought our MacBooks.
- graypegg 4y agoIf I had to guess, it might have to do with the Mac standard of “save on close window” which is a vestige of an old apple desktop metaphor where every window represents a file somewhere. So you end up with a lot of duplicates from either trying to edit a read only document in some apps (which spawns an editable copy) or accidentally hitting duplicate (generate a copy of the current file in the same directory) which is the mac equivalent of “Save As”. It’s on you the user to move the new file where you want it.
- dahart 4y agoThere’s nothing special about Macs or Mac users, this article is just in a Mac-focused blog, so it’s not implying anything about other OSes. Deduping is a mainstay of computer housekeeping on any OS, for some people, though maybe not you. There are lots of workflows that end up with duplicates, but certainly not everyone uses them. Here are just a few ways I’ve ended up with enough dupes to want to dedupe, which has happened on Windows and Linux for me as frequently as on Mac: Backing up mobile phones, using version control for multiple repositories that share files, losing access to repository servers and archiving or backup up the files, forking an old archived folder and making modifications to it that I want to keep, not to mention homegrown schemes & scripts for archiving and backing up when I either don’t want to pay for backup software or can’t even find anything appropriate to all my needs (cross platform, saving to a NAS, shared with my family, etc.)
- ComodoHacker 4y ago>Finally, why don’t some housekeeping utilities recognise clones? That’s because there’s no official way to tell them apart. Is it intentional?
- tacostakohashi 4y agoIf there were any official way to tell them apart, would they still be clones?
- dahart 4y agoI used to dedupe a lot to save space, on any OS, Mac, Windows, and Linux. But one of the main reasons I end up with many dupes is various archiving and backup schemes I use, and I’ve realized deduplicating reduces backup safety. If the source file gets somehow corrupted for any reason, all copies are lost. I’m sure a deduping file system is slightly better here than a deduping utility and/or my own scripts, but still having only one copy of the bits is more precarious than saving multiple copies.
- js2 4y agoIn my 40+ years of computing dating back to an Apple II, I've used storage as varied as 140K floppy disks, to EMC RAID arrays attached via SANs to Sun Enterprise 4500s, to NetApp NFS clusters, to a TrueNAS box in my basement with terabytes of space. I've been a sysadmin managing SunOS and Linux systems at scale. I've had personal machines running a variety of OSs: FreeBSD, Linux, Windows, Mac OS, OS X, to say nothing off all the different filesystems involved across those OSes. In all that time, I can't think of a single time I've ever bothered to de-duplicate files. Once upon a time, you used to occasionally need to defragment some file systems. But it's been more than two decades since I've even thought about defragmenting. Every modern filesystem just handles that automatically. What even is the use case where folks end up with enough duplicate files that it's worth worrying about?
- dghlsakjg 4y agoI can think of exactly one case! In college there was, for a glorious, brief, time, a program called mytunes. It would scan the network for iTunes instances that had the network sharing turned on. Then it allowed you to copy over entire iTunes libraries (iTunes only let you play songs from other users). I would log on to the dorm network which normally had dozens of iTunes instances running at a go, and just copy everything to my, at the time, massive 160gb drive. Then you would dedupe. That’s the last time I thought about deduping
- jeroenhd 4y agoI never really bothered myself, but when one of my SSDs was running low on storage, I decided to run a deduplicator over some folders. It turns out the node_modules folders I've collected over the years had several gigabytes of duplication. The same was true to a lesser extend for Python code (because the dependency folders and venvs were a lot smaller).
- Strom 4y agoI'll give you two use cases, one for non-technical and one for technical people. Non-technical person will copy-paste a folder of documents to make a backup before editing a file. A crude form of version control. I've seen this method lead to the consumption of hundreds of gigabytes for duplicate content, because the folders contain a lot of auxiliary data like PDFs and JPEGs. Technical person will run npm install and get a massive node_modules directory that is almost identical to the node_modules directory in another project.
- didgetmaster 4y agoSo if you change one byte in a clone, then it will just update a single block in the changed file while leaving all the other clone's data streams unchanged. But what happens if you add (or delete) a byte near the front of a 10GB file? Does that cause the entire data stream to be written, since the rest of the data must be shifted up or down?
- jerf 4y agoYes, but precisely because of that there is a bit of resistance to people using that operation on large files, so it is not a common thing. It wasn't that long ago that rewriting a 10GB file from top to bottom could easily take over three minutes (50MB/s write speeds to a spinning rust hard drive = 200 seconds, and if we account for also reading the file it will be even slower), so people tend to learn to program without asking for that operation. While there are many people who weirdly (to me at least) grotesquely overestimate the value of deduplication, it is legitimately useful because we tend not to want to completely rewrite large files very often like that, so even modifications to that file are very likely to either be appends or modifications of something in the middle without shifting the entire file.
- CharlesW 4y agoFor folks who know they have duplicate photos, but also have a photo library large enough that it's hard to manually de-dupe them, I can strongly recommend PhotoSweeper (no affiliation). https://overmacs.com/ https://overmacs.com/ What's unique about PhotoSweeper is that it gives you deep control over what you mean by "dupe" and supports pro image formats like RAW and DNG. For example, I was able to automatically de-dupe byte-for-byte equivalents, then manually evaluate dupes with the same image data but slightly different metadata. Of course, it can do things like find dupes at different resolutions and keep the highest-quality version, etc. It saved me a ton of time.
- euroderf 4y agoI would like to know if there are similar utilities for audio files.