4 ms·

In my 40+ years of computing dating back to an Apple II, I've used storage as varied as 140K floppy disks, to EMC RAID arrays attached via SANs to Sun Enterpris

by js2 4y ago

In my 40+ years of computing dating back to an Apple II, I've used storage as varied as 140K floppy disks, to EMC RAID arrays attached via SANs to Sun Enterprise 4500s, to NetApp NFS clusters, to a TrueNAS box in my basement with terabytes of space. I've been a sysadmin managing SunOS and Linux systems at scale. I've had personal machines running a variety of OSs: FreeBSD, Linux, Windows, Mac OS, OS X, to say nothing off all the different filesystems involved across those OSes. In all that time, I can't think of a single time I've ever bothered to de-duplicate files. Once upon a time, you used to occasionally need to defragment some file systems. But it's been more than two decades since I've even thought about defragmenting. Every modern filesystem just handles that automatically. What even is the use case where folks end up with enough duplicate files that it's worth worrying about?

dghlsakjg 4y ago
I can think of exactly one case! In college there was, for a glorious, brief, time, a program called mytunes. It would scan the network for iTunes instances that had the network sharing turned on. Then it allowed you to copy over entire iTunes libraries (iTunes only let you play songs from other users). I would log on to the dorm network which normally had dozens of iTunes instances running at a go, and just copy everything to my, at the time, massive 160gb drive. Then you would dedupe. That’s the last time I thought about deduping
jeroenhd 4y ago
I never really bothered myself, but when one of my SSDs was running low on storage, I decided to run a deduplicator over some folders. It turns out the node_modules folders I've collected over the years had several gigabytes of duplication. The same was true to a lesser extend for Python code (because the dependency folders and venvs were a lot smaller).
Strom 4y ago
I'll give you two use cases, one for non-technical and one for technical people. Non-technical person will copy-paste a folder of documents to make a backup before editing a file. A crude form of version control. I've seen this method lead to the consumption of hundreds of gigabytes for duplicate content, because the folders contain a lot of auxiliary data like PDFs and JPEGs. Technical person will run npm install and get a massive node_modules directory that is almost identical to the node_modules directory in another project.
- BrandoElFollito 4y ago
  For the second one I would love to have some kind of centralized node_modules used across my projects. I am pretty sure that a large part is duplicated, just for the fact that I use the same framework everywhere
jamesu 4y ago
One time I had to consolidate a late family members personal files spread across several types of media which ended up totalling several TBs. None of it was organized, so there were duplicates everywhere. Probably only 20% of the data was non-duplicated stuff I genuinely wanted to keep.
firstlink 4y ago
> Every modern filesystem just handles that automatically. Do they? I always assumed they just try to avoid fragments but ignore them once created, and rely on bigger faster disks to hide any problems this may cause.

by js2 4y ago

dghlsakjg 4y ago

I can think of exactly one case! In college there was, for a glorious, brief, time, a program called mytunes. It would scan the network for iTunes instances that had the network sharing turned on. Then it allowed you to copy over entire iTunes libraries (iTunes only let you play songs from other users). I would log on to the dorm network which normally had dozens of iTunes instances running at a go, and just copy everything to my, at the time, massive 160gb drive. Then you would dedupe. That’s the last time I thought about deduping

jeroenhd 4y ago

I never really bothered myself, but when one of my SSDs was running low on storage, I decided to run a deduplicator over some folders. It turns out the node_modules folders I've collected over the years had several gigabytes of duplication. The same was true to a lesser extend for Python code (because the dependency folders and venvs were a lot smaller).

Strom 4y ago

I'll give you two use cases, one for non-technical and one for technical people. Non-technical person will copy-paste a folder of documents to make a backup before editing a file. A crude form of version control. I've seen this method lead to the consumption of hundreds of gigabytes for duplicate content, because the folders contain a lot of auxiliary data like PDFs and JPEGs. Technical person will run npm install and get a massive node_modules directory that is almost identical to the node_modules directory in another project.

BrandoElFollito 4y ago
For the second one I would love to have some kind of centralized node_modules used across my projects. I am pretty sure that a large part is duplicated, just for the fact that I use the same framework everywhere

jamesu 4y ago

One time I had to consolidate a late family members personal files spread across several types of media which ended up totalling several TBs. None of it was organized, so there were duplicates everywhere. Probably only 20% of the data was non-duplicated stuff I genuinely wanted to keep.

firstlink 4y ago

> Every modern filesystem just handles that automatically. Do they? I always assumed they just try to avoid fragments but ignore them once created, and rely on bigger faster disks to hide any problems this may cause.