3 ms·

Funny to see this here, recently I decided to clean up my personal NAS that had 60 million files and 17 TB used. I tried various tools, but this NAS was the bac

by kator 3y ago

Funny to see this here, recently I decided to clean up my personal NAS that had 60 million files and 17 TB used. I tried various tools, but this NAS was the backup target for many of my machines over a decade or more of time. I finally gave up and wrote a multi-threaded c++ program to "inventory" every file and then I imported that into sqlite and started building out scripts to do bulk actions, set a flag on for files to delete and some scripting to show the files that would be deleted etc. It was quite a project and took me "spare" time of a month to trim. On a related note in 2019 Apple destroyed all my music copies in one of the itunes updates, and part of this process I gathered all forms of media into various zfs datasets and de-duped and managed to find all my mp4's from the '90s lol. It's amazing how much digital garbage we can accumulate, I often wonder what my kids will do when I pass, will they try and sift through all of it, or just toss it in a closet somewhere to be discovered in 1,000 years by some digital archeologist?

mongol 3y ago
My bet: They will toss it in a closet, but it will not be discovered in 1000 years. It will be gone before then. If you want future generations to see your photos, print them out. People will have too much else to worry about than decrypt old storage media.
- deleted 3y ago
  [deleted]
- theamk 3y ago
  My 30 years paper photos are mostly lost now.. too many house moves, too much physical stuff. On the other hand, my high school projects from decades ago, as well as my first digital photos, are still with me, somewhere in the "archive" folder. They are only a few megabytes after all, an insignificant amount of space for a long time. It's easier to keep copying entire "archive/" to new machines rather than sort it. And for photos in particular, I know there is an archive on each machine I owned, and on many external drives, at least a dozen copies. I'd say the chances of them surviving 1000 years in are pretty high, much higher than physical photos.
  - actionfromafar 3y ago
    But will the digital be discovered by anyone? I still believe the physical photos have a larger chance of actually seeing the light of day. One with is to print albums yearly and send out to friends and family. Some of them will probably survive for a long time. Nothing stops you from hanging on to your digital archive aswell.
WXLCKNO 3y ago
I figure our human level general AI assistant (if it doesn't kill us) will sort through my mess of backups in a nanosecond in about ten years, so I'm not gonna start worrying until then lol.
TacticalCoder 3y ago
> I finally gave up and wrote a multi-threaded c++ program to "inventory" every file and then I imported that into sqlite and started building out scripts to do bulk actions, set a flag on for files to delete and some scripting to show the files that would be deleted etc. I did something not dissimilar. Basically at some point, years ago, I got fed up of deleting for the ten times that one picture I already had deleted in the past. So I created a little database of files that are "tagged", based on their size first, then on their hash. The "tag" is basically: "File with this size and hash can always be deleted, no matter what". Or "File with this size and hash and this name can always be renamed to this-or-that-name". I've also got a few rules: "Empty directory with that name can always be deleted". Then I launch my "berzerker" on HDDs/SDDs/SD cards and whatnots I have on shelves and get rid of thousands of files. It always feel insanely good. When it comes to old CDs/DVDs (I know with your 17TB these aren't going to be a thing for you but I still have CDs and DVDs from the 90s and 2000s and once in a rare while I'll still burn a DVD), I copy them to my computer, run the berzerker, and burn a new DVD. Then when all is good and well, I physically destroy the old CD/DVD. I also transcode movies (say family movies) for which I don't care that much about the quality to a much smaller size (say to x265) and then immediately tag the old file as "can always be deleted" in my database. Also I haven't inventoried every single file. But I'll typically use "jdupes" (much faster than "fdupes") to find the worst offenders: biggest files appearing often are the ones I tend to inventory first. What's really great with such inventories/tags is that it's something you basically do once. Once a file (and its size + hash) is tagged for deletion, that file has lost the war. It's over: my berzerker shall regularly run and it shall find it. And the file is gone. The files tagged for deletion are "dead files walking". > It's amazing how much digital garbage we can accumulate, I often wonder what my kids will do when I pass, will they try and sift through all of it, or just toss it in a closet somewhere to be discovered in 1,000 years by some digital archeologist? They'll use a dumb but convenient "AI": find me all the pictures, movies and writings from my dad and sort them, de-dupe them, rate them and show all that to me chronologically and nicely sorted. At times I'm wondering why I even bother cleaning that mess. I could just wait and have some software sort it for me in 20 years (or if I'm dead by then, my kid shall do it). Now that said: we're not the first ones to come up with our own DB of files that can always be deleted, always be renamed, etc. I'm kinda surprised there's not yet some open-source software out there doing just that. Mine is not shippable as it's Clojure code I run directly from the REPL. P.S: for the hash, to be really quick, I use Blake3 (which may not suit everybody as it's kinda recent but, darn, is Blake3 quick -- and automatically running in parallel).

by kator 3y ago

mongol 3y ago

My bet: They will toss it in a closet, but it will not be discovered in 1000 years. It will be gone before then. If you want future generations to see your photos, print them out. People will have too much else to worry about than decrypt old storage media.

deleted 3y ago
[deleted]
theamk 3y ago
My 30 years paper photos are mostly lost now.. too many house moves, too much physical stuff. On the other hand, my high school projects from decades ago, as well as my first digital photos, are still with me, somewhere in the "archive" folder. They are only a few megabytes after all, an insignificant amount of space for a long time. It's easier to keep copying entire "archive/" to new machines rather than sort it. And for photos in particular, I know there is an archive on each machine I owned, and on many external drives, at least a dozen copies. I'd say the chances of them surviving 1000 years in are pretty high, much higher than physical photos.
- actionfromafar 3y ago
  But will the digital be discovered by anyone? I still believe the physical photos have a larger chance of actually seeing the light of day. One with is to print albums yearly and send out to friends and family. Some of them will probably survive for a long time. Nothing stops you from hanging on to your digital archive aswell.

WXLCKNO 3y ago

I figure our human level general AI assistant (if it doesn't kill us) will sort through my mess of backups in a nanosecond in about ten years, so I'm not gonna start worrying until then lol.

TacticalCoder 3y ago

> I finally gave up and wrote a multi-threaded c++ program to "inventory" every file and then I imported that into sqlite and started building out scripts to do bulk actions, set a flag on for files to delete and some scripting to show the files that would be deleted etc. I did something not dissimilar. Basically at some point, years ago, I got fed up of deleting for the ten times that one picture I already had deleted in the past. So I created a little database of files that are "tagged", based on their size first, then on their hash. The "tag" is basically: "File with this size and hash can always be deleted, no matter what". Or "File with this size and hash and this name can always be renamed to this-or-that-name". I've also got a few rules: "Empty directory with that name can always be deleted". Then I launch my "berzerker" on HDDs/SDDs/SD cards and whatnots I have on shelves and get rid of thousands of files. It always feel insanely good. When it comes to old CDs/DVDs (I know with your 17TB these aren't going to be a thing for you but I still have CDs and DVDs from the 90s and 2000s and once in a rare while I'll still burn a DVD), I copy them to my computer, run the berzerker, and burn a new DVD. Then when all is good and well, I physically destroy the old CD/DVD. I also transcode movies (say family movies) for which I don't care that much about the quality to a much smaller size (say to x265) and then immediately tag the old file as "can always be deleted" in my database. Also I haven't inventoried every single file. But I'll typically use "jdupes" (much faster than "fdupes") to find the worst offenders: biggest files appearing often are the ones I tend to inventory first. What's really great with such inventories/tags is that it's something you basically do once. Once a file (and its size + hash) is tagged for deletion, that file has lost the war. It's over: my berzerker shall regularly run and it shall find it. And the file is gone. The files tagged for deletion are "dead files walking". > It's amazing how much digital garbage we can accumulate, I often wonder what my kids will do when I pass, will they try and sift through all of it, or just toss it in a closet somewhere to be discovered in 1,000 years by some digital archeologist? They'll use a dumb but convenient "AI": find me all the pictures, movies and writings from my dad and sort them, de-dupe them, rate them and show all that to me chronologically and nicely sorted. At times I'm wondering why I even bother cleaning that mess. I could just wait and have some software sort it for me in 20 years (or if I'm dead by then, my kid shall do it). Now that said: we're not the first ones to come up with our own DB of files that can always be deleted, always be renamed, etc. I'm kinda surprised there's not yet some open-source software out there doing just that. Mine is not shippable as it's Clojure code I run directly from the REPL. P.S: for the hash, to be really quick, I use Blake3 (which may not suit everybody as it's kinda recent but, darn, is Blake3 quick -- and automatically running in parallel).