30 ms·
I asked chat GPT-4 to write me a rust program that recurses all files in my external drives, generates a xxhash of each file, then inserts it into an RDF graph.
by JimmyRuska 3y ago
I asked chat GPT-4 to write me a rust program that recurses all files in my external drives, generates a xxhash of each file, then inserts it into an RDF graph.
That way I can scan for any files I don't have centralized before wiping out each older drive, and query for specific cases.
It's also nice to recurse the old unimportant video files and re-encode them with ffmpeg vp9 or av1, opus, so they don't take up as much space. Get rid of those raw video recordings and xvid/divx codec videos.
Re-encode all your old 7z, .zip, bz2, .tar.gz into zstd compressed files. It can also save a ton of space.
Buy manufacturer recertified HDDs, you can get like 18TB for $190 or so. Buy a few and put them in a truenas server. Hoard all the data you want, forever.
- chaxor 3y agoI don't understand this idea of making an rdf - what value does the rdf provide? Is there a way to view a diff in terminal that makes it faster / more intuitive? I'm genuinely interested to do this or something like it for drives that are slightly out of sync with each other (yes I know, it's dumb but it happened).
- JimmyRuska 3y agoRDF is just an easy way to put everything into a graph, since file systems are stored in a hierarchy. You can use neo4j or whichever graph database, it doesn't have to be rdf. From there you can compare hashes that are in one file system but not in another, and do querying for constraints like largest files, folders with the most files, most recent files, etc. If your file system is slightly out of sync, try out meld for an easy way to diff.