3 ms·
Ask HN: What filesystem(s) are best suited for long-term use?
Given the need for digital preservation in anticipation of current-and-future digital archaeology, beyond mere file formats (and/or descriptions of how to convert them), what file systems are best suited to long-term digital data storage and retrieval?
- aborsy 5y agoZFS!
- warrenm 5y agobecause...?
- thesuperbigfrog 5y agoPlease define "long-term". 5 years? 10 years? 50 years? How many years do you mean by "long-term"?
- warrenm 5y agoAll of the above - feel free to fill-in your own definition of "long-term" in arguing for (or against) any given filesystem :)
- PaulHoule 5y agoDo expect to keep the data on an active array (so that you can scrub it periodically as ZFS would?) or are you keeping it on some write-once medium (optical disks?)
- warrenm 5y agoYes Either Both Maybe on some other kind of media we don't have yet Media is certainly important As is file format But that media has to have a file system of some kind And files have to live in some kind of file system
- fatfreddie 5y agoWhat are you trying to achieve? A file system, any system for that matter, is only good for as long as there are drivers to decode the format in which they are written. My concern would be to store whatever data you have in the simplest format you can think of and thoroughly document it. I don't think any file system is trying to solve that problem. To me the things that spring to my mind are cpio, tar, zip, torrent, zsync, etc.
- warrenm 5y agoThe question is intentionally open-ended - though some filesystems are going to be inherently "wrong" to mention (eg the old MSDOS FAT) because they don't support large enough files, etc All those file formats you mention have to live somewhere - and that's something I've yet to see anyone write/speak much (if at all?) about
- lizknope 5y agoDo you want to put a bunch of data on a hard drive, tape, DVD, whatever, and leave it for 10 to 100 years and come back hoping it works? I think this is a horrible strategy. Data needs to be verified periodically and migrated to new formats. I have files that started on floppy disk in 1991. Over the years they migrated to hard drives, QIC tape, PD phase change optical discs, CD-RW, DVD+RW, and back to hard drives. The file systems have changed from FAT16, ext2, ISO9660, UDF, ext3, ext4. I strongly suggest creating a list of checksums. A simple for loop on every file running md5sum, sha256sum, or similar and storing along with the filenames. You can then run again, compare, and see if all of the data is still intact. Some filesystems like btrfs and zfs will do the checksum calculation every time you read the file. Personally I use ext4 and run cshatag on all files every 6 months. This stores a timestamp and sha256 checksum as ext4 extended attribute metadata. If the file contents have changed but the timestamp has not then it will report corruption. https://github.com/rfjakob/cshatag https://github.com/rfjakob/cshatag You can also create parity info with Parchive. These are basically "sidecar" files with the same file name as the main file with a suffix. If there are errors in the file you can use parchive with the parity data to reconstruct the file. You can also adjust how much parity data to create (more parity takes more space) https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive
- warrenm 5y agoI want to know what filesystems are most aligned with the idea of long-term storage and prepping for digital archaeology :) Whether I “plan” to put data on some piece of media to be be touched for a decade or century or it just “happens”...i think it's something we need to be thinking about in the realm of digital preservation In an ideal world... Sure: you can check your files on a highly-routine basis to ensure no bitrot has happened In the real world... Stuff gets dropped on some random media and forgotten about That's a problem - there is an enormous amount of data that's on perfectly good media, but that cannot be recovered because of filesystem incompatibility