3 ms·
> the way storage media devices have been going... My experience is that drives have been getting substantially better in this regard. If a drive is failing, i
by Taek 11y ago
> the way storage media devices have been going...
My experience is that drives have been getting substantially better in this regard. If a drive is failing, it will usually collapse into read only mode and let you know that it is failing, giving you ample time to recover data.
That said, I do think backups are a good idea, and that Reed-Solomon codes are a good way to protect your data. I also think that managing bitrot is really the responsibility of the drive itself and maybe of the filesystem. Zfs and btrfs both offer bitrot protection.
And even solving bitrot, risks like fires, earthquakes, or war (for less fortunate countries) mean that data should be backed up in multiple physical locations anyway.
These are problems I've been attempting to address at https://sia.tech https://sia.tech for the past two years. It's still in beta, but a full platform should be ready by March.
- mbq 11y agoThere are also distributed file systems (MooseFS, Gluster, Ceph) which can duplicate data across physical locations, protect it with checksums, scrub in a distributed manner and auto-heal, all of that transparently.
- scrupulusalbion 11y agoHow easy are those to setup and how reliable are they?
- mbq 11y agoAbout hard as setting up any UNIX daemon on a few computers. Reliability is a complex topic, depends on what you need -- for an anecdotal evidence I maintain a microscopic (10-20TB/4-6 desktop hosts) Moose deployment for a distributed home dir use-case over 5 years without any data loss despite two full computer losses, one disk malfunction and numerous network and power outages (it never detected a bit-rot event though). There are many serious success and horror stories one Google query away.
- ashmud 11y ago> If a drive is failing, it will usually collapse into read only mode and let you know that it is failing, giving you ample time to recover data. Do you have any real world references of this kind of fail behavior occurring? I only have experience with mechanical HDD failure (RAID drives gone bad -> degraded until rebuild; non-RAID drives getting massive bad sectors; non-RAID drives click -> dead). I've read about drives supposedly being able to enter a "nice" readonly mode, but that was a couple years or more ago, about SSDs. More recently, I've only read about drives either entering readonly mode until power cycle, then completely inaccessible; or simply lockup and completely inaccessible.