3 ms·

One important thing is (at least a few years back) we realised that HDDs fail at an alarming rate (> 80%) if you disconnect them and leave them for a few years.

One important thing is (at least a few years back) we realised that HDDs fail at an alarming rate (> 80%) if you disconnect them and leave them for a few years. Either that or someone brought a gamma ray source through our lab. Keeping the disks hot was a different story though.

sgerenser 3y ago
Interesting, I've heard of issues with poor shelf-life for SSDs, but not HDDs. Any data/articles on this topic or is it strictly anecdotal?
- hinkley 3y ago
  Bearings seizing up is the example I usually hear.
  - rbanffy 3y ago
    Most revive after being left in the fridge for some time. The bits and pieces contract at different rates, so cooling (or, I must assume, heating) is enough to get them unstuck. Many drives I did that with lived happy afterlives after being revived, some having been brought back from the dead more than once.
- adrian_b 3y ago
  I am not aware of any large-scale study, but poor HDD reliability in cold storage corresponds to my experience too. I have archived data (almost a couple hundred TB in double copies) on more than 60 HDDs, about half from WD and half from Seagate, from several HDD generations, i.e. with capacities of 2 TB, 3 TB, 4 TB, 6 TB and 8 TB, most being from the 4 up to 8 TB generations. All HDDs were more expensive WD and Seagate models (i.e. with longer warranty times), not their cheapest consumer HDDs. All files were checksummed for error detection. When the HDD content was transferred to tapes after some 3 to 6 years since the initial archival, almost all HDDs had a few errors, which sometimes were not reported by the drives even when they were detected by the file checksums. Only the fact that all files were stored on at least 2 HDDs has prevented data loss, because even when both HDDs had errors they were in different locations.
  - PaulHoule 3y ago
    I've never used tape on a regular basis but when I have I've had terrible experiences with low end tapes. I got kicked out of the computer center at my Alma Mater because I created a bunch of newsgroups. They (said that they) handed me my files on a SunTape that seemed to be empty when I tried to restore it later. A few years later I got a QIC tape for my 486 PC and had similar experiences. There was even the time I tried to restore a several kb configuration file from a Tivoli tape robot and was quoted that it would take 18 hours and figured I could recreate it in much less time. If I did it all the time I'm sure I would do better, but 2 out of 3 times or so I had tape backups fail on me, contrast that to almost always being able to read HDDs, even if there is some bit rot.
  - Zuiii 3y ago
    > 2 copies, checksummed Would using something like ZFS done effectively the same thing? Not being critical. I genuinely want to know more.
- throw0101c 3y ago
  For SSDs, per JEDEC, Client SSDs have to retain data for 1 year at 30C, and Enterprise SSDs have to retain data for 3 months at 40C: * https://www.jedec.org/sites/default/files/Alvin_Cox%20[Compatibility%20Mode]_0.pdf https://www.jedec.org/sites/default/files/Alvin_Cox%20[Compa... I'm sure most last longer, but as a CYA I wouldn't necessarily want to rely on that assumption.
- SAI_Peregrinus 3y ago
  I'd expect the various bearings to seize if left unused for long enough.
- NikkiA 3y ago
  With HDDs it's usually that the bearings dry up and seize if they don't spin regularly.
smeyer 3y ago
Is the 80% sampling a variety of manufacturers and batches, or a single batch of drives that were all ordered at the same time from the same manufacturer? If the latter, I'd suspect it's more of a realization that you got a flawed batch of drives (which is a real risk to manage, but not universal).
- ramraj07 3y ago
  It's a variety of sizes and companies spanning 8 years. And we were talking about 100ish drives. Basically we realised our entire lab backup was useless.
- jamesfinlayson 3y ago
  I think a data centre company posted some findings on this recently.
FpUser 3y ago
I have bunch of HDD's starting from more than 15+ years ago. I was recently checking and all of them are readable. Due tp the total number of drives I would notice 80% failure rate. They're not used as a backup, rather I would start with the data disk and as the price goes down and my needs grow the drive would be replaced with the higher capacity one. And there is an extra set of drives (copy of those) I always back them up.
samstave 3y ago
Expand on keeping-them-hot story.... The biggest expense in TAPE storage has always been the tape-changing-robot. (and if you ever worked with SUN HW -- the software was super expensive and the UI/UX of the backup software (logato? I cant recall the name -- was a PoS (not point of sale) and I hated it... However, with that said, the price/GB/TB for tape is very cheap... What would be a good idea would be to use tape duplication robots to take one tape to N tapes on a fairly regulated basis (like every 2,3,5 years) where you read and copy to new such that the medium (the cheap tape) is cycled through...

sgerenser 3y ago

Interesting, I've heard of issues with poor shelf-life for SSDs, but not HDDs. Any data/articles on this topic or is it strictly anecdotal?

hinkley 3y ago
Bearings seizing up is the example I usually hear.
- rbanffy 3y ago
  Most revive after being left in the fridge for some time. The bits and pieces contract at different rates, so cooling (or, I must assume, heating) is enough to get them unstuck. Many drives I did that with lived happy afterlives after being revived, some having been brought back from the dead more than once.
adrian_b 3y ago
I am not aware of any large-scale study, but poor HDD reliability in cold storage corresponds to my experience too. I have archived data (almost a couple hundred TB in double copies) on more than 60 HDDs, about half from WD and half from Seagate, from several HDD generations, i.e. with capacities of 2 TB, 3 TB, 4 TB, 6 TB and 8 TB, most being from the 4 up to 8 TB generations. All HDDs were more expensive WD and Seagate models (i.e. with longer warranty times), not their cheapest consumer HDDs. All files were checksummed for error detection. When the HDD content was transferred to tapes after some 3 to 6 years since the initial archival, almost all HDDs had a few errors, which sometimes were not reported by the drives even when they were detected by the file checksums. Only the fact that all files were stored on at least 2 HDDs has prevented data loss, because even when both HDDs had errors they were in different locations.
- PaulHoule 3y ago
  I've never used tape on a regular basis but when I have I've had terrible experiences with low end tapes. I got kicked out of the computer center at my Alma Mater because I created a bunch of newsgroups. They (said that they) handed me my files on a SunTape that seemed to be empty when I tried to restore it later. A few years later I got a QIC tape for my 486 PC and had similar experiences. There was even the time I tried to restore a several kb configuration file from a Tivoli tape robot and was quoted that it would take 18 hours and figured I could recreate it in much less time. If I did it all the time I'm sure I would do better, but 2 out of 3 times or so I had tape backups fail on me, contrast that to almost always being able to read HDDs, even if there is some bit rot.
- Zuiii 3y ago
  > 2 copies, checksummed Would using something like ZFS done effectively the same thing? Not being critical. I genuinely want to know more.
throw0101c 3y ago
For SSDs, per JEDEC, Client SSDs have to retain data for 1 year at 30C, and Enterprise SSDs have to retain data for 3 months at 40C: * https://www.jedec.org/sites/default/files/Alvin_Cox%20[Compatibility%20Mode]_0.pdf https://www.jedec.org/sites/default/files/Alvin_Cox%20[Compa... I'm sure most last longer, but as a CYA I wouldn't necessarily want to rely on that assumption.
SAI_Peregrinus 3y ago
I'd expect the various bearings to seize if left unused for long enough.
NikkiA 3y ago
With HDDs it's usually that the bearings dry up and seize if they don't spin regularly.

smeyer 3y ago

Is the 80% sampling a variety of manufacturers and batches, or a single batch of drives that were all ordered at the same time from the same manufacturer? If the latter, I'd suspect it's more of a realization that you got a flawed batch of drives (which is a real risk to manage, but not universal).

ramraj07 3y ago
It's a variety of sizes and companies spanning 8 years. And we were talking about 100ish drives. Basically we realised our entire lab backup was useless.
jamesfinlayson 3y ago
I think a data centre company posted some findings on this recently.

FpUser 3y ago

I have bunch of HDD's starting from more than 15+ years ago. I was recently checking and all of them are readable. Due tp the total number of drives I would notice 80% failure rate. They're not used as a backup, rather I would start with the data disk and as the price goes down and my needs grow the drive would be replaced with the higher capacity one. And there is an extra set of drives (copy of those) I always back them up.

samstave 3y ago

Expand on keeping-them-hot story.... The biggest expense in TAPE storage has always been the tape-changing-robot. (and if you ever worked with SUN HW -- the software was super expensive and the UI/UX of the backup software (logato? I cant recall the name -- was a PoS (not point of sale) and I hated it... However, with that said, the price/GB/TB for tape is very cheap... What would be a good idea would be to use tape duplication robots to take one tape to N tapes on a fairly regulated basis (like every 2,3,5 years) where you read and copy to new such that the medium (the cheap tape) is cycled through...