6 ms·

I'm curious what this data would look like collated by drive birth date rather than (or in 3D addition to) age. I wouldn't use that as the "primary" way to loo

I'm curious what this data would look like collated by drive birth date rather than (or in 3D addition to) age. I wouldn't use that as the "primary" way to look at things, but it could pop some interesting bits. Maybe one of the manufacturers had a shipload of subpar grease? Slightly shittier magnets? Poor quality silicon? There's all kinds of things that could cause a few months of hard drive manufacture to be slightly less reliable… (Also: "Accumulated power on time, hours:minutes 37451*:12, Manufactured in week 27 of year 2014" — I might want to replace these :D — * pretty sure that overflowed at 16 bit, they were powered on almost continuously & adding 65536 makes it 11.7 years.)

sdenton4 1y ago
I think it's helpful to put on our statistics hats when looking at data like this... We have some observed values and a number of available covariates, which, perhaps, help explain the observed variability. Some legitimate sources of variation (eg, proximity to cooling in the NFS box, whether the hard drive was dropped as a child, stray cosmic rays) will remain obscured to us - we cannot fully explain all the variation. But when we average over more instances, those unexplainable sources of variation are captured as a residual to the explanations we can make, given the avialable covariates. The averaging acts a kind of low-pass filter over the data, which helps reveal meaningful trends. Meanwhile, if we slice the data up three ways to hell and back, /all/ we see is unexplainable variation - every point is unique. This is where PCA is helpful - given our set of covariates, what combination of variables best explain the variation, and how much of the residual remains? If there's a lot of residual, we should look for other covariates. If it's a tiny residual, we don't care, and can work on optimizing the known major axes.
- burnished 1y ago
  Well said, and made me want to go review my stats text.
- tanvach 1y ago
  I find it more straight forward to just model the failure rate with the variables directly, and look metrics like AUC for out of sample data.
- mnw21cam 1y ago
  I personally am looking forward to BackBlaze inventing error bars and statistical tests.
- IgorPartola 1y ago
  Exactly. I used to pore over the Backblaze data but so much of it is in the form of “we got 1,200 drives four months ago and so far none have failed”. That is a relatively small number over a small amount of time. On top of that it seems like by the time there is a clear winner for reliability, the manufacturer no longer makes that particular model and the newer models are just not a part of the dataset yet. Basically, you can’t just go “Hitachi good, Seagate bad”. You have to look at specific models and there are what? Hundreds? Thousands?
  - toast0 1y ago
    > On top of that it seems like by the time there is a clear winner for reliability, the manufacturer no longer makes that particular model and the newer models are just not a part of the dataset yet. That's how things work in general. Even if it is the same model, likely parts have changed anyway. For data storage, you can expect all devices to fail, so redundancy and backup plans are key, and once you have that set, reliability is mostly just a input into your cost calculations. (Ideally you do something to mitigate correlated failures from bad manufacturing or bad firmware)
  - account42 1y ago
    "Actually HGST was better on average than WD"is probably about the only kind of conclusion you can make. As you have noted, looking at specific models doesn't get you anything useful because by the time you have enough data the model is already replaced by a different one - but you can make out trends for manufacturers.
- topaz0 1y ago
  > if we slice the data up three ways to hell and back, /all/ we see is unexplainable variation It's certainly true that you can go too far, but this is a case where we can know a priori that the mfg date could be causing bias in the numbers they're showing, because the estimated failure rates at 5 years cannot contain data from any drives newer than 2020, whereas failure rates at 1 year can. At a minimum you might want to exclude newer drives from the analysis, e.g. exclude anything after 2020 if you want to draw conclusions about how the failure rate changes up to the 5-year mark.
Animats 1y ago
Right. Does the trouble at year 8 reflect bad manufacturing 8 years ago?
- dylan604 1y ago
  Honestly, at 8 years, I'd be leaning towards dirty power on the user's end. For a company like BackBlaze, I'd assume a data center would have conditioned power. For someone at home running a NAS with the same drive connected straight to mains, they may not receive the same life span for a drive from the same batch. Undervolting when the power dips is gnarly on equipment. It's amazing to me how the use of a UPS is not as ubiquitous at home.
  - john01dav 1y ago
    > It's amazing to me how the use of a UPS is not as ubiquitous at home. Most users don't see enough failures that they can attribute to bad power to justify the cost in their mind. Furthermore, USPes are extremely expensive per unit of energy storage, so the more obviously useful use case (of not having your gaming session interrupted by a power outage) simply isn't there.
  - toast0 1y ago
    UPSes are a PITA. I have frequent enough outages that I use them on all of my desktops, and they need a new battery every couple years, and now I'm reaching the point where the whole thing needs replacement. When they fail, they turn short dips, which a power supply might have been able to ride through into an instant failure, and they make terrible beeping at the same time. At least the models I have do their test with the protected load, so if you test regularly, it fails by having an unscheduled shutdown, so that's not great either. And there's not many vendors and my vendor is starting to push dumb cloud shit. Ugh.
    - dylan604 1y ago
      [flagged]
tanvach 1y ago
Agreed, these type of analyses benefit from grouping by cohort years. Standard practice in analytics.
dylan604 1y ago
Over the past couple of years, I've been side hustling a project that requires buying ingredients from multiple vendors. The quantities never work out 1:1, so some ingredients from the first order get used with some from a new order from a different vendor. Each item has its own batch number which when used together for the final product yields a batch number on my end. I logged my batch number with the batch number for each of the ingredients in my product. As a solo person, it is a mountain of work, but nerdy me goes to that effort. I'd assume that a drive manufacture does similar knowing which batch from which vendor the magnets, grease, or silicon all comes from. You hope you never need to use these records to do any kind of forensic research, but the one time you do need it makes a huge difference. So many people doing similar products that I do look at me with a tilted head while their eyes go wide and glaze over as if I'm speaking an alien language discussing lineage tracking.
- gorgoiler 1y ago
  Are you using a merkle tree for batch ids?: crust = f({flour, butter}) filling = f({fruit, sugar}) pie = f({crust, filling}) …where f = hash for a merkle tree with fixed size (but huge!) batch numbers, and f = repr for increasingly large but technically decipherable pie IDs.
  - dylan604 1y ago
    No. That sounds like someone that's had some sort of theory before. I just started up an uneducated man's database in the form of a spreadsheet. My batch 1001 sheet has all of the data necessary. My batch 1002 sheet has all of its data. I'm just a simpleton when it comes to this stuff.
- normie3000 1y ago
  > it is a mountain of work, but nerdy me goes to that effort. Presumably required for compliance, if you're selling your products.. Are there decent softwares for tracking this? Or do you use custom spreadsheets or something?
  - 1y ago
calvinmorrison 1y ago
Yes this is fairly standard in manufacturing environments. builds of material and lot or down to serial # level are tracked for production of complex goods.
numpad0 1y ago
(with a tinfoil hat on) I'm convinced that Backblaze is intentionally withholding and ambiguating data to prevent producing too-easily understood visualization that Seagate is consistently the worst of the last 3 remaining drive manufacturers. Their online notoriety only started after a flooding in Thailand that contaminated all manufacturing clean room for spindle motors in existence, causing bunch of post-flood ST3000DM001 to fail quickly, which probably incentivized enough people for the Backblaze stat tracking to gain recognition and to continue to this date. But even if one puts aside such models affected by the same problem, Seagate drives always exhibited shorter real world MTBF. Since it's not in interest of Backblaze or anyone to smear their brand, they must be tweaking data processing to leave out some of those obvious figures.
- kstrauser 1y ago
  OTOH, Seagate never sold customers SMR drives mislabeled for NAS use.
  - whizzter 1y ago
    Ugh, source on that? In the market for a new NAS/Homeserver soonish (realized my drives are almost at 10 years of power on time) and would like to have spinning rust behind ssd for larger storage.
    - kstrauser 1y ago
      It was a whole thing a while back. This was maybe the original article, but once it landed this was the headline of all tech news for a couple of days. https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shingled-magnetic-recording/ https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shin... SMR drives aren’t inherently bad, but you must not use them in a NAS. The may work well, up until they don’t, and then they really don’t. WD snuck these into their Red line, the one marketed at NAS users. The end result after a huge reputational hit was to promise to keep the Red Pro line on HMR, but the plain Red line is still a coin flip, AFAIK. I will not use WD drives in a NAS. It’s all about trust, and they violated it to as astonishing degree.
TimByte 1y ago
That drive's probably earned a quiet retirement at this point
beAbU 1y ago
Reminds me of the capacitor blight of the late aughts. Knowing when the device was manufactured helped troubleshoot likely suspect components during repair.

sdenton4 1y ago

I think it's helpful to put on our statistics hats when looking at data like this... We have some observed values and a number of available covariates, which, perhaps, help explain the observed variability. Some legitimate sources of variation (eg, proximity to cooling in the NFS box, whether the hard drive was dropped as a child, stray cosmic rays) will remain obscured to us - we cannot fully explain all the variation. But when we average over more instances, those unexplainable sources of variation are captured as a residual to the explanations we can make, given the avialable covariates. The averaging acts a kind of low-pass filter over the data, which helps reveal meaningful trends. Meanwhile, if we slice the data up three ways to hell and back, /all/ we see is unexplainable variation - every point is unique. This is where PCA is helpful - given our set of covariates, what combination of variables best explain the variation, and how much of the residual remains? If there's a lot of residual, we should look for other covariates. If it's a tiny residual, we don't care, and can work on optimizing the known major axes.

burnished 1y ago
Well said, and made me want to go review my stats text.
tanvach 1y ago
I find it more straight forward to just model the failure rate with the variables directly, and look metrics like AUC for out of sample data.
mnw21cam 1y ago
I personally am looking forward to BackBlaze inventing error bars and statistical tests.
IgorPartola 1y ago
Exactly. I used to pore over the Backblaze data but so much of it is in the form of “we got 1,200 drives four months ago and so far none have failed”. That is a relatively small number over a small amount of time. On top of that it seems like by the time there is a clear winner for reliability, the manufacturer no longer makes that particular model and the newer models are just not a part of the dataset yet. Basically, you can’t just go “Hitachi good, Seagate bad”. You have to look at specific models and there are what? Hundreds? Thousands?
- toast0 1y ago
  > On top of that it seems like by the time there is a clear winner for reliability, the manufacturer no longer makes that particular model and the newer models are just not a part of the dataset yet. That's how things work in general. Even if it is the same model, likely parts have changed anyway. For data storage, you can expect all devices to fail, so redundancy and backup plans are key, and once you have that set, reliability is mostly just a input into your cost calculations. (Ideally you do something to mitigate correlated failures from bad manufacturing or bad firmware)
- account42 1y ago
  "Actually HGST was better on average than WD"is probably about the only kind of conclusion you can make. As you have noted, looking at specific models doesn't get you anything useful because by the time you have enough data the model is already replaced by a different one - but you can make out trends for manufacturers.
topaz0 1y ago
> if we slice the data up three ways to hell and back, /all/ we see is unexplainable variation It's certainly true that you can go too far, but this is a case where we can know a priori that the mfg date could be causing bias in the numbers they're showing, because the estimated failure rates at 5 years cannot contain data from any drives newer than 2020, whereas failure rates at 1 year can. At a minimum you might want to exclude newer drives from the analysis, e.g. exclude anything after 2020 if you want to draw conclusions about how the failure rate changes up to the 5-year mark.

Animats 1y ago

Right. Does the trouble at year 8 reflect bad manufacturing 8 years ago?

dylan604 1y ago
Honestly, at 8 years, I'd be leaning towards dirty power on the user's end. For a company like BackBlaze, I'd assume a data center would have conditioned power. For someone at home running a NAS with the same drive connected straight to mains, they may not receive the same life span for a drive from the same batch. Undervolting when the power dips is gnarly on equipment. It's amazing to me how the use of a UPS is not as ubiquitous at home.
- john01dav 1y ago
  > It's amazing to me how the use of a UPS is not as ubiquitous at home. Most users don't see enough failures that they can attribute to bad power to justify the cost in their mind. Furthermore, USPes are extremely expensive per unit of energy storage, so the more obviously useful use case (of not having your gaming session interrupted by a power outage) simply isn't there.
- toast0 1y ago
  UPSes are a PITA. I have frequent enough outages that I use them on all of my desktops, and they need a new battery every couple years, and now I'm reaching the point where the whole thing needs replacement. When they fail, they turn short dips, which a power supply might have been able to ride through into an instant failure, and they make terrible beeping at the same time. At least the models I have do their test with the protected load, so if you test regularly, it fails by having an unscheduled shutdown, so that's not great either. And there's not many vendors and my vendor is starting to push dumb cloud shit. Ugh.
  - dylan604 1y ago
    [flagged]

tanvach 1y ago

Agreed, these type of analyses benefit from grouping by cohort years. Standard practice in analytics.

dylan604 1y ago

Over the past couple of years, I've been side hustling a project that requires buying ingredients from multiple vendors. The quantities never work out 1:1, so some ingredients from the first order get used with some from a new order from a different vendor. Each item has its own batch number which when used together for the final product yields a batch number on my end. I logged my batch number with the batch number for each of the ingredients in my product. As a solo person, it is a mountain of work, but nerdy me goes to that effort. I'd assume that a drive manufacture does similar knowing which batch from which vendor the magnets, grease, or silicon all comes from. You hope you never need to use these records to do any kind of forensic research, but the one time you do need it makes a huge difference. So many people doing similar products that I do look at me with a tilted head while their eyes go wide and glaze over as if I'm speaking an alien language discussing lineage tracking.

gorgoiler 1y ago
Are you using a merkle tree for batch ids?: crust = f({flour, butter}) filling = f({fruit, sugar}) pie = f({crust, filling}) …where f = hash for a merkle tree with fixed size (but huge!) batch numbers, and f = repr for increasingly large but technically decipherable pie IDs.
- dylan604 1y ago
  No. That sounds like someone that's had some sort of theory before. I just started up an uneducated man's database in the form of a spreadsheet. My batch 1001 sheet has all of the data necessary. My batch 1002 sheet has all of its data. I'm just a simpleton when it comes to this stuff.
normie3000 1y ago
> it is a mountain of work, but nerdy me goes to that effort. Presumably required for compliance, if you're selling your products.. Are there decent softwares for tracking this? Or do you use custom spreadsheets or something?
- 1y ago

calvinmorrison 1y ago

Yes this is fairly standard in manufacturing environments. builds of material and lot or down to serial # level are tracked for production of complex goods.

numpad0 1y ago

(with a tinfoil hat on) I'm convinced that Backblaze is intentionally withholding and ambiguating data to prevent producing too-easily understood visualization that Seagate is consistently the worst of the last 3 remaining drive manufacturers. Their online notoriety only started after a flooding in Thailand that contaminated all manufacturing clean room for spindle motors in existence, causing bunch of post-flood ST3000DM001 to fail quickly, which probably incentivized enough people for the Backblaze stat tracking to gain recognition and to continue to this date. But even if one puts aside such models affected by the same problem, Seagate drives always exhibited shorter real world MTBF. Since it's not in interest of Backblaze or anyone to smear their brand, they must be tweaking data processing to leave out some of those obvious figures.

kstrauser 1y ago
OTOH, Seagate never sold customers SMR drives mislabeled for NAS use.
- whizzter 1y ago
  Ugh, source on that? In the market for a new NAS/Homeserver soonish (realized my drives are almost at 10 years of power on time) and would like to have spinning rust behind ssd for larger storage.
  - kstrauser 1y ago
    It was a whole thing a while back. This was maybe the original article, but once it landed this was the headline of all tech news for a couple of days. https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shingled-magnetic-recording/ https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shin... SMR drives aren’t inherently bad, but you must not use them in a NAS. The may work well, up until they don’t, and then they really don’t. WD snuck these into their Red line, the one marketed at NAS users. The end result after a huge reputational hit was to promise to keep the Red Pro line on HMR, but the plain Red line is still a coin flip, AFAIK. I will not use WD drives in a NAS. It’s all about trust, and they violated it to as astonishing degree.

TimByte 1y ago

That drive's probably earned a quiet retirement at this point

beAbU 1y ago

Reminds me of the capacitor blight of the late aughts. Knowing when the device was manufactured helped troubleshoot likely suspect components during repair.