3 ms·

> You always want to use mixed lot numbers or disks with significantly different manufacturing dates in your arrays. For the few of us left who know how to do

by mmt 8y ago

> You always want to use mixed lot numbers or disks with significantly different manufacturing dates in your arrays. For the few of us left who know how to do this (or even that it's beneficial to do in the first place), it's becoming less practical to do at scale. It's hard enough finding someone who will hire me and use any of These Thngs I Know about operating hardware, isntead of just my "automation" skills, as if all problems can be solved in software.

viraptor 8y ago
> as if all problems can be solved in software. Isn't this one true to some extent though? We already have scrubbing to detect/fix bitrot. Why not make it (slightly) purposefully unbalanced, so you can avoid the simultaneous failure. I expect your time is worth more money than making one drive fail a few weeks early.
- mmt 8y ago
  > Isn't this one true to some extent though? People who write software for a living tend to think so, but the "some extent" is the real issue. For many classes of problems, merely spending more money on hardware (or procedures/process) is objectively better, but one has to know the alternative exists and be willing to make the comparison. Some problems just have physical limitations. > Why not make it (slightly) purposefully unbalanced, so you can avoid the simultaneous failure. I think the short answer is because drive failure is non-deterministic and relatively unlikely in the general case. > I expect your time is worth more money than making one drive fail a few weeks early. My (ops) time may or may not be worth more than the software engineering time required to make that drive fail early. (NB that the main cost of the hardware solution isn't necessarily time but is often reduced availability and therefore higher cost of suitable parts). Your proposal also still ignores the fact that some of these drive failures may simply not be manipulable by software. For example, any failure that's correlated with power-on (or spun-up) time, rather than usage, such as bearing failures, could still happen simultaneously (and affect hot spares, a nightmarish situation). The tried-and-true engineering solution, which happens to be hardware/process based, actually works, and can be shown to address nearly all known drive failures [1], and has a measurable cost. The same can't be said for a software-only attempt to replace it. [1] firmware bugs, that do things like returning reads of all-zeros on just-written blocks, being a notable exception.
  - viraptor 8y ago
    > I think the short answer is because drive failure is non-deterministic and relatively unlikely in the general case. If it's deterministic enough to be worth sourcing drives from different batches, why wouldn't it be enough to add small amount of writes on purpose? > Your proposal also still ignores the fact that some of these drive failures may simply not be manipulable by software. For example, any failure that's correlated with power-on (or spun-up) time, rather than usage, such as bearing failures, could still happen simultaneously (and affect hot spares, a nightmarish situation). Power cut / spinup / other conditions can be replicated from the OS level as well. I didn't list rather than ignored them. It does sound like a good idea to do those as well, considering it could save you from losing all the drives after a power loss / system crash.
    - mmt 8y ago
      > If it's deterministic enough to be worth sourcing drives from different batches, I suspect you're using a mistaken premise. It's worth sourcing from different batches because failures are not deterministic. Instead, we merely have probabilities based on past experiences (usually from vast data generously provided by operators of spindles at huge scale). > why wouldn't it be enough to add small amount of writes on purpose Well, it's not enough, because it might only protect against simultaneity of certain failures. It also doesn't actually reduce the potential impact of the failures, merely buying more reaction time. By distributing a single batch of drives across many arrays, even a simultaneous failure is just increased replacement maintenance cost (if that's even the strategy, rather than enough hot spares and abandon-in-place), without the looming data loss. With the software staggering of write amplification, each failure could be the start of a cascade, in which case replacement takes on a time-critical aspect. This replacement emergency ends up being an operational (not software) solution, as well. My worry would be that the software scheme provides a false sense of security. Additionally, you may want to quantify what "small amount" is, considering you're suggesting such an algorithm would allow for failure multiple weeks apart. 3 weeks is 2% of 3 years. For an array of 12 drives, does that mean that the 12th drive would need 22% the writes of the 1st drive? Of course, beyond any performance hit, write amplification for SSDs has other deleterious effects (as per the article). A software solution would have to account for yet another corner case.. or just stop trying to re-invent in software what already has a pretty comprehensive solution in operations. > Power cut / spinup / other conditions can be replicated from the OS level as well. Not necessarily, although I suspect that true nearly always on modern equipment. However, that's not what I meant. What I meant was failures that occur more frequently merely with the time the drive has spent powered on (or powered on and spinning). Even if that could be simulated relativistically somehow, that wouldn't be a software solution, either. Also, adding a "chaos monkey" of the kind that powers down a drive in a running array would both introduce a performance hit that I expect a majority of environments would find unacceptable (more than would find write amplification acceptable) and would introduce additional wear and tear on mechanical drives. The latter may be worth it, but I'd be hard pressed to quantify it. It would be different if limited to hot spares, but that's also of limited utility. You'd also have to be extremely careful in implementation, as a bug here could make a previously viable array into a data-lost array. If such a technique reveals a drive failure, I'd want it to stop immediately so as to be able to replace it with a different one from a different batch and have enough replacements on hand, in case all the rest suffer the same fate. > I didn't list rather than ignored them. Unfortunately, it's impossible to tell the difference in discussions on this topic, because, as I mentioned, so few people have first hand knowledge (or have done the research). Even before "the cloud", there was more mythology than hard data (including about temperature, until Google published data debunking that).
siscia 8y ago
If you are willing to move in Geneva I believe that CERN or any of the LHC experiments could use your skills. It may be worthed to visit their career page.
- mmt 8y ago
  Relocation isn't something I'm open to, at this point. (Being in the SF Bay Area, I'm not yet worried that this limits me excessively). I suppose it's also worth noting that I'm sceptical that any organization that large wouldn't have an equally narrow interest in my skillset. My goal is the able to apply as close to the full breadth of what I know and can do, rather than something like specifically avoiding automation or specifically exercising my storage knowledge. For that, startups and other small companies seem best, though, oddly, not lately.

by mmt 8y ago

viraptor 8y ago

> as if all problems can be solved in software. Isn't this one true to some extent though? We already have scrubbing to detect/fix bitrot. Why not make it (slightly) purposefully unbalanced, so you can avoid the simultaneous failure. I expect your time is worth more money than making one drive fail a few weeks early.

mmt 8y ago
> Isn't this one true to some extent though? People who write software for a living tend to think so, but the "some extent" is the real issue. For many classes of problems, merely spending more money on hardware (or procedures/process) is objectively better, but one has to know the alternative exists and be willing to make the comparison. Some problems just have physical limitations. > Why not make it (slightly) purposefully unbalanced, so you can avoid the simultaneous failure. I think the short answer is because drive failure is non-deterministic and relatively unlikely in the general case. > I expect your time is worth more money than making one drive fail a few weeks early. My (ops) time may or may not be worth more than the software engineering time required to make that drive fail early. (NB that the main cost of the hardware solution isn't necessarily time but is often reduced availability and therefore higher cost of suitable parts). Your proposal also still ignores the fact that some of these drive failures may simply not be manipulable by software. For example, any failure that's correlated with power-on (or spun-up) time, rather than usage, such as bearing failures, could still happen simultaneously (and affect hot spares, a nightmarish situation). The tried-and-true engineering solution, which happens to be hardware/process based, actually works, and can be shown to address nearly all known drive failures [1], and has a measurable cost. The same can't be said for a software-only attempt to replace it. [1] firmware bugs, that do things like returning reads of all-zeros on just-written blocks, being a notable exception.
- viraptor 8y ago
  > I think the short answer is because drive failure is non-deterministic and relatively unlikely in the general case. If it's deterministic enough to be worth sourcing drives from different batches, why wouldn't it be enough to add small amount of writes on purpose? > Your proposal also still ignores the fact that some of these drive failures may simply not be manipulable by software. For example, any failure that's correlated with power-on (or spun-up) time, rather than usage, such as bearing failures, could still happen simultaneously (and affect hot spares, a nightmarish situation). Power cut / spinup / other conditions can be replicated from the OS level as well. I didn't list rather than ignored them. It does sound like a good idea to do those as well, considering it could save you from losing all the drives after a power loss / system crash.
  - mmt 8y ago
    > If it's deterministic enough to be worth sourcing drives from different batches, I suspect you're using a mistaken premise. It's worth sourcing from different batches because failures are not deterministic. Instead, we merely have probabilities based on past experiences (usually from vast data generously provided by operators of spindles at huge scale). > why wouldn't it be enough to add small amount of writes on purpose Well, it's not enough, because it might only protect against simultaneity of certain failures. It also doesn't actually reduce the potential impact of the failures, merely buying more reaction time. By distributing a single batch of drives across many arrays, even a simultaneous failure is just increased replacement maintenance cost (if that's even the strategy, rather than enough hot spares and abandon-in-place), without the looming data loss. With the software staggering of write amplification, each failure could be the start of a cascade, in which case replacement takes on a time-critical aspect. This replacement emergency ends up being an operational (not software) solution, as well. My worry would be that the software scheme provides a false sense of security. Additionally, you may want to quantify what "small amount" is, considering you're suggesting such an algorithm would allow for failure multiple weeks apart. 3 weeks is 2% of 3 years. For an array of 12 drives, does that mean that the 12th drive would need 22% the writes of the 1st drive? Of course, beyond any performance hit, write amplification for SSDs has other deleterious effects (as per the article). A software solution would have to account for yet another corner case.. or just stop trying to re-invent in software what already has a pretty comprehensive solution in operations. > Power cut / spinup / other conditions can be replicated from the OS level as well. Not necessarily, although I suspect that true nearly always on modern equipment. However, that's not what I meant. What I meant was failures that occur more frequently merely with the time the drive has spent powered on (or powered on and spinning). Even if that could be simulated relativistically somehow, that wouldn't be a software solution, either. Also, adding a "chaos monkey" of the kind that powers down a drive in a running array would both introduce a performance hit that I expect a majority of environments would find unacceptable (more than would find write amplification acceptable) and would introduce additional wear and tear on mechanical drives. The latter may be worth it, but I'd be hard pressed to quantify it. It would be different if limited to hot spares, but that's also of limited utility. You'd also have to be extremely careful in implementation, as a bug here could make a previously viable array into a data-lost array. If such a technique reveals a drive failure, I'd want it to stop immediately so as to be able to replace it with a different one from a different batch and have enough replacements on hand, in case all the rest suffer the same fate. > I didn't list rather than ignored them. Unfortunately, it's impossible to tell the difference in discussions on this topic, because, as I mentioned, so few people have first hand knowledge (or have done the research). Even before "the cloud", there was more mythology than hard data (including about temperature, until Google published data debunking that).

siscia 8y ago

If you are willing to move in Geneva I believe that CERN or any of the LHC experiments could use your skills. It may be worthed to visit their career page.

mmt 8y ago
Relocation isn't something I'm open to, at this point. (Being in the SF Bay Area, I'm not yet worried that this limits me excessively). I suppose it's also worth noting that I'm sceptical that any organization that large wouldn't have an equally narrow interest in my skillset. My goal is the able to apply as close to the full breadth of what I know and can do, rather than something like specifically avoiding automation or specifically exercising my storage knowledge. For that, startups and other small companies seem best, though, oddly, not lately.