9 ms·

Could you dumb it down? Are you saying that because all the SSDs were from the same manufacturer and installed at the same time their chance of collectively wea

by kuwze 8y ago

Could you dumb it down? Are you saying that because all the SSDs were from the same manufacturer and installed at the same time their chance of collectively wearing out simultaneously was high?

ApolloFortyNine 8y ago
Most definitely yes. But in addition to this, standard Raid5 does not periodically read the data, so it's actually rather common for issues to only arise on a resilver. This is why proper maintenance in ZFS is to run ZFS Scrub (basically check every file) once a week.
- mmt 8y ago
  That's also proper RAID5/6 maintenance. My main/recent familiarity is with LSI hardware RAID implementation, where they call it a "patrol read". I believe mdraid has checkarray. I'm not sure if you meant to imply that ZFS is different from standard RAID in this regard, but it doesn't seem as though it is.
  - jacquesm 8y ago
    It's called scrubbing.
    - mmt 8y ago
      Did you mean to reply to my above question? If so, I'm unclear as to what you're trying to get across. Is ZFS scrubbing different than the other RAIDs' (scheduled or schedulable) reads of the entire array, other than nomenclature?
- Siecje 8y ago
  We use ZFS, how can I check if we are doing a ZFS Scrub every week?
  - simcop2387 8y ago
    'zpool status' will tell you the last time a scrub was run, or if one is running currently and information about it. Then check your crons and see that they make sense and match what you see with zpool status.
    - Siecje 8y ago
      Well it is not every week. > scan: scrub repaired 0 in 0h36m with 0 errors on Sun Aug 12 06:00:50 2018 I don't see any cron jobs though...
      - simcop2387 8y ago
        Given the speed of that, I'd bet you don't have a huge pool (or if you do, that's a really nice speed). I'd bet someone's doing it manually. That's what I do for my own systems (about once a month) since they're not heavy use (personal and parent's file servers).
- agapon 8y ago
  Once a week sounds extreme unless we talk about a smallish SSD-only pool. You may wear out your (spinning) disks with scrubs more than you do with real workloads. Also, depending on a pool size it may take days for a scrub to complete.
  - mmt 8y ago
    > You may wear out your (spinning) disks with scrubs more than you do with real workloads. This seems like an extraordinary claim requiring extraordinary evidence, especially since the notion of wear out is only applicable to SSDs (as shorthand for write endurance). I certainly believe that mechanical disks, with all those moving parts, can have their failure rates increased by increased use, but it's not safe to assume even something as high as a linearly proportional relationship, considering which parts move when.
da_chicken 8y ago
That's true for any type of disks. If you install a disk array using disks from the same lot from the same manufacturer, it's extremely likely that you'll get disk failure at more or less the exact same time. You always want to use mixed lot numbers or disks with significantly different manufacturing dates in your arrays. Some people buy spares from multiple manufacturers, others buy from multiple vendors since it's unlikely to get the same lot that way. RAID 1 naturally spreads writes across all disks. RAID 5 is designed to spread writes evenly across all disks. Sure, certain applications will do uneven writing, but in general it will be fairly even and this is by design. RAID 3 and RAID 4 were abandoned partially because they used dedicated parity drives, and the result was that the parity drives had much higher write loads and so they failed all the time. This meant that the arrays were more often rebuilding or running with degraded protection.
- mmt 8y ago
  > You always want to use mixed lot numbers or disks with significantly different manufacturing dates in your arrays. For the few of us left who know how to do this (or even that it's beneficial to do in the first place), it's becoming less practical to do at scale. It's hard enough finding someone who will hire me and use any of These Thngs I Know about operating hardware, isntead of just my "automation" skills, as if all problems can be solved in software.
  - viraptor 8y ago
    > as if all problems can be solved in software. Isn't this one true to some extent though? We already have scrubbing to detect/fix bitrot. Why not make it (slightly) purposefully unbalanced, so you can avoid the simultaneous failure. I expect your time is worth more money than making one drive fail a few weeks early.
    - mmt 8y ago
      > Isn't this one true to some extent though? People who write software for a living tend to think so, but the "some extent" is the real issue. For many classes of problems, merely spending more money on hardware (or procedures/process) is objectively better, but one has to know the alternative exists and be willing to make the comparison. Some problems just have physical limitations. > Why not make it (slightly) purposefully unbalanced, so you can avoid the simultaneous failure. I think the short answer is because drive failure is non-deterministic and relatively unlikely in the general case. > I expect your time is worth more money than making one drive fail a few weeks early. My (ops) time may or may not be worth more than the software engineering time required to make that drive fail early. (NB that the main cost of the hardware solution isn't necessarily time but is often reduced availability and therefore higher cost of suitable parts). Your proposal also still ignores the fact that some of these drive failures may simply not be manipulable by software. For example, any failure that's correlated with power-on (or spun-up) time, rather than usage, such as bearing failures, could still happen simultaneously (and affect hot spares, a nightmarish situation). The tried-and-true engineering solution, which happens to be hardware/process based, actually works, and can be shown to address nearly all known drive failures [1], and has a measurable cost. The same can't be said for a software-only attempt to replace it. [1] firmware bugs, that do things like returning reads of all-zeros on just-written blocks, being a notable exception.
tetha 8y ago
There are two nasty thing going on with raid5: a) You can tolerate 2 drive failures, no more. One less, but if you tolerated one, you can tolerate one more, and no more. AND b) raid rebuild causes MASSIVE stress on the remaining drives. Seriously massive stress, beyond what distributed systems or raid 1 / 6 based systems do. This occurs because RAID5 both rewrites parities, but it also has to re-read data from all drives while writing parities to all drives. That's a lot of random access and especially spin-drives and larger drives (see the coincidence?) dislike that. That tends to cause similar-aged drives to die and then your raid is gone. I'd suppose this is less hard on SSDs than HDDs. But there's still a lot of rewrites going on, and SSDs don't like that either.
- rincebrain 8y ago
  a) RAID5 can fault precisely one drive without data loss. Two overlapping errors (e.g. two errors in one stripe) and you're up shit creek without a paddle. This is the generic definition for RAID5, not an implementation detail. b) What makes you think RAID6 doesn't also incur this? The only difference is that RAID6 also includes a Q parity block in each stripe, so the only thing you get saved from is if you don't need to read the parity on the stripes, you save 1/(N+P-1) IOs per drive. RAID6 is still going to need to recompute ~2/(N+P) parities (one P, and one Q) for rebuilding a drive over (N+P) stripes; and reconstruct the data for the rest (depending on how P and Q are implemented, they could interweave which they use for reconstructions, but AIUI it's generally more expensive to recompute from Q than P, and R than P or Q, in many instances of this math). c) Many RAID systems can rebuild starting from the starts of the respective disks and streaming along (or, in recent ZFS's case, coalescing the IOs to be in sequential order groups and issuing them), though certainly not all of them. The logic "usually" goes that RAID5/RAID6 rebuilds are dangerous because they involve reading all the bits, so to speak, so if you don't have an equivalent of scheduled patrol reads to be sure bits at rest that haven't been read by users haven't gone south, you'll first discover this...during a rebuild, and with RAID5, you're SOL.
simooooo 8y ago
Also true for light bulbs

by kuwze 8y ago

Could you dumb it down? Are you saying that because all the SSDs were from the same manufacturer and installed at the same time their chance of collectively wearing out simultaneously was high?

ApolloFortyNine 8y ago

Most definitely yes. But in addition to this, standard Raid5 does not periodically read the data, so it's actually rather common for issues to only arise on a resilver. This is why proper maintenance in ZFS is to run ZFS Scrub (basically check every file) once a week.

mmt 8y ago
That's also proper RAID5/6 maintenance. My main/recent familiarity is with LSI hardware RAID implementation, where they call it a "patrol read". I believe mdraid has checkarray. I'm not sure if you meant to imply that ZFS is different from standard RAID in this regard, but it doesn't seem as though it is.
- jacquesm 8y ago
  It's called scrubbing.
  - mmt 8y ago
    Did you mean to reply to my above question? If so, I'm unclear as to what you're trying to get across. Is ZFS scrubbing different than the other RAIDs' (scheduled or schedulable) reads of the entire array, other than nomenclature?
Siecje 8y ago
We use ZFS, how can I check if we are doing a ZFS Scrub every week?
- simcop2387 8y ago
  'zpool status' will tell you the last time a scrub was run, or if one is running currently and information about it. Then check your crons and see that they make sense and match what you see with zpool status.
  - Siecje 8y ago
    Well it is not every week. > scan: scrub repaired 0 in 0h36m with 0 errors on Sun Aug 12 06:00:50 2018 I don't see any cron jobs though...
    - simcop2387 8y ago
      Given the speed of that, I'd bet you don't have a huge pool (or if you do, that's a really nice speed). I'd bet someone's doing it manually. That's what I do for my own systems (about once a month) since they're not heavy use (personal and parent's file servers).
agapon 8y ago
Once a week sounds extreme unless we talk about a smallish SSD-only pool. You may wear out your (spinning) disks with scrubs more than you do with real workloads. Also, depending on a pool size it may take days for a scrub to complete.
- mmt 8y ago
  > You may wear out your (spinning) disks with scrubs more than you do with real workloads. This seems like an extraordinary claim requiring extraordinary evidence, especially since the notion of wear out is only applicable to SSDs (as shorthand for write endurance). I certainly believe that mechanical disks, with all those moving parts, can have their failure rates increased by increased use, but it's not safe to assume even something as high as a linearly proportional relationship, considering which parts move when.

da_chicken 8y ago

That's true for any type of disks. If you install a disk array using disks from the same lot from the same manufacturer, it's extremely likely that you'll get disk failure at more or less the exact same time. You always want to use mixed lot numbers or disks with significantly different manufacturing dates in your arrays. Some people buy spares from multiple manufacturers, others buy from multiple vendors since it's unlikely to get the same lot that way. RAID 1 naturally spreads writes across all disks. RAID 5 is designed to spread writes evenly across all disks. Sure, certain applications will do uneven writing, but in general it will be fairly even and this is by design. RAID 3 and RAID 4 were abandoned partially because they used dedicated parity drives, and the result was that the parity drives had much higher write loads and so they failed all the time. This meant that the arrays were more often rebuilding or running with degraded protection.

mmt 8y ago
> You always want to use mixed lot numbers or disks with significantly different manufacturing dates in your arrays. For the few of us left who know how to do this (or even that it's beneficial to do in the first place), it's becoming less practical to do at scale. It's hard enough finding someone who will hire me and use any of These Thngs I Know about operating hardware, isntead of just my "automation" skills, as if all problems can be solved in software.
- viraptor 8y ago
  > as if all problems can be solved in software. Isn't this one true to some extent though? We already have scrubbing to detect/fix bitrot. Why not make it (slightly) purposefully unbalanced, so you can avoid the simultaneous failure. I expect your time is worth more money than making one drive fail a few weeks early.
  - mmt 8y ago
    > Isn't this one true to some extent though? People who write software for a living tend to think so, but the "some extent" is the real issue. For many classes of problems, merely spending more money on hardware (or procedures/process) is objectively better, but one has to know the alternative exists and be willing to make the comparison. Some problems just have physical limitations. > Why not make it (slightly) purposefully unbalanced, so you can avoid the simultaneous failure. I think the short answer is because drive failure is non-deterministic and relatively unlikely in the general case. > I expect your time is worth more money than making one drive fail a few weeks early. My (ops) time may or may not be worth more than the software engineering time required to make that drive fail early. (NB that the main cost of the hardware solution isn't necessarily time but is often reduced availability and therefore higher cost of suitable parts). Your proposal also still ignores the fact that some of these drive failures may simply not be manipulable by software. For example, any failure that's correlated with power-on (or spun-up) time, rather than usage, such as bearing failures, could still happen simultaneously (and affect hot spares, a nightmarish situation). The tried-and-true engineering solution, which happens to be hardware/process based, actually works, and can be shown to address nearly all known drive failures [1], and has a measurable cost. The same can't be said for a software-only attempt to replace it. [1] firmware bugs, that do things like returning reads of all-zeros on just-written blocks, being a notable exception.

tetha 8y ago

There are two nasty thing going on with raid5: a) You can tolerate 2 drive failures, no more. One less, but if you tolerated one, you can tolerate one more, and no more. AND b) raid rebuild causes MASSIVE stress on the remaining drives. Seriously massive stress, beyond what distributed systems or raid 1 / 6 based systems do. This occurs because RAID5 both rewrites parities, but it also has to re-read data from all drives while writing parities to all drives. That's a lot of random access and especially spin-drives and larger drives (see the coincidence?) dislike that. That tends to cause similar-aged drives to die and then your raid is gone. I'd suppose this is less hard on SSDs than HDDs. But there's still a lot of rewrites going on, and SSDs don't like that either.

rincebrain 8y ago
a) RAID5 can fault precisely one drive without data loss. Two overlapping errors (e.g. two errors in one stripe) and you're up shit creek without a paddle. This is the generic definition for RAID5, not an implementation detail. b) What makes you think RAID6 doesn't also incur this? The only difference is that RAID6 also includes a Q parity block in each stripe, so the only thing you get saved from is if you don't need to read the parity on the stripes, you save 1/(N+P-1) IOs per drive. RAID6 is still going to need to recompute ~2/(N+P) parities (one P, and one Q) for rebuilding a drive over (N+P) stripes; and reconstruct the data for the rest (depending on how P and Q are implemented, they could interweave which they use for reconstructions, but AIUI it's generally more expensive to recompute from Q than P, and R than P or Q, in many instances of this math). c) Many RAID systems can rebuild starting from the starts of the respective disks and streaming along (or, in recent ZFS's case, coalescing the IOs to be in sequential order groups and issuing them), though certainly not all of them. The logic "usually" goes that RAID5/RAID6 rebuilds are dangerous because they involve reading all the bits, so to speak, so if you don't have an equivalent of scheduled patrol reads to be sure bits at rest that haven't been read by users haven't gone south, you'll first discover this...during a rebuild, and with RAID5, you're SOL.

simooooo 8y ago

Also true for light bulbs