4 ms·

Before I put my first zfs system in production, I prepared for the following scenarios, using the actual hardware: Corrupt bits on three disks. Cut power to dis

by dusted 7y ago

Before I put my first zfs system in production, I prepared for the following scenarios, using the actual hardware: Corrupt bits on three disks. Cut power to disk Pulled disk out while running. Decided I hated a healthy disk and replaced it with a different disk. Pulled a disk from the zfs and the root mirror and installed a new root mirror without backups. So I now know that it has the ability to recover from most things I could come up with. If I had only kept notes on how to do it, but I deferred that with "I'll do the google and maybe setup a play-system in the same state to test stuff before doing it on the real system".. not a good admin this one (me, myself, not the author, kudos for them to write about their experience and make us all smarter)..

gravypod 7y ago
I think every single person in charge of a production storage system has pulled their hair out for a while dry running these things. If only we built playbooks for our software that showed exactly what to do in these simple failure modes. This post is a very strange failure mode so it'd probably not be in that guide.
- dusted 7y ago
  yeh, it's somewhat strange, then again, disks trying their best, and drivers trying their best, to serve, probably result in many silent retries that just appear as slight performance jitter.. It's not like the FS would know the first time a disk re-reads a sector, it might not even be relocated the first time it happens.. Maybe we need some mechanism for reporting "yeah, we got your bits but it wasn't as smooth sailing as usual" to filesystems such as zfs.
  - marcosdumay 7y ago
    Isn't that what smart was supposed to do?
  - toast0 7y ago
    As marcosdumay says, smart should handle this, but you do have to look. Iffy sectors get marked as Pending, Reallocated sectors are also marked. There's some smart stats for seek times too, I believe (but those weren't as clearly associated with failure as sector counts). There is a danger that manufacturers will avoid acknowledging problems in smart, but so far, I haven't heard about that happening too often. There are certainly some drives that are dead or dying with fine smart values, but it seems rare, and there's often a not obvious dependency where a drive that is very unhealthy may not be able to record new smart values.
    - ianhowson 7y ago
      In my experience, SMART doesn't detect anything of value. 95% of my disk failures are: - Seek time gets very high (>80%). - Drive drops dead and won't respond SMART doesn't detect either of these. Setting an alert on seek times > 30ms has been by far the best predictor of drive failure. If the seek times go over ~400ms then other components start complaining about slow/missing I/O, but by that point you've already gone crazy because your array is unusably slow. If a drive develops bad sectors SMART will report it, but that's rare enough that I don't bother watching for it. It's a non-event with ZFS because of the checksumming. These drives never have one random error; they start raising hundreds in the space of minutes, and that's the moment you replace the drive.
    - dusted 7y ago
      Smart shouldn't be used for this, the sata transport layer should be updated to allow indicating it, syscalls should pass this information along, so filesystems can react to it.. otherwise we'd need a new smart field, as the semantics of the existing fields are insufficiently described, and reacting to them would require adding a smart-query to every filesystem call that touches disk. thing is, "iffy" depends on what firmware decides is iffy.. Maybe a firmware relocates a sector the very first time it fails to read, maybe it relocates after try 200. What we want is to be notified, when we request a sector, if reading it was 100% painless. Depending on your demands for reliability, maybe it's okay to re-read a sector a few times, and still not reallocate it, but maybe your filesystem would like to gather these stats from the disks anyway.
  - derefr 7y ago
    > It’s not like the FS would know the first time a disk re-reads a sector I can understand why the default mode of operation for disks involves the disk presenting a clean “you get a block or you don’t” interface to the OS over e.g. SATA; but I’m surprised that it’s what we’re stuck with 100% of the time. We have ECC memory (where the memory controller reports errors to the OS), but no ECC disks. Maybe this is what Apple is trying to do with putting their T2 chip in charge of being a storage controller. That way, it doesn’t just get to do encryption things; it can also make highly abstract policy-level decisions on what to do about these sorts of error events. Too bad such an approach is only really tenable if you’re building your own storage system out of your own raw NAND; it’d be nice to be able to have an external low-level storage controller built into e.g. the RAID card of a RAID array of regular HDDs, that those HDDs would then slave themselves to, and which ran OS-uploadable firmware.
- pheleven 7y ago
  This actually isn't all that strange a failure mode. We have several large ZFS arrays in service and replace 1-2 failed disks every month. About 90% of the time the first warning you get is exactly this - a message from the CAM controller saying it failed a read in the syslog. ZFS nor SMART often notice these until they get pretty bad/frequent. By the time they're bad enough for other software to notice, your pool is performing pretty poorly. We deal with this by watching for these errors, printing to a log specifically for Icinga to watch for and alert on, and preemptively replace the disks. It would be nice if the other software (ZFS, SMART) would notice these in time to not become severe.
- SamuelAdams 7y ago
  This reminds me of an episode (the pilot) of House, MD. The full script is here [1]. The excerpt: > In a typical case, if you don't cook pork well enough, you digest live tapeworm larvae. They've got these little hooks, they grab onto your bowel, they live, they grow up, they reproduce. Reproduce? There's only one lesion, and it's nowhere near her bowel. That's because this is not a typical case. New professionals always expect things to be similar to what they studied. What they learn: the books and lectures often covered the "most common" situations, but all those edge cases and weird scenarios come up on a daily basis. [1]: https://www.springfieldspringfield.co.uk/view_episode_scripts.php?tv-show=house-md&episode=s01e01 https://www.springfieldspringfield.co.uk/view_episode_script...
  - kbenson 7y ago
    On this topic (and related in many ways) is Bryan Cantrill's talk Zebras All the way Down.[1] The gist is that in medicine there's the concept of diseases that are zebras and horses, rare and common, and the saying goes "when you hear hoof beats think horses not Zebras". Unlike medicine, in software most the easy problems have already been abstracted away or made trivial through our software stacks, so often what's left are Zebras. What used to be the exceedingly rare bugs now account for a large proportion of the odd behavior we see in very robust tech stacks, because the easy bugs have been hunted down and eliminated already. 1: https://www.youtube.com/watch?v=fE2KDzZaxvE https://www.youtube.com/watch?v=fE2KDzZaxvE
    - brandmeyer 7y ago
      “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.” Boris Beizer, Software Testing Techniques. Second edition. 1990
      - deegles 7y ago
        Extremely topical given the outages we're seeing lately.
octosphere 7y ago
In the early days I used one monolithic OS for everything, but I have since virtualized everything with VMWARE and no longer have to worry about my OS failing and then having to check my phone for solutions on how to fix my broken OS. One obvious caveat to virtualisation is the overhead, but with time, you can learn how to use virtualisation with ease and re-provision an entire new OS (and virtual disk) if needed. In terms of how this relates to disk health: well the answer is you now have multiple healthy disks, and only rarely have to then deal with failing disks, which due to compartmentalization, the risk of a failing disk ruining your day is minimized and hopefully the failing disks have nothing too crucial on them. Throw in some cloud storage solutions for backups and this makes things even more bareable.
- tracker1 7y ago
  I have pretty much embraced containerization... as long as the source is out there, and data is well backed up, I'm relatively safe. To me, if it isn't a scripted deployment, it's not really exercised. Aside: One thing I've done a few times is setup a queue and service to replicate data in an rdbms to elasticsearch or mongo as completely denormalized records for searching against. When I did that, it became very easy to store a .json.gz of these same records in s3/blob storage as a secondary backup.

by dusted 7y ago

gravypod 7y ago

I think every single person in charge of a production storage system has pulled their hair out for a while dry running these things. If only we built playbooks for our software that showed exactly what to do in these simple failure modes. This post is a very strange failure mode so it'd probably not be in that guide.

dusted 7y ago
yeh, it's somewhat strange, then again, disks trying their best, and drivers trying their best, to serve, probably result in many silent retries that just appear as slight performance jitter.. It's not like the FS would know the first time a disk re-reads a sector, it might not even be relocated the first time it happens.. Maybe we need some mechanism for reporting "yeah, we got your bits but it wasn't as smooth sailing as usual" to filesystems such as zfs.
- marcosdumay 7y ago
  Isn't that what smart was supposed to do?
- toast0 7y ago
  As marcosdumay says, smart should handle this, but you do have to look. Iffy sectors get marked as Pending, Reallocated sectors are also marked. There's some smart stats for seek times too, I believe (but those weren't as clearly associated with failure as sector counts). There is a danger that manufacturers will avoid acknowledging problems in smart, but so far, I haven't heard about that happening too often. There are certainly some drives that are dead or dying with fine smart values, but it seems rare, and there's often a not obvious dependency where a drive that is very unhealthy may not be able to record new smart values.
  - ianhowson 7y ago
    In my experience, SMART doesn't detect anything of value. 95% of my disk failures are: - Seek time gets very high (>80%). - Drive drops dead and won't respond SMART doesn't detect either of these. Setting an alert on seek times > 30ms has been by far the best predictor of drive failure. If the seek times go over ~400ms then other components start complaining about slow/missing I/O, but by that point you've already gone crazy because your array is unusably slow. If a drive develops bad sectors SMART will report it, but that's rare enough that I don't bother watching for it. It's a non-event with ZFS because of the checksumming. These drives never have one random error; they start raising hundreds in the space of minutes, and that's the moment you replace the drive.
  - dusted 7y ago
    Smart shouldn't be used for this, the sata transport layer should be updated to allow indicating it, syscalls should pass this information along, so filesystems can react to it.. otherwise we'd need a new smart field, as the semantics of the existing fields are insufficiently described, and reacting to them would require adding a smart-query to every filesystem call that touches disk. thing is, "iffy" depends on what firmware decides is iffy.. Maybe a firmware relocates a sector the very first time it fails to read, maybe it relocates after try 200. What we want is to be notified, when we request a sector, if reading it was 100% painless. Depending on your demands for reliability, maybe it's okay to re-read a sector a few times, and still not reallocate it, but maybe your filesystem would like to gather these stats from the disks anyway.
- derefr 7y ago
  > It’s not like the FS would know the first time a disk re-reads a sector I can understand why the default mode of operation for disks involves the disk presenting a clean “you get a block or you don’t” interface to the OS over e.g. SATA; but I’m surprised that it’s what we’re stuck with 100% of the time. We have ECC memory (where the memory controller reports errors to the OS), but no ECC disks. Maybe this is what Apple is trying to do with putting their T2 chip in charge of being a storage controller. That way, it doesn’t just get to do encryption things; it can also make highly abstract policy-level decisions on what to do about these sorts of error events. Too bad such an approach is only really tenable if you’re building your own storage system out of your own raw NAND; it’d be nice to be able to have an external low-level storage controller built into e.g. the RAID card of a RAID array of regular HDDs, that those HDDs would then slave themselves to, and which ran OS-uploadable firmware.
pheleven 7y ago
This actually isn't all that strange a failure mode. We have several large ZFS arrays in service and replace 1-2 failed disks every month. About 90% of the time the first warning you get is exactly this - a message from the CAM controller saying it failed a read in the syslog. ZFS nor SMART often notice these until they get pretty bad/frequent. By the time they're bad enough for other software to notice, your pool is performing pretty poorly. We deal with this by watching for these errors, printing to a log specifically for Icinga to watch for and alert on, and preemptively replace the disks. It would be nice if the other software (ZFS, SMART) would notice these in time to not become severe.
SamuelAdams 7y ago
This reminds me of an episode (the pilot) of House, MD. The full script is here [1]. The excerpt: > In a typical case, if you don't cook pork well enough, you digest live tapeworm larvae. They've got these little hooks, they grab onto your bowel, they live, they grow up, they reproduce. Reproduce? There's only one lesion, and it's nowhere near her bowel. That's because this is not a typical case. New professionals always expect things to be similar to what they studied. What they learn: the books and lectures often covered the "most common" situations, but all those edge cases and weird scenarios come up on a daily basis. [1]: https://www.springfieldspringfield.co.uk/view_episode_scripts.php?tv-show=house-md&episode=s01e01 https://www.springfieldspringfield.co.uk/view_episode_script...
- kbenson 7y ago
  On this topic (and related in many ways) is Bryan Cantrill's talk Zebras All the way Down.[1] The gist is that in medicine there's the concept of diseases that are zebras and horses, rare and common, and the saying goes "when you hear hoof beats think horses not Zebras". Unlike medicine, in software most the easy problems have already been abstracted away or made trivial through our software stacks, so often what's left are Zebras. What used to be the exceedingly rare bugs now account for a large proportion of the odd behavior we see in very robust tech stacks, because the easy bugs have been hunted down and eliminated already. 1: https://www.youtube.com/watch?v=fE2KDzZaxvE https://www.youtube.com/watch?v=fE2KDzZaxvE
  - brandmeyer 7y ago
    “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.” Boris Beizer, Software Testing Techniques. Second edition. 1990
    - deegles 7y ago
      Extremely topical given the outages we're seeing lately.

octosphere 7y ago

In the early days I used one monolithic OS for everything, but I have since virtualized everything with VMWARE and no longer have to worry about my OS failing and then having to check my phone for solutions on how to fix my broken OS. One obvious caveat to virtualisation is the overhead, but with time, you can learn how to use virtualisation with ease and re-provision an entire new OS (and virtual disk) if needed. In terms of how this relates to disk health: well the answer is you now have multiple healthy disks, and only rarely have to then deal with failing disks, which due to compartmentalization, the risk of a failing disk ruining your day is minimized and hopefully the failing disks have nothing too crucial on them. Throw in some cloud storage solutions for backups and this makes things even more bareable.

tracker1 7y ago
I have pretty much embraced containerization... as long as the source is out there, and data is well backed up, I'm relatively safe. To me, if it isn't a scripted deployment, it's not really exercised. Aside: One thing I've done a few times is setup a queue and service to replicate data in an rdbms to elasticsearch or mongo as completely denormalized records for searching against. When I did that, it became very easy to store a .json.gz of these same records in s3/blob storage as a secondary backup.