4 ms·

Luckily, there are filesystems like ZFS to [help] counter the myriad of bizarre failure modes in today's storage devices...

• baruch 13y ago
  ZFS would help to handle some of the failure types of SSDs. I doubt that it can help in the case of a multiple disk failures due to a power failure. As for the original report, if you rely on your storage devices to survive a power outage you should be testing them strongly for this feature. It is a feature that is easy to get wrong and when it fails on you it is quite nearly the end of the world as this is a correlated failure of multiple devices at the same time with no time for any rebuild operation. I've seen different devices even of high end caliber that had flaws in their power-hold circuitry. If you rely on it, test it. The same is true for HDDs too. If you rely on your UPS to hold you for the power outage, be sure to test that one too. And test all of these also as they age, the age affects the ability to hold power in both battery and capacitors.
  • chongli 13y ago
    >ZFS would help to handle some of the failure types of SSDs. I doubt that it can help in the case of a multiple disk failures due to a power failure. It can if there's enough redundancy. If your pool is a stripe of mirrors, then you can mitigate this effect by putting each half of a mirror on separate power supplies.
    • rodgerd 13y ago
      With enough redundant devices any RAID scheme will survive a large number of drive losses. ZFS is not magic.
      • chongli 13y ago
        Sure, but ZFS can survive intermittent drive power failure on bother halves of a mirror. As long as one good copy of a block exists, ZFS will repair the mirror by resilvering its counterpart. This is the advantage of checksumming every block on disk and using copy-on-write.
        • sargun 13y ago
          Yeah, but as opposed to data corruption, your data will throw an error when you try to read it. It's just a different failure mode. In production, SSDs will often accept a write, take a sync/flush (acknowledge they've been written to durable storage), and when reading them back, they will not return the same data that was previously written out to them. Mirroring is also part of ZFS, but it's not ZFS magic. My preference is the Dynamo / Hadoop / layered approach in which the application is aware of multiple underlying storage devices, and is written in a way that's defensible to underlying device failure.
          • chongli 13y ago
            Yeah, but as opposed to data corruption, your data will throw an error when you try to read it. No, the data will be repaired transparently and you won't notice a thing. Errors will be logged, of course, but you'll know that the data is intact. My preference is the Dynamo / Hadoop / layered approach in which the application is aware of multiple underlying storage devices, and is written in a way that's defensible to underlying device failure. And ZFS should be the underlying file system upon which these applications are built. If you don't have protection against silent data corruption at the lowest level then you aren't going to have any idea when it happens.
            • sargun 13y ago
              Only if you mirror the data. If you only have one copy of the data, it wont magically repair itself.
• kabdib 13y ago
  I wrote (as far as I know) one of the first resilient file systems for flash, for the Apple Newton, back in '92. We supported transactional object stores on NOR flash, which allowed random reads at full bus speed, and random writes as long as you were programming zeros. You could erase whole blocks back to all 1s, of course. One of the things I had to model was the failure mode of the device. It was simple, I assumed: - Stuck bits (at 1 or 0) would cause either a data write failure, or a failed attempt to write to a tracking structure. Both recoverable by using other storage. - Failure to erase would result in a bad block. - Writes to other memory wouldn't cause disturbance to already committed writes - ... a few other things as well. NOR makes it easy, I guess. For the time, and for the quality of flash, it worked pretty well. The Intel flash guys told us we were being too paranoid, even as it was. NAND flash looks a lot more byzantine, and I can well imagine that survival of your data depends in large part on how much the firmware team understands error recovery and transactions. Having been there, I'm really cautious about SSD brands that I use (no names here, but you can probably guess who I avoid).