3 ms·

RAID is not sufficient to protect against data loss. If anything, it can provide a false sense of protection.

dijit 3y ago
RAID is literally designed to prevent data corruption using parity from data and gives resilience in the event of drive failures, even intermittent ones. Like all "additional components", RAID controllers come with their own quirks and I have heard of rare cases of RAID controllers being the cause of data loss, but RAID as a concept is designed to combat bit-rot and lossy hardware. ZFS in the same vein is also designed around this concept and attempts to join RAID, an LVM and a filesystem to make "better" choices on how to handle blocks of data. Since RAID only sees raw blocks and is not volume or filesystem aware there are cases where it's slower. That said, I have to also mention that when I was investigating HBASE there was no way to force consistency of data, there was no fsync() call in the code, it only writes to VFS and you have to pray your OS flushes the cache to disk before the system fails. HBASE Parity is configured by HDFS which is essentially doing exactly what RAID does. Except only to VFS and without parity bits.
- zdragnar 3y ago
  Would it be fair to say that "preventing data loss", broadly speaking, requires defense in depth, and that RAID alone is not sufficient? If so, then both things in the gp are true: raid isn't enough, and can be a false sense of security.
  - dijit 3y ago
    Given the context, why would raid and hdfs not be equivalent?
    - ianburrell 3y ago
      RAID is distributed across drives on one machine. That whole machine can fail. Plus, it can take a while to recover the machine or array and it is common for another drive to fail during recovery. HDFS is distributed across multiple machines, each one which can have RAID. It is unlikely that enough machines will fail to lose data.
      - dijit 3y ago
        I believe that its essentially equivalent and neither raid nor hdfs are good enough to exist without backups.

RAID is literally designed to prevent data corruption using parity from data and gives resilience in the event of drive failures, even intermittent ones. Like all "additional components", RAID controllers come with their own quirks and I have heard of rare cases of RAID controllers being the cause of data loss, but RAID as a concept is designed to combat bit-rot and lossy hardware. ZFS in the same vein is also designed around this concept and attempts to join RAID, an LVM and a filesystem to make "better" choices on how to handle blocks of data. Since RAID only sees raw blocks and is not volume or filesystem aware there are cases where it's slower. That said, I have to also mention that when I was investigating HBASE there was no way to force consistency of data, there was no fsync() call in the code, it only writes to VFS and you have to pray your OS flushes the cache to disk before the system fails. HBASE Parity is configured by HDFS which is essentially doing exactly what RAID does. Except only to VFS and without parity bits.

zdragnar 3y ago
Would it be fair to say that "preventing data loss", broadly speaking, requires defense in depth, and that RAID alone is not sufficient? If so, then both things in the gp are true: raid isn't enough, and can be a false sense of security.
- dijit 3y ago
  Given the context, why would raid and hdfs not be equivalent?
  - ianburrell 3y ago
    RAID is distributed across drives on one machine. That whole machine can fail. Plus, it can take a while to recover the machine or array and it is common for another drive to fail during recovery. HDFS is distributed across multiple machines, each one which can have RAID. It is unlikely that enough machines will fail to lose data.
    - dijit 3y ago
      I believe that its essentially equivalent and neither raid nor hdfs are good enough to exist without backups.