5 ms·

> This could happen if there’s disk corruption, a transient read failure, or a transient write failure caused bad data to be written. Wouldn't transient I/O er

by magnat 9y ago

> This could happen if there’s disk corruption, a transient read failure, or a transient write failure caused bad data to be written. Wouldn't transient I/O error get corrected by block driver itself by repeating the request a few times? Does a filesystem driver assume any error reported by storage device is permanent? > The 2017 tests above used an 8k file where the first block that contained file data either returned an error at the block device level or was corrupted, depending on the test. Are there any tests on how filesystems deal with metadata corruption or I/O error? Apart from NTFS "self-healing" introduced in Windows 7/2008, do modern filesystem drivers attempt to correct broken metadata completely on the kernel side, or just give up to avoid making things worse?

binarycrusader 9y ago
I believe apfs (apple's new filesystem), btrfs, and zfs all checksum metadata and will attempt to correct it.
- tscs37 9y ago
  I recommend the OP article since it specifically mentions apfs.
  - binarycrusader 9y ago
    The OP article mentions apfs does not checksum data, but I assure you that it does checksum metadata: https://en.wikipedia.org/wiki/Apple_File_System#Data_integrity https://en.wikipedia.org/wiki/Apple_File_System#Data_integri...
    - tscs37 9y ago
      Even ext4 has check summing of metadata, it's a fairly boring feature by now.
      - binarycrusader 9y ago
        ext4 only received checksumming of metadata in Linux 3.16, which was released in March 2016 -- not that long ago really. So I wouldn't call it a fairly boring feature since it's barely been two years since it was integrated and I imagine some Linux distributions might not even have it yet. Furthermore, it's unclear to me whether ext4 just uses those checksums for integrity checks or whether it actually has any "self-healing" capabilities. My understanding is apfs, btrfs, and zfs all have self-healing capabilities.
        tscs37 9y ago
        Checksums do not provide "self-healing". Ever. They can only tell you if a piece of data is wrong or not. In the case of BTRFS it's less a case of self-healing, the data is not healed, rather that metadata is replicated across the disk. I believe ZFS does the same but I'm not sure of this. I'm certain that ZFS does this if it has several disks to replicate from. I'm not aware of APFS being self-healing on a single disk.
        binarycrusader 9y ago
        Yes, you obviously need additional pieces of information for reconstruction/repair, sorry for the confusing wording.
        jorangreef 9y ago
        Yes, ZFS also replicates metadata on a single disk.
        binarycrusader 9y ago
        You can even tell ZFS to do the same thing with user data with the ditto blocks feature.
the8472 9y ago
depends on the OS, according to this comment on the ZoL github [0] solaris passes through errors from the hardware while linux retries. [0] https://github.com/zfsonlinux/zfs/issues/1256#issuecomment-13247057 https://github.com/zfsonlinux/zfs/issues/1256#issuecomment-1...
asveikau 9y ago
> Wouldn't transient I/O error get corrected by block driver itself by repeating the request a few times? I am surprised at the seeming assumption here that errors are always transient and you shouldn't worry because the retry will fix it. Sure that probably won't hurt, but it won't address the case where retries fail.

by magnat 9y ago

binarycrusader 9y ago

I believe apfs (apple's new filesystem), btrfs, and zfs all checksum metadata and will attempt to correct it.

tscs37 9y ago
I recommend the OP article since it specifically mentions apfs.
- binarycrusader 9y ago
  The OP article mentions apfs does not checksum data, but I assure you that it does checksum metadata: https://en.wikipedia.org/wiki/Apple_File_System#Data_integrity https://en.wikipedia.org/wiki/Apple_File_System#Data_integri...
  - tscs37 9y ago
    Even ext4 has check summing of metadata, it's a fairly boring feature by now.
    - binarycrusader 9y ago
      ext4 only received checksumming of metadata in Linux 3.16, which was released in March 2016 -- not that long ago really. So I wouldn't call it a fairly boring feature since it's barely been two years since it was integrated and I imagine some Linux distributions might not even have it yet. Furthermore, it's unclear to me whether ext4 just uses those checksums for integrity checks or whether it actually has any "self-healing" capabilities. My understanding is apfs, btrfs, and zfs all have self-healing capabilities.
      - tscs37 9y ago
        Checksums do not provide "self-healing". Ever. They can only tell you if a piece of data is wrong or not. In the case of BTRFS it's less a case of self-healing, the data is not healed, rather that metadata is replicated across the disk. I believe ZFS does the same but I'm not sure of this. I'm certain that ZFS does this if it has several disks to replicate from. I'm not aware of APFS being self-healing on a single disk.
        binarycrusader 9y ago
        Yes, you obviously need additional pieces of information for reconstruction/repair, sorry for the confusing wording.
        jorangreef 9y ago
        Yes, ZFS also replicates metadata on a single disk.
        binarycrusader 9y ago
        You can even tell ZFS to do the same thing with user data with the ditto blocks feature.

the8472 9y ago

depends on the OS, according to this comment on the ZoL github [0] solaris passes through errors from the hardware while linux retries. [0] https://github.com/zfsonlinux/zfs/issues/1256#issuecomment-13247057 https://github.com/zfsonlinux/zfs/issues/1256#issuecomment-1...

asveikau 9y ago

> Wouldn't transient I/O error get corrected by block driver itself by repeating the request a few times? I am surprised at the seeming assumption here that errors are always transient and you shouldn't worry because the retry will fix it. Sure that probably won't hurt, but it won't address the case where retries fail.