5 ms·

Can you explain what you mean by that?

by javert 4y ago

willis936 4y ago
Filesystems keep checksums of every block of data. If single bits are flipped then they can be corrected. If you encrypt at a lower level than the filesystem then you're at the mercy of that lower level's error correction, but in practice it is rare to encrypt at a lower level. Typically it's done at the filesystem level or higher, including when using self-encrypting drives.
- colejohnson66 4y ago
  > If you encrypt at a lower level than the filesystem then you're at the mercy of that lower level's error correction, but in practice it is rare to encrypt at a lower level. My understanding is that many SSDs do encryption transparently. The ATA protocol even has a “SECURE ERASE” command that instructs the drive to wipe just the encryption key. This allowed even “bad blocks” to be erased securely.
  - willis936 4y ago
    SSDs have a lot more aggressive error correction than most filesystems because they anticipate a high error rate and are constantly changing the map of logical blocks to physical blocks. Tapes at rest don't have to worry about that though.
    - danachow 4y ago
      > SSDs have a lot more aggressive error correction than most filesystems Most filesystems do not do any error correction. ZFS, btrfs, ReFS, bcachefs are a few notable exceptions. And for the most part these schemes are for multiple device resilience - which is architected quite differently from the schemes used in an SSD or tape (yes, even tape). > Tapes at rest don't have to worry about that though. This isn't really true. Pretty much every physical medium at the densities used in modern time requires robust error correction because all physical media has flaws either manufactured or acquired from wear and degradation. For instance, modern LTO tapes use relatively robust 2D Reed-Solomon forward error correction similar to DVD/Blu-Ray.
- jaytaylor 4y ago
  Which filesystems support this degree of integrity checking? Presumably ZFS, but what about EXT4/3, ReiserFS, BTRFS, ZFS, NTFS, and FAT32? It would be wonderful if they all have the feature, but I thought only ZFS was really that paranoid.
  - willis936 4y ago
    ZFS and BTRFS have nice online scrubbing features, but nearly every filesystem these days is journaling, including NTFS and XFS (and its contemporaries). Journaling means every block has a checksum. Sure, FAT32 doesn't have that, but no one should ever have the expectation of data integrity on FAT32. You can run checkdisk on journaling filesystems to scrub for errors.
    - shakna 4y ago
      FAT is a requirement of UEFI though, isn't it? So if you can boot from the drive, you can't rely on it to have the filesystem integrity preserved at the disk level.
    - danachow 4y ago
      > Journaling means every block has a checksum. No it doesn’t. Many journal filesystems use a checksum for log entries, but that is certainly not covering every block of the filesystem with a checksum. And that checksum only comes into play during log recoveries. Once a block is committed to disk there is no checksum in play (unless the fs has special support for it). Some newer journaling filesystems support metadata checksumming, but that is not some requirement to be journaling. XFS has not always supported metadata checksumming, and it’s a relatively recent addition to ext4 (like last decade). NTFS doesn’t do checksums on even metadata. This is one reason why ReFS is a thing.
    - wang_li 4y ago
      Journaling means that there is a two phase commit to the metadata of the file system. This helps avoid file system corruption on unclean shutdown and speed up recovery after an unclean shutdown. But it has nothing to do with data checksums. You can’t perform any scrub like behavior to validate your on, e.g., ext4 just because it has a journal.
- NovemberWhiskey 4y ago
  Block-level encryption in SAN is pretty common.
- danachow 4y ago
  > Filesystems keep checksums of every block of data. False. A limited number of filesystems keep checksums of data - most notably ZFS and btrfs. Some like ext4 and APFS will do it for metadata only. One of the most commonly used filesystems, NTFS, does not for either data or metadata. > Typically it's done at the filesystem level or higher, including when using self-encrypting drives. I don’t know where you got this idea from, but it’s basically the opposite of true. > If single bits are flipped then they can be corrected. Also false. Most checksums are used for error detection, not correction. CRCs as are typically used for filesystems are not particularly well suited for error correction.
  - willis936 4y ago
    Those filesystems (ZFS, BTRFS) have robust error correction, but every modern filesystem has checksums for every block. Checkdisk isn't powered by magic. Also, look into TCG Pyrite. Almost all consumer drives with SED features are Pyrite.
    - danachow 4y ago
      > but every modern filesystem has checksums for every block. At least 2 people have informed you that you’re wrong. Now it’s up to you if you choose to educate yourself on this topic or remain a fool. > Checkdisk isn't powered by magic. What is “checkdisk”? If you’re talking about chkdsk, or Check Disk, or fsck like tools - none of those require checksums to do what they do. At a basic level they check the integrity of on disk data structures - the actual connectivity, valid counts, etc. How in your mind does a checksum contribute to this task? Chkdsk exists for FAT32, which you already seem to admit has no checksums. How do you think it works? > Also, look into TCG Pyrite. Almost all consumer drives with SED features are Pyrite. What does this have to do with anything - it certainly isn’t filesystem level encryption.
meibo 4y ago
Correctly configured RAID setups also make it possible to detect and recover errors across drives & data without downtime, this is commonly how it's done in datacenters.

Can you explain what you mean by that?

by javert 4y ago

Can you explain what you mean by that?

Filesystems keep checksums of every block of data. If single bits are flipped then they can be corrected. If you encrypt at a lower level than the filesystem then you're at the mercy of that lower level's error correction, but in practice it is rare to encrypt at a lower level. Typically it's done at the filesystem level or higher, including when using self-encrypting drives.

colejohnson66 4y ago
> If you encrypt at a lower level than the filesystem then you're at the mercy of that lower level's error correction, but in practice it is rare to encrypt at a lower level. My understanding is that many SSDs do encryption transparently. The ATA protocol even has a “SECURE ERASE” command that instructs the drive to wipe just the encryption key. This allowed even “bad blocks” to be erased securely.
- willis936 4y ago
  SSDs have a lot more aggressive error correction than most filesystems because they anticipate a high error rate and are constantly changing the map of logical blocks to physical blocks. Tapes at rest don't have to worry about that though.
  - danachow 4y ago
    > SSDs have a lot more aggressive error correction than most filesystems Most filesystems do not do any error correction. ZFS, btrfs, ReFS, bcachefs are a few notable exceptions. And for the most part these schemes are for multiple device resilience - which is architected quite differently from the schemes used in an SSD or tape (yes, even tape). > Tapes at rest don't have to worry about that though. This isn't really true. Pretty much every physical medium at the densities used in modern time requires robust error correction because all physical media has flaws either manufactured or acquired from wear and degradation. For instance, modern LTO tapes use relatively robust 2D Reed-Solomon forward error correction similar to DVD/Blu-Ray.
jaytaylor 4y ago
Which filesystems support this degree of integrity checking? Presumably ZFS, but what about EXT4/3, ReiserFS, BTRFS, ZFS, NTFS, and FAT32? It would be wonderful if they all have the feature, but I thought only ZFS was really that paranoid.
- willis936 4y ago
  ZFS and BTRFS have nice online scrubbing features, but nearly every filesystem these days is journaling, including NTFS and XFS (and its contemporaries). Journaling means every block has a checksum. Sure, FAT32 doesn't have that, but no one should ever have the expectation of data integrity on FAT32. You can run checkdisk on journaling filesystems to scrub for errors.
  - shakna 4y ago
    FAT is a requirement of UEFI though, isn't it? So if you can boot from the drive, you can't rely on it to have the filesystem integrity preserved at the disk level.
  - danachow 4y ago
    > Journaling means every block has a checksum. No it doesn’t. Many journal filesystems use a checksum for log entries, but that is certainly not covering every block of the filesystem with a checksum. And that checksum only comes into play during log recoveries. Once a block is committed to disk there is no checksum in play (unless the fs has special support for it). Some newer journaling filesystems support metadata checksumming, but that is not some requirement to be journaling. XFS has not always supported metadata checksumming, and it’s a relatively recent addition to ext4 (like last decade). NTFS doesn’t do checksums on even metadata. This is one reason why ReFS is a thing.
  - wang_li 4y ago
    Journaling means that there is a two phase commit to the metadata of the file system. This helps avoid file system corruption on unclean shutdown and speed up recovery after an unclean shutdown. But it has nothing to do with data checksums. You can’t perform any scrub like behavior to validate your on, e.g., ext4 just because it has a journal.
NovemberWhiskey 4y ago
Block-level encryption in SAN is pretty common.
danachow 4y ago
> Filesystems keep checksums of every block of data. False. A limited number of filesystems keep checksums of data - most notably ZFS and btrfs. Some like ext4 and APFS will do it for metadata only. One of the most commonly used filesystems, NTFS, does not for either data or metadata. > Typically it's done at the filesystem level or higher, including when using self-encrypting drives. I don’t know where you got this idea from, but it’s basically the opposite of true. > If single bits are flipped then they can be corrected. Also false. Most checksums are used for error detection, not correction. CRCs as are typically used for filesystems are not particularly well suited for error correction.
- willis936 4y ago
  Those filesystems (ZFS, BTRFS) have robust error correction, but every modern filesystem has checksums for every block. Checkdisk isn't powered by magic. Also, look into TCG Pyrite. Almost all consumer drives with SED features are Pyrite.
  - danachow 4y ago
    > but every modern filesystem has checksums for every block. At least 2 people have informed you that you’re wrong. Now it’s up to you if you choose to educate yourself on this topic or remain a fool. > Checkdisk isn't powered by magic. What is “checkdisk”? If you’re talking about chkdsk, or Check Disk, or fsck like tools - none of those require checksums to do what they do. At a basic level they check the integrity of on disk data structures - the actual connectivity, valid counts, etc. How in your mind does a checksum contribute to this task? Chkdsk exists for FAT32, which you already seem to admit has no checksums. How do you think it works? > Also, look into TCG Pyrite. Almost all consumer drives with SED features are Pyrite. What does this have to do with anything - it certainly isn’t filesystem level encryption.

meibo 4y ago

Correctly configured RAID setups also make it possible to detect and recover errors across drives & data without downtime, this is commonly how it's done in datacenters.