5 ms·

I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself

I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for. With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? If the concerns about "data integrity" are really about time until a disk is replaced, then it makes sense. Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.

cthalupa 7y ago
>I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for. Nothing in your setup there protects you from bitrot. >With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? Drive rebuild time is primarily a function of a variety of things - speed of the disks, processing power involved, etc. There's nothing magic about any solution here. ZFS does get a leg up in that it knows what blocks are in use, but hardware raid and mdadm are quite fast when they find empty blocks too, so it's not some overwhelmingly massive improvement. >Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems. I think you're fundamentally confused about what journaling provides a filesystem. A journal helps you in situations where your system crashes, loses power, etc. and you had not finished committing your changes to the filesystem. It does not protect you from data integrity issues that are unrelated to this sort of situation - a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal.
- anon9001 7y ago
  > a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot. Is the difference that ZFS scrubs in real-time? Sorry if I'm being dense here, but I did a fair amount of reading about this, and I'm trying to understand what I'm missing.
  - cthalupa 7y ago
    >Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot. Nope. Parity bits aren't used to detect bitrot. If you get a URE when you go to read, the parity bits will be used to get you data. If you read successfully, nothing happens at all. If you scrub, and the data is found to be inconsistent, md assumes that the parity is wrong and overwrites it. https://linux.die.net/man/4/md https://linux.die.net/man/4/md >If a read error is detected during this process, the normal read-error handling causes correct data to be found from other devices and to be written back to the faulty device. In many case this will effectively fix the bad block. >If all blocks read successfully but are found to not be consistent, then this is regarded as a mismatch. >If check was used, then no action is taken to handle the mismatch, it is simply recorded. If repair was used, then a mismatch will be repaired in the same way that resync repairs arrays. For RAID5/RAID6 new parity blocks are written. For RAID1/RAID10, all but one block are overwritten with the content of that one block.
    - anon9001 7y ago
      Thanks for the explanation. I didn't realize md didn't use the parity checks unless there's a URE. Is it correct to say that ZFS parity is used to reconcile on each read even without UREs, but with md parity is only used when there are UREs? If that's true, it makes much more sense why ZFS is considered the right choice. Also, do you know if adding luks in the mix changes that? My data goes md->lvm->luks->ext4 and I hadn't considered which part of that is supposed to catch the error if it wasn't a URE. It makes sense that md would update parity if there was no URE. It also makes sense that LVM wouldn't know or care about data integrity. I would think luks would require data integrity to work properly, but I'm not sure what happens if there is bitrot in a luks block. I'm also surprised ext4 doen't have checksumming, but apparently it's only for able to checksum for metadata and it's disabled by default: https://serverfault.com/questions/456319/how-to-get-checksum-data-integrity-on-ext4 https://serverfault.com/questions/456319/how-to-get-checksum... Dm-integrity sounds worth a look, but https://raid.wiki.kernel.org/index.php/Dm-integrity https://raid.wiki.kernel.org/index.php/Dm-integrity doesn't inspire much confidence.
saalweachter 7y ago
I went ~2 years without running a scrub when I first started using ZFS many years ago; when I ran my first scrub, it found 2 file corruptions, one in a random picture back when I still copied pictures from a digital camera to a hard drive, one in a random file in the browser cache. (If I'd been scrubbing regularly, these almost certain would have been found before corruption crept into the mirrored copy of the file, and been recoverable.)
- cmurf 7y ago
  This is a great point about how pernicious corruption is: it replicates. Detection is more important than correction, at least you can stop the corruption from spreading. It's typical for people to reuse storage hardware, and each derivative inherits prior corruption as well as contributing its own. Most people don't keep a long term unchanging archive, and even that archive likely has no means of unambiguous verification of data integrity.
  - saalweachter 7y ago
    To be honest I'm not sure if the corruption was replicated directly, or if it's just that the same file got unlucky on two different drives -- I can't recall if this was before or after a drive failure.
Quekid5 7y ago
Your setup won't save you from e.g. bad cables silently corrupting data, or drives simply returning bad data, etc. etc. Btw, journaling has nothing to do with integrity, though I believe it is checksummed these days.
- anon9001 7y ago
  > Your setup won't save you from e.g. bad cables silently corrupting data, or drives simply returning bad data, etc. etc. I thought this was what RAID was designed to prevent. I don't know what a cable failure would look like in my setup in practice, but I would think the disk would start failing parity checks against the rest of the array and be flagged as bad.
  - cthalupa 7y ago
    Well, each RAID implementation has different designs and goals in mind, but the original design was simply meant to replace expensive single drives. Using more spindles you could get more performance out of a bunch of cheap disks, and to help with the fact that these cheap disks failed more often, you made them redundant. In general, dealing with data corruption that does not generate an error is not within the purview of RAID implementations. ZFS and btrfs handle it because they're more than just RAID implementations. mdadm can do similar things in a RAID setup if dm-integrity is used on all of the drives (but it is not used by default when just setting up mdadm/lvm/etc.)
  - Quekid5 7y ago
    RAID is primarily about redundancy and speed, not really end-to-end integrity. (Though some setups will certainly have some measures against corruption.) ZFS is really designed for end-to-end integrity. Once you're stored data on ZFS, you're basically guaranteed that you'll always get the exact same thing back (or an error)... no matter where in the "chain" of I/O the failure occurred.
- rodgerd 7y ago
  Or a power supply problem, which was a source of great pain for me at one point.
ilikejam 7y ago
On the last round of zpool scrubs of the 13PB-used set of zfs filesystems here (raidz3, mostly SAS drives, all hosts with ECC RAM), 50MB of data was repaired. 50MB isn't much compared with 13PB, but silent corruption is a real thing.
- anon9001 7y ago
  That makes sense, but mdadm also scrubs weekly on a cronjob. Is there something about ZFS that makes corruption more recoverable than mdadm's RAID?
  - InvaderFizz 7y ago
    One thing is that ZFS tells you exactly what file is corrupted. I recently took over a ZFS setup that was built with plain ZFS on top of a 24-drive wide RAID6. Drives failed, rebuilt the RAID6, nothing appeared to be wrong. ZFS scrub revealed two corrupted files. I was able to pull backups of those two files, replace them on the FS, and rerun the scrub. Obviously I moved off the RAID6 soon after, but this is a powerful example of why the checksuming of ZFS is so useful.
  - ilikejam 7y ago
    RAID5 corruption is non-recoverable - you can't tell which is right, the data or the parity. RAID6 is recoverable - you can get consensus between the data and the two parity bits. ZFS stores hashes of each block, so as long as you have two copies, you can just pick the data+hash that's correct and restore the corrupt block, even with only two copies of the data.
  - saltcured 7y ago
    The core ZFS feature being references here is storage checksums. The general argument is that as storage volumes have increased in size and/or workload, the conventional checksum methods have not been changed to increase their reliability in a way that prevents real world corruption on real world storage during their real world lifetimes. A RAID system can only reliably recover when an IO error is reported. In this case, it can assume which element of the stored data is no longer trustworthy and use the other parts to reconstruct it. A RAID scrub operation that finds inconsistencies, without a reported IO error, cannot really determine why it is inconsistent. It doesn't know which element is corrupt and which other elements to use to reconstruct it. In effect, ZFS adds another data integrity layer over the physical storage, so you can have a lower probability of silent failures where the disk controller or IO channels do their own "parity" or other checks and report a successful read when they actually returned corrupt values. Something like mdadm or LVM would need to add a new layer of checksums over the physical devices to provide this level of protection. Each block/chunk/stripe written to disk would need another integrity checksum written as well. During operation, these checksums would be verified before deciding that a read operation was actually successful. If the verification fails, a virtual IO error is synthesized to flag the read as bad, so that the other RAID recovery techniques can then be applied while knowing which block/chunk/stripe is invalid.
throw0101a 7y ago
> With RAID6 I can have 2 drives fail without data loss. If you don't have checksums on your data, how do you know it's good (even without drive failures)? Your RAID rebuild could be recreating junk. > Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems. Journaling helps with speeding up post-crash reboots: instead of doing a full fsck, it just replays the log.
- iRobbery 7y ago
  "Your RAID rebuild could be recreating junk.." From experience i can tell this can happen. nervous trauma twitch
JackMcMack 7y ago
The issue is not a loudly failing disk. If you get read errors, or if the disk is simply offline, it's easy to recalculate the data. But what if you have silent data corruption: a single bit? Even if you have parity (raid5) or a duplicate (raid1), how do you know which part is correct, and which part to repair? Without checksums. I've had to restore 1TB from Crashplan. They apparently had checksums in their software, and something like 10-15 files were damaged. Without checksums, you simply don't know if your data is damaged, even on a single hard drive. ZFS gives you data integrity with checksums, and redundancy with mirrors or raid-z. This is why a regular scrub is recommended, you verify the checksums and repair if possible.
- rrauenza 7y ago
  I was won over to btrfs for my new NAS (~6 years ago!) when I saw a demo (which I expect is also valid on zfs) where the presenter starts with a mirrored setup, finds the exact block offset location of the file's contents and then overwrites a byte in it with a zero using /usr/bin/dd against the raw disk, /dev/sdX. He then reads the file, uncorrupted, with a kernel warning about a bad checksum that was fixed (from the mirror) during the read. This demonstration is why I switched to btrfs (and ECC) for my new home server/NAS. That, along with compression sealed the deal. I was disappointed about not having dedupe at the time, but it probably really wouldn't have saved me that much space. (I keep meaning to run the dedupe tool to see what it might save me now.)
- loeg 7y ago
  Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent. This sort of gets into it but isn't a fantastic source: https://www.enterprisestorageforum.com/storage-hardware/drive-reliability-studies-2.html https://www.enterprisestorageforum.com/storage-hardware/driv... See also https://en.wikipedia.org/wiki/Data_corruption#Silent https://en.wikipedia.org/wiki/Data_corruption#Silent and https://en.wikipedia.org/wiki/Hard_disk_drive#ERRORRATESHANDLING https://en.wikipedia.org/wiki/Hard_disk_drive#ERRORRATESHAND... . Notably, Apple's new filesystem introduced in 2017 does not detect data corruption; it only checksums metadata. https://en.wikipedia.org/wiki/Apple_File_System#Data_integrity https://en.wikipedia.org/wiki/Apple_File_System#Data_integri... Apparently Apple Engineers are of the opinion that Apple SSD and HDD media simply don't have silent data corruption. They have some pretty smart engineers, so maybe they're on to something. It does beggar the question, though: why have metadata checksums at all? http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ Anyway, I'm not exactly supporting threat models that ignore silent data corruption. Just pointing out that they're not exactly as ridiculous as they might seem on their face.
cmurf 7y ago
The journal contains information that can be replayed on a crash to hopefully avoid an file system check and repair to make the file system consistent. It's intended to address how long fsck can take as file systems get bigger. It has nothing to do with data integrity per se. Recently ext4 adds journal and metadata checksums (feature metadata_csum) to detect corruption, but data is not checksummed by the file system. Each drive sector is checksummed. The expectation by ext4 (and XFS) when it comes to data integrity is that it's the responsibility of the hardware or some other layer, like dm-integrity. The data integrity problem is you don't have a way of unambiguously determining data integrity. You have to trust the hardware. Obviously that's good enough most of the time, but it can transiently misstep without outright failing: bit flips, torn or misdirected writes. By the way, make sure you check for timeout mismatches if you're using consumer drives. Affects Btrfs, mdadm and LVM raid, and I assume ZFS on Linux. https://raid.wiki.kernel.org/index.php/Timeout_Mismatch https://raid.wiki.kernel.org/index.php/Timeout_Mismatch

cthalupa 7y ago

>I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for. Nothing in your setup there protects you from bitrot. >With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? Drive rebuild time is primarily a function of a variety of things - speed of the disks, processing power involved, etc. There's nothing magic about any solution here. ZFS does get a leg up in that it knows what blocks are in use, but hardware raid and mdadm are quite fast when they find empty blocks too, so it's not some overwhelmingly massive improvement. >Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems. I think you're fundamentally confused about what journaling provides a filesystem. A journal helps you in situations where your system crashes, loses power, etc. and you had not finished committing your changes to the filesystem. It does not protect you from data integrity issues that are unrelated to this sort of situation - a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal.

anon9001 7y ago
> a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot. Is the difference that ZFS scrubs in real-time? Sorry if I'm being dense here, but I did a fair amount of reading about this, and I'm trying to understand what I'm missing.
- cthalupa 7y ago
  >Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot. Nope. Parity bits aren't used to detect bitrot. If you get a URE when you go to read, the parity bits will be used to get you data. If you read successfully, nothing happens at all. If you scrub, and the data is found to be inconsistent, md assumes that the parity is wrong and overwrites it. https://linux.die.net/man/4/md https://linux.die.net/man/4/md >If a read error is detected during this process, the normal read-error handling causes correct data to be found from other devices and to be written back to the faulty device. In many case this will effectively fix the bad block. >If all blocks read successfully but are found to not be consistent, then this is regarded as a mismatch. >If check was used, then no action is taken to handle the mismatch, it is simply recorded. If repair was used, then a mismatch will be repaired in the same way that resync repairs arrays. For RAID5/RAID6 new parity blocks are written. For RAID1/RAID10, all but one block are overwritten with the content of that one block.
  - anon9001 7y ago
    Thanks for the explanation. I didn't realize md didn't use the parity checks unless there's a URE. Is it correct to say that ZFS parity is used to reconcile on each read even without UREs, but with md parity is only used when there are UREs? If that's true, it makes much more sense why ZFS is considered the right choice. Also, do you know if adding luks in the mix changes that? My data goes md->lvm->luks->ext4 and I hadn't considered which part of that is supposed to catch the error if it wasn't a URE. It makes sense that md would update parity if there was no URE. It also makes sense that LVM wouldn't know or care about data integrity. I would think luks would require data integrity to work properly, but I'm not sure what happens if there is bitrot in a luks block. I'm also surprised ext4 doen't have checksumming, but apparently it's only for able to checksum for metadata and it's disabled by default: https://serverfault.com/questions/456319/how-to-get-checksum-data-integrity-on-ext4 https://serverfault.com/questions/456319/how-to-get-checksum... Dm-integrity sounds worth a look, but https://raid.wiki.kernel.org/index.php/Dm-integrity https://raid.wiki.kernel.org/index.php/Dm-integrity doesn't inspire much confidence.

saalweachter 7y ago

I went ~2 years without running a scrub when I first started using ZFS many years ago; when I ran my first scrub, it found 2 file corruptions, one in a random picture back when I still copied pictures from a digital camera to a hard drive, one in a random file in the browser cache. (If I'd been scrubbing regularly, these almost certain would have been found before corruption crept into the mirrored copy of the file, and been recoverable.)

cmurf 7y ago
This is a great point about how pernicious corruption is: it replicates. Detection is more important than correction, at least you can stop the corruption from spreading. It's typical for people to reuse storage hardware, and each derivative inherits prior corruption as well as contributing its own. Most people don't keep a long term unchanging archive, and even that archive likely has no means of unambiguous verification of data integrity.
- saalweachter 7y ago
  To be honest I'm not sure if the corruption was replicated directly, or if it's just that the same file got unlucky on two different drives -- I can't recall if this was before or after a drive failure.

Quekid5 7y ago

Your setup won't save you from e.g. bad cables silently corrupting data, or drives simply returning bad data, etc. etc. Btw, journaling has nothing to do with integrity, though I believe it is checksummed these days.

anon9001 7y ago
> Your setup won't save you from e.g. bad cables silently corrupting data, or drives simply returning bad data, etc. etc. I thought this was what RAID was designed to prevent. I don't know what a cable failure would look like in my setup in practice, but I would think the disk would start failing parity checks against the rest of the array and be flagged as bad.
- cthalupa 7y ago
  Well, each RAID implementation has different designs and goals in mind, but the original design was simply meant to replace expensive single drives. Using more spindles you could get more performance out of a bunch of cheap disks, and to help with the fact that these cheap disks failed more often, you made them redundant. In general, dealing with data corruption that does not generate an error is not within the purview of RAID implementations. ZFS and btrfs handle it because they're more than just RAID implementations. mdadm can do similar things in a RAID setup if dm-integrity is used on all of the drives (but it is not used by default when just setting up mdadm/lvm/etc.)
- Quekid5 7y ago
  RAID is primarily about redundancy and speed, not really end-to-end integrity. (Though some setups will certainly have some measures against corruption.) ZFS is really designed for end-to-end integrity. Once you're stored data on ZFS, you're basically guaranteed that you'll always get the exact same thing back (or an error)... no matter where in the "chain" of I/O the failure occurred.
rodgerd 7y ago
Or a power supply problem, which was a source of great pain for me at one point.

ilikejam 7y ago

On the last round of zpool scrubs of the 13PB-used set of zfs filesystems here (raidz3, mostly SAS drives, all hosts with ECC RAM), 50MB of data was repaired. 50MB isn't much compared with 13PB, but silent corruption is a real thing.

anon9001 7y ago
That makes sense, but mdadm also scrubs weekly on a cronjob. Is there something about ZFS that makes corruption more recoverable than mdadm's RAID?
- InvaderFizz 7y ago
  One thing is that ZFS tells you exactly what file is corrupted. I recently took over a ZFS setup that was built with plain ZFS on top of a 24-drive wide RAID6. Drives failed, rebuilt the RAID6, nothing appeared to be wrong. ZFS scrub revealed two corrupted files. I was able to pull backups of those two files, replace them on the FS, and rerun the scrub. Obviously I moved off the RAID6 soon after, but this is a powerful example of why the checksuming of ZFS is so useful.
- ilikejam 7y ago
  RAID5 corruption is non-recoverable - you can't tell which is right, the data or the parity. RAID6 is recoverable - you can get consensus between the data and the two parity bits. ZFS stores hashes of each block, so as long as you have two copies, you can just pick the data+hash that's correct and restore the corrupt block, even with only two copies of the data.
- saltcured 7y ago
  The core ZFS feature being references here is storage checksums. The general argument is that as storage volumes have increased in size and/or workload, the conventional checksum methods have not been changed to increase their reliability in a way that prevents real world corruption on real world storage during their real world lifetimes. A RAID system can only reliably recover when an IO error is reported. In this case, it can assume which element of the stored data is no longer trustworthy and use the other parts to reconstruct it. A RAID scrub operation that finds inconsistencies, without a reported IO error, cannot really determine why it is inconsistent. It doesn't know which element is corrupt and which other elements to use to reconstruct it. In effect, ZFS adds another data integrity layer over the physical storage, so you can have a lower probability of silent failures where the disk controller or IO channels do their own "parity" or other checks and report a successful read when they actually returned corrupt values. Something like mdadm or LVM would need to add a new layer of checksums over the physical devices to provide this level of protection. Each block/chunk/stripe written to disk would need another integrity checksum written as well. During operation, these checksums would be verified before deciding that a read operation was actually successful. If the verification fails, a virtual IO error is synthesized to flag the read as bad, so that the other RAID recovery techniques can then be applied while knowing which block/chunk/stripe is invalid.

throw0101a 7y ago

> With RAID6 I can have 2 drives fail without data loss. If you don't have checksums on your data, how do you know it's good (even without drive failures)? Your RAID rebuild could be recreating junk. > Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems. Journaling helps with speeding up post-crash reboots: instead of doing a full fsck, it just replays the log.

iRobbery 7y ago
"Your RAID rebuild could be recreating junk.." From experience i can tell this can happen. nervous trauma twitch

JackMcMack 7y ago

The issue is not a loudly failing disk. If you get read errors, or if the disk is simply offline, it's easy to recalculate the data. But what if you have silent data corruption: a single bit? Even if you have parity (raid5) or a duplicate (raid1), how do you know which part is correct, and which part to repair? Without checksums. I've had to restore 1TB from Crashplan. They apparently had checksums in their software, and something like 10-15 files were damaged. Without checksums, you simply don't know if your data is damaged, even on a single hard drive. ZFS gives you data integrity with checksums, and redundancy with mirrors or raid-z. This is why a regular scrub is recommended, you verify the checksums and repair if possible.

rrauenza 7y ago
I was won over to btrfs for my new NAS (~6 years ago!) when I saw a demo (which I expect is also valid on zfs) where the presenter starts with a mirrored setup, finds the exact block offset location of the file's contents and then overwrites a byte in it with a zero using /usr/bin/dd against the raw disk, /dev/sdX. He then reads the file, uncorrupted, with a kernel warning about a bad checksum that was fixed (from the mirror) during the read. This demonstration is why I switched to btrfs (and ECC) for my new home server/NAS. That, along with compression sealed the deal. I was disappointed about not having dedupe at the time, but it probably really wouldn't have saved me that much space. (I keep meaning to run the dedupe tool to see what it might save me now.)
loeg 7y ago
Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent. This sort of gets into it but isn't a fantastic source: https://www.enterprisestorageforum.com/storage-hardware/drive-reliability-studies-2.html https://www.enterprisestorageforum.com/storage-hardware/driv... See also https://en.wikipedia.org/wiki/Data_corruption#Silent https://en.wikipedia.org/wiki/Data_corruption#Silent and https://en.wikipedia.org/wiki/Hard_disk_drive#ERRORRATESHANDLING https://en.wikipedia.org/wiki/Hard_disk_drive#ERRORRATESHAND... . Notably, Apple's new filesystem introduced in 2017 does not detect data corruption; it only checksums metadata. https://en.wikipedia.org/wiki/Apple_File_System#Data_integrity https://en.wikipedia.org/wiki/Apple_File_System#Data_integri... Apparently Apple Engineers are of the opinion that Apple SSD and HDD media simply don't have silent data corruption. They have some pretty smart engineers, so maybe they're on to something. It does beggar the question, though: why have metadata checksums at all? http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ Anyway, I'm not exactly supporting threat models that ignore silent data corruption. Just pointing out that they're not exactly as ridiculous as they might seem on their face.

cmurf 7y ago

The journal contains information that can be replayed on a crash to hopefully avoid an file system check and repair to make the file system consistent. It's intended to address how long fsck can take as file systems get bigger. It has nothing to do with data integrity per se. Recently ext4 adds journal and metadata checksums (feature metadata_csum) to detect corruption, but data is not checksummed by the file system. Each drive sector is checksummed. The expectation by ext4 (and XFS) when it comes to data integrity is that it's the responsibility of the hardware or some other layer, like dm-integrity. The data integrity problem is you don't have a way of unambiguously determining data integrity. You have to trust the hardware. Obviously that's good enough most of the time, but it can transiently misstep without outright failing: bit flips, torn or misdirected writes. By the way, make sure you check for timeout mismatches if you're using consumer drives. Affects Btrfs, mdadm and LVM raid, and I assume ZFS on Linux. https://raid.wiki.kernel.org/index.php/Timeout_Mismatch https://raid.wiki.kernel.org/index.php/Timeout_Mismatch