5 ms·

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (called cigar strings) and I am getting more and more convinced that the discrepancies I run into are actually bit flips on the hardware. Anybody else have any experience with this? BTRFS would be a good tool to prevent such bitrot, imo...

sethhochberg 6y ago
Depending on how you store this data (the number of physical disks involved) you may also want to explore ZFS. ZFS and BTRFS have pretty comparable features, but given how battle-hardened ZFS is, and its ever-increasing integration into the Linux ecosystem, I'm seeing less and less reason to prefer BTRFS. Both do checksumming and regular scrubbing to help detect bitrot, but both need sufficient redundancy configured to actually be able to repair the corrupted data when it is detected.
- ZFSKing 6y ago
  That, this, is the most diplomatical way of saying 'just use ZFS' we've seen in a decayed. Mod parent up. BTRFS will never be a suck-zess. Use ext4 or ZFS for life.
- hda111 6y ago
  There is exactly one distro that comes with ZFS, so it's a bit exaggeration to say it's well integrated with Linux, considering how much effort is applied by kernel developers to make it even harder to use ZFS. And this one distro that includes ZFS is just one lawsuit away from removing it again.
  - jacob019 6y ago
    It's easy to use zfs on Debian and derivatives apt install dkms spl-dkms apt install zfs-dkms zfsutils-linux
  - enz 6y ago
    I am testing OpenZFS with Ubuntu 20.04 LTS right now. OpenZFS seems pretty solid so far (I tried various scenarios of failure/corruption) and Ubuntu seems to actually care about ZFS support and its integration in Linux.
  - projct 6y ago
    Ubuntu, Proxmox, Manjaro all have root on ZFS via the installer and there are others that I don't remember offhand.
_frkl 6y ago
Or ZFS, I wouldn't store that kind of data on other filesystem's without that kind of protection...
Volundr 6y ago
Bit-flips are a lot more common than people think as are read errors on drives. Surprisingly, usually a bit flipping here or there doesn't cause any real problems so they tend to go unnoticed. If your working with TBs of data using a checksumming filesystem like ZFS of BTRFS would certainly be recommended, but you should ALSO make sure your using ECC RAM. Checking the integrity of the data as it comes off the disk doesn't do you much good if a bit then flips while it's in memory.
- oarsinsync 6y ago
  To emphasise, using ECC RAM is another step that can be used to increase the integrity of your data. It’s like security - it’s done in layers. Using a checksumming filesystem without ECC RAM is an improvement on not using a checksumming filesystem at all. Do not consider ECC RAM to be a requirement for ZFS / BTRFS. It does add another layer of improvement though, so if you can, you should! And even if you can’t, and even if you can’t provide the redundancy to enable bitflips to be resolved, still use a checksumming filesystem if you can!
- zionic 6y ago
  DDR5 mainstreaming ECC RAM as the default is super exciting!
dsr_ 6y ago
I once ran into a RAID card that would write bad data if you put it on a 90-degree PCI connector (needed for a 2U server box). No errors appeared anywhere until you encountered corrupted files.
- bashinator 6y ago
  Hardware RAID falls into the "never again" category for me.
  - nitrogen 6y ago
    Though in this case it sounds like any PCI card that couldn't handle the degraded signals from the angle connector would have exhibited data corruption. I have problems with USB packets getting corrupted on their way to my audio interface. Signal integrity is really underappreciated in the PC peripheral world.
    - bashinator 6y ago
      I think /dev/md raid would have made that failure more noticeable, more quickly. Hardware RAID cards don’t always pass on disk errors to the kernel.
the8472 6y ago
btrfs definitely helps catching errors, even without ECC or redundancy. An example: On a server with multiple VMs on a big consumer NVMe that was starting to get unreliable Windows reported nothing other than bluescreening occasionally (sfc and chkdsk showed no errors). Linux on the other hand reported errors during a bi-weekly scrub and in the kernel logs. The SMART thresholds on the drive were strange too. A small but non-zero amount of read errors was not considered a health problem yet. I only realized that after the scrub reported problems.
- curt15 6y ago
  >On a server with multiple VMs on a big consumer NVMe How did you avoid the crippling performance penalty with storing VM disks on btrfs? The usual workaround suggested to btrfs users is to disable copy-on-write for VM images, but doing so also turns off checksumming thus disabling the very features that make btrfs, well, btrfs.
  - the8472 6y ago
    The VMs were running on raw partitions. The btrfs reporting errors was inside a linux VM.
ksk 6y ago
What software do you use ? We use bwa/samtools/varscan/bcftools and we've reprocessed old data right from the fastq files a few times. So far we haven't gotten different results.
- teekert 6y ago
  I encounter an error that I can't currently find exactly but it happens when running mismatch_profile from the RSeQC package. It reports a mismatch in the length of the cigar notation and the actual read inside a BAM file. Lately I have become 99% sure that this sometimes happens in files that did not show this error before (we reprocess a lot of data from time to time). I suspect it is bit flips because I think bit flips can easily go unnoticed, but not in this case, because the cigar notation is a shorter description of the actual read and they should thus match. The mismatch signals a change in the file that occurred without us touching it.
  - ksk 6y ago
    Hmm, yeah that is odd! Maybe its a bug in the alignment tool?
    - teekert 6y ago
      It's not reproducible and I say I'm 99% sure it comes from bit flips is because I know with 99% certainty the process did finish successfully on a file at some point but then failed several weeks later, on the same file.
nitrogen 6y ago
I have a lot of video files (e.g. old camera tapes) stored on spinning hard drives. Bit flips are distressingly common, and increase with age. My next storage setup will definitely use a checksumming filesystem.
pstrateman 6y ago
btrfs has eaten 100% of the data I have stored on it. 0/10 would not suggest. ZFS has similar features and has eaten 0% of the data I have stored on it. would suggest.
- teekert 6y ago
  n=1 is n=None ;)
enz 6y ago
As other folks are suggesting, ECC may be suitable for you (alongside a protective FS). But if you want to purchase your next workstation with ECC support, be very careful: some motherboards manufacturers just say 'ECC' in their specs but that doesn't mean that they actually use ECC capabilities, that only means that you can plug ECC-RAM into it. Make sure the specs explicitly say something like "Corrects single-bit errors and detects double-bit errors" (you need both, some MBs support only single-bit correction). I think ECC RAM and a good FS are complementary tools to achieve storage high reliability. One without another may not be that useful.
08-15 6y ago
Err, no. The CIGAR is not a checksum, even though you'll notice if it's corrupted, because it won't match the read length anymore. However, BAM is zlib-compressed (even if you asked samtools not to compress it). Each block therefore contains an actual checksum, and corruption will show up during decompression. Therefore, if you can decompress BAM, but the CIGAR doesn't make sense, you've got a software problem, not a hardware problem. From my experience in beerinformatics, you got some software that... uhhh... interprets the BAM specification differently than you do. It's depressingly common.
- teekert 6y ago
  Hmm, thank you for the information. But I'm by now pretty sure that I saw the checking process finish successfully at one moment, and not at others. It's not reproducible and all those algorithms should be deterministic, right? If not for bit flipping I really can't explain this.
  - 08-15 6y ago
    It really can't be bit flipping on disk. First off, the checksum would catch it. Second, the file is compressed, so flipping a bit turns it into complete garbage. Third, BAM uses a length-prefixed coding in several places. A flipped bit in any length field would mess up the decoding. Those bits could still flip in memory, which isn't BTRFS's fault. That's still unlikely, because you'd typically get segfaults, not silent corruption. So, it's probably bad software. > those algorithms should be deterministic, right? Combine bad code (samtools) with multithreading, and you will quickly become convinced that your computer is possessed by an evil spirit.
    - teekert 6y ago
      :D Currently not using btrfs by the way. But you are positive that BAM files should be self consistent? And when would you catch a bitflip then? What would it look like in a compressed file lake BAM? A much larger effect than just a cigar length mismatch supposedly?
rleigh 6y ago
I'd personally go with ZFS. There are already people in your field using ZFS to ensure the integrity of their data (based on past conversations here on HN). I've used both ZFS and Btrfs extensively. Only Btrfs has lost data and behaved badly. I would highly recommend evaluating ZFS.

sethhochberg 6y ago

Depending on how you store this data (the number of physical disks involved) you may also want to explore ZFS. ZFS and BTRFS have pretty comparable features, but given how battle-hardened ZFS is, and its ever-increasing integration into the Linux ecosystem, I'm seeing less and less reason to prefer BTRFS. Both do checksumming and regular scrubbing to help detect bitrot, but both need sufficient redundancy configured to actually be able to repair the corrupted data when it is detected.

ZFSKing 6y ago
That, this, is the most diplomatical way of saying 'just use ZFS' we've seen in a decayed. Mod parent up. BTRFS will never be a suck-zess. Use ext4 or ZFS for life.
hda111 6y ago
There is exactly one distro that comes with ZFS, so it's a bit exaggeration to say it's well integrated with Linux, considering how much effort is applied by kernel developers to make it even harder to use ZFS. And this one distro that includes ZFS is just one lawsuit away from removing it again.
- jacob019 6y ago
  It's easy to use zfs on Debian and derivatives apt install dkms spl-dkms apt install zfs-dkms zfsutils-linux
- enz 6y ago
  I am testing OpenZFS with Ubuntu 20.04 LTS right now. OpenZFS seems pretty solid so far (I tried various scenarios of failure/corruption) and Ubuntu seems to actually care about ZFS support and its integration in Linux.
- projct 6y ago
  Ubuntu, Proxmox, Manjaro all have root on ZFS via the installer and there are others that I don't remember offhand.

_frkl 6y ago

Or ZFS, I wouldn't store that kind of data on other filesystem's without that kind of protection...

Volundr 6y ago

Bit-flips are a lot more common than people think as are read errors on drives. Surprisingly, usually a bit flipping here or there doesn't cause any real problems so they tend to go unnoticed. If your working with TBs of data using a checksumming filesystem like ZFS of BTRFS would certainly be recommended, but you should ALSO make sure your using ECC RAM. Checking the integrity of the data as it comes off the disk doesn't do you much good if a bit then flips while it's in memory.

oarsinsync 6y ago
To emphasise, using ECC RAM is another step that can be used to increase the integrity of your data. It’s like security - it’s done in layers. Using a checksumming filesystem without ECC RAM is an improvement on not using a checksumming filesystem at all. Do not consider ECC RAM to be a requirement for ZFS / BTRFS. It does add another layer of improvement though, so if you can, you should! And even if you can’t, and even if you can’t provide the redundancy to enable bitflips to be resolved, still use a checksumming filesystem if you can!
zionic 6y ago
DDR5 mainstreaming ECC RAM as the default is super exciting!

dsr_ 6y ago

I once ran into a RAID card that would write bad data if you put it on a 90-degree PCI connector (needed for a 2U server box). No errors appeared anywhere until you encountered corrupted files.

bashinator 6y ago
Hardware RAID falls into the "never again" category for me.
- nitrogen 6y ago
  Though in this case it sounds like any PCI card that couldn't handle the degraded signals from the angle connector would have exhibited data corruption. I have problems with USB packets getting corrupted on their way to my audio interface. Signal integrity is really underappreciated in the PC peripheral world.
  - bashinator 6y ago
    I think /dev/md raid would have made that failure more noticeable, more quickly. Hardware RAID cards don’t always pass on disk errors to the kernel.

the8472 6y ago

btrfs definitely helps catching errors, even without ECC or redundancy. An example: On a server with multiple VMs on a big consumer NVMe that was starting to get unreliable Windows reported nothing other than bluescreening occasionally (sfc and chkdsk showed no errors). Linux on the other hand reported errors during a bi-weekly scrub and in the kernel logs. The SMART thresholds on the drive were strange too. A small but non-zero amount of read errors was not considered a health problem yet. I only realized that after the scrub reported problems.

curt15 6y ago
>On a server with multiple VMs on a big consumer NVMe How did you avoid the crippling performance penalty with storing VM disks on btrfs? The usual workaround suggested to btrfs users is to disable copy-on-write for VM images, but doing so also turns off checksumming thus disabling the very features that make btrfs, well, btrfs.
- the8472 6y ago
  The VMs were running on raw partitions. The btrfs reporting errors was inside a linux VM.

ksk 6y ago

What software do you use ? We use bwa/samtools/varscan/bcftools and we've reprocessed old data right from the fastq files a few times. So far we haven't gotten different results.

teekert 6y ago
I encounter an error that I can't currently find exactly but it happens when running mismatch_profile from the RSeQC package. It reports a mismatch in the length of the cigar notation and the actual read inside a BAM file. Lately I have become 99% sure that this sometimes happens in files that did not show this error before (we reprocess a lot of data from time to time). I suspect it is bit flips because I think bit flips can easily go unnoticed, but not in this case, because the cigar notation is a shorter description of the actual read and they should thus match. The mismatch signals a change in the file that occurred without us touching it.
- ksk 6y ago
  Hmm, yeah that is odd! Maybe its a bug in the alignment tool?
  - teekert 6y ago
    It's not reproducible and I say I'm 99% sure it comes from bit flips is because I know with 99% certainty the process did finish successfully on a file at some point but then failed several weeks later, on the same file.

nitrogen 6y ago

I have a lot of video files (e.g. old camera tapes) stored on spinning hard drives. Bit flips are distressingly common, and increase with age. My next storage setup will definitely use a checksumming filesystem.

pstrateman 6y ago

btrfs has eaten 100% of the data I have stored on it. 0/10 would not suggest. ZFS has similar features and has eaten 0% of the data I have stored on it. would suggest.

teekert 6y ago
n=1 is n=None ;)

enz 6y ago

As other folks are suggesting, ECC may be suitable for you (alongside a protective FS). But if you want to purchase your next workstation with ECC support, be very careful: some motherboards manufacturers just say 'ECC' in their specs but that doesn't mean that they actually use ECC capabilities, that only means that you can plug ECC-RAM into it. Make sure the specs explicitly say something like "Corrects single-bit errors and detects double-bit errors" (you need both, some MBs support only single-bit correction). I think ECC RAM and a good FS are complementary tools to achieve storage high reliability. One without another may not be that useful.

08-15 6y ago

Err, no. The CIGAR is not a checksum, even though you'll notice if it's corrupted, because it won't match the read length anymore. However, BAM is zlib-compressed (even if you asked samtools not to compress it). Each block therefore contains an actual checksum, and corruption will show up during decompression. Therefore, if you can decompress BAM, but the CIGAR doesn't make sense, you've got a software problem, not a hardware problem. From my experience in beerinformatics, you got some software that... uhhh... interprets the BAM specification differently than you do. It's depressingly common.

teekert 6y ago
Hmm, thank you for the information. But I'm by now pretty sure that I saw the checking process finish successfully at one moment, and not at others. It's not reproducible and all those algorithms should be deterministic, right? If not for bit flipping I really can't explain this.
- 08-15 6y ago
  It really can't be bit flipping on disk. First off, the checksum would catch it. Second, the file is compressed, so flipping a bit turns it into complete garbage. Third, BAM uses a length-prefixed coding in several places. A flipped bit in any length field would mess up the decoding. Those bits could still flip in memory, which isn't BTRFS's fault. That's still unlikely, because you'd typically get segfaults, not silent corruption. So, it's probably bad software. > those algorithms should be deterministic, right? Combine bad code (samtools) with multithreading, and you will quickly become convinced that your computer is possessed by an evil spirit.
  - teekert 6y ago
    :D Currently not using btrfs by the way. But you are positive that BAM files should be self consistent? And when would you catch a bitflip then? What would it look like in a compressed file lake BAM? A much larger effect than just a cigar length mismatch supposedly?

rleigh 6y ago

I'd personally go with ZFS. There are already people in your field using ZFS to ensure the integrity of their data (based on past conversations here on HN). I've used both ZFS and Btrfs extensively. Only Btrfs has lost data and behaved badly. I would highly recommend evaluating ZFS.