4 ms·

Err, no. The CIGAR is not a checksum, even though you'll notice if it's corrupted, because it won't match the read length anymore. However, BAM is zlib-compre

by 08-15 6y ago

Err, no. The CIGAR is not a checksum, even though you'll notice if it's corrupted, because it won't match the read length anymore. However, BAM is zlib-compressed (even if you asked samtools not to compress it). Each block therefore contains an actual checksum, and corruption will show up during decompression. Therefore, if you can decompress BAM, but the CIGAR doesn't make sense, you've got a software problem, not a hardware problem. From my experience in beerinformatics, you got some software that... uhhh... interprets the BAM specification differently than you do. It's depressingly common.

teekert 6y ago
Hmm, thank you for the information. But I'm by now pretty sure that I saw the checking process finish successfully at one moment, and not at others. It's not reproducible and all those algorithms should be deterministic, right? If not for bit flipping I really can't explain this.
- 08-15 6y ago
  It really can't be bit flipping on disk. First off, the checksum would catch it. Second, the file is compressed, so flipping a bit turns it into complete garbage. Third, BAM uses a length-prefixed coding in several places. A flipped bit in any length field would mess up the decoding. Those bits could still flip in memory, which isn't BTRFS's fault. That's still unlikely, because you'd typically get segfaults, not silent corruption. So, it's probably bad software. > those algorithms should be deterministic, right? Combine bad code (samtools) with multithreading, and you will quickly become convinced that your computer is possessed by an evil spirit.
  - teekert 6y ago
    :D Currently not using btrfs by the way. But you are positive that BAM files should be self consistent? And when would you catch a bitflip then? What would it look like in a compressed file lake BAM? A much larger effect than just a cigar length mismatch supposedly?
    - 08-15 6y ago
      Oy vey, veering off topic, but whatever... BAM is a sequence of zlib-compressed blocks. Each of them has a checksum, and any corruption gets past it with probability 2^-32. Any sane software will give you an error. When in doubt, just decompress the stupid file using gunzip or zcat, those will verify the checksum. Don't guess, check. But if you insist on using software that just ignores the checksum (I think, not even samtools, which segfaults(!) on misspelled file names(!) is that awful), all bets are off. Decompressing a corrupted file yields nonsense that doesn't even have the same length as the original data. BAM would also quickly get out of sync, and subsequent records would contain complete nonsense. You'd know if you saw that. However... > you are positive that BAM files should be self consistent? That is a loaded question. In addition to being syntactically correct, BAM files should maintain a few variants that are hard to check. They rearely do. BWA(!) produces inconsistent BAM files, and I've seen many, many scripts in the wild that produce complete bullshit that only happens to work in a very specific setting. Like I said, you're probably using some awful software.
      - teekert 6y ago
        Thank you very much, this is very helpful!

by 08-15 6y ago

teekert 6y ago

Hmm, thank you for the information. But I'm by now pretty sure that I saw the checking process finish successfully at one moment, and not at others. It's not reproducible and all those algorithms should be deterministic, right? If not for bit flipping I really can't explain this.

08-15 6y ago
It really can't be bit flipping on disk. First off, the checksum would catch it. Second, the file is compressed, so flipping a bit turns it into complete garbage. Third, BAM uses a length-prefixed coding in several places. A flipped bit in any length field would mess up the decoding. Those bits could still flip in memory, which isn't BTRFS's fault. That's still unlikely, because you'd typically get segfaults, not silent corruption. So, it's probably bad software. > those algorithms should be deterministic, right? Combine bad code (samtools) with multithreading, and you will quickly become convinced that your computer is possessed by an evil spirit.
- teekert 6y ago
  :D Currently not using btrfs by the way. But you are positive that BAM files should be self consistent? And when would you catch a bitflip then? What would it look like in a compressed file lake BAM? A much larger effect than just a cigar length mismatch supposedly?
  - 08-15 6y ago
    Oy vey, veering off topic, but whatever... BAM is a sequence of zlib-compressed blocks. Each of them has a checksum, and any corruption gets past it with probability 2^-32. Any sane software will give you an error. When in doubt, just decompress the stupid file using gunzip or zcat, those will verify the checksum. Don't guess, check. But if you insist on using software that just ignores the checksum (I think, not even samtools, which segfaults(!) on misspelled file names(!) is that awful), all bets are off. Decompressing a corrupted file yields nonsense that doesn't even have the same length as the original data. BAM would also quickly get out of sync, and subsequent records would contain complete nonsense. You'd know if you saw that. However... > you are positive that BAM files should be self consistent? That is a loaded question. In addition to being syntactically correct, BAM files should maintain a few variants that are hard to check. They rearely do. BWA(!) produces inconsistent BAM files, and I've seen many, many scripts in the wild that produce complete bullshit that only happens to work in a very specific setting. Like I said, you're probably using some awful software.
    - teekert 6y ago
      Thank you very much, this is very helpful!