4 ms·

Corrupting a ZFS File on Purpose

85 points by zdw 4mo ago

anonymous_user9 4mo ago
> The DVA was correct, the sector math was correct, the dd command was correct. The right place, the wrong mental model. God the intensity is tiresome. Whether or not it's AI slop, it's also bad writing. Things can be fun or interesting or worthwhile without being a harrowing battle of discovery!
- calcifer 4mo ago
  > Things can be fun or interesting or worthwhile without being a harrowing battle of discovery! The quoted sentences used "correct", "right" and "wrong". Hardly the sensationalist words you're implying.
  - eigencoder 4mo ago
    I get what they're saying, it's more about the punchy tone than the word choice
  - rcxdude 4mo ago
    It's not the word choice, it's the whole tone and structure of the sentence. It reads like a horror writer building up the tension before a big reveal but it just keeps drawing it out over a whole article and for something that isn't worth the build-up. It gets quite tiring to read IMO (LLM writing in general tends to have a grandiosity to it which really grates with something which is meant to be more informative, in my experience. They will explain a section of tax law like it's the second coming of Christ).
ralferoo 4mo ago
Hmmm, it's been a long long time since I actually had a failed drive (and also I don't use zfs), but from what I remember of my last failing drive 20 years ago, the drive was able to detect that sectors had been corrupted, and then failed the read rather than just returning silently corrupted data. If my memory is correct, replacing random bytes on disk wouldn't actually reflect the typical way data corruption manifests itself. I always thought that the reason zfs did its extensive CRC checks was primarily to detect data corruption while it was in RAM or over the network, with a side effect that in the rare cares that data on disk got corrupted without the drive detecting it because the CRC was still valid, it'd also be spotted. But anyway, it might be worth testing by replacing some of the disk images with actually truncated ones so that there are holes when reading, so that it returns an actual read error rather than junk data.
- adrian_b 4mo ago
  The error-correcting codes used by HDDs/SSDs correct or detect the most frequent errors, but sometimes, when there are too many erroneous bits in a sector, they can mis-correct the data and then the HDD/SSD returns a corrupted sector without signaling any error. I have seen this a few times on HDDs that had been used for the cold storage of archival data, for several years (around 5 years or even more). For each archive file, I had my own hash values that were used to detect corrupted files, which allowed me to detect all such cases. I had duplicates for all such HDDs. Sometimes both HDD copies had a few silent corrupted sectors, but they were not in the same locations, so in all cases I could recover the corrupted files from their duplicates. If I had stored the archival data without redundancy, I would have lost it. If you do not use hashes or other error-detecting codes for all your files, like I do, you may have had some failures in your HDDs without recognizing them, but such errors are much more likely to happen in files that have been stored for many years.
  - ramses0 4mo ago
    And/Or: `*.par` files. https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive
    - wongarsu 4mo ago
      Or rar files with recovery records. Same concept, but in one self-contained file instead of a number of sidecar files
    - adrian_b 4mo ago
      Yes, already for many years, I have also used par2create/par2verify for adding redundancy to archive files and repairing any corrupted files. However, I use both par2create and duplicate storage media, because duplicates that are preferably stored in different geographic locations are the only solution that guards against incidents so serious that they would destroy partially or totally the storage device. By itself, when an adequate amount of added redundancy is chosen, par2create is sufficient to recover archive files that are only affected by a few sporadic corrupted sectors, like on a HDD that has been stored in good conditions for some years. It will not help if the entire HDD becomes unusable, due to some mechanical or electrical defect, which may happen in HDDs used for cold storage, instead of being used continuously.
lanycrost 4mo ago
I miss ZFS, only had a chance once to work with it in production and liked it very much. It's have performance overhead compared to journal filesystems but greatly designed.
igtztorrero 4mo ago
I always run my servers on zfs pool mirrored using raid1 on 2 nvme drives, because when nvme fails, fail completely. How can a File be corrupted on normal operations?
- rincebrain 4mo ago
  Various ways. Drives sometimes are worse at their internal error detection than you might hope, and might return incorrect bytes. You might have faulty hardware flipping bits between when you computed a checksum/parity/etc over your data and when you wrote it out, either in memory, or over the wire. You might have a software bug or an interaction with a hardware erratum that causes the CPU to misbehave and mangle your bits in certain cases, maybe around switching from running code in a VM to not and back. You might have had, say, the Samsung HD204UI hard drive, which loses data after it tells the OS that it's written because of a bug around its write cache, so you get no error back, but you go to read the data back later and it's actually whatever was there before you tried overwriting it. SSDs, NVMe and otherwise, _can_ fail in ways that aren't just vanishing from the bus, it's just much less common than with mechanical drives, IME. I have sometimes seen SSDs return incorrect bytes inconsistently or consistently, or start spitting up read/write errors rather than entirely vanishing from the bus. Each of the above examples is a real thing I saw happen. None of them is particularly likely, plenty of people never have dumb shit like any of that come up. But it's not never.
guardiangod 4mo ago
I ran 5 external USB + SMR hard disks in RAIDZ 5 for 10 years. The only thing I had to change was to use Highpoint's enterprise level USB controllers- commercial USB controllers from Realtek and Renasus are junk and will drop the drives after a while. Even then, I had multiple cases where files were corrupted, and once the whole array refused to be online due to corrupted metadata. I had to make ZFS to replay the journal log with undocumented commands. Sometimes it takes a few days of hair-rising recovery but I always manage to get the array back intact. The files that are corrupted are always extremely large files (>50 GB) with many small read/writes (eg. iSCSI image files.) It's pretty impressive how resilient ZFS is, really, given I had what likely to be the worst possible hardware combination.
- BuildTheRobots 4mo ago
  Out of curiosity, why were you using a >50gb file on a dataset as as iSCSI target rather than a zvol or did I misunderstand?
  - oasisaimlessly 4mo ago
    Why use zvols? Aren't they essentially just single-file ZFS datasets (allowing e.g. independent snapshotting)?
    - ssl-3 4mo ago
      They are as you describe. Except zvols present as real-live block devices that can do block-device things instead of regular-file things, and that's important for some stuff. But AFAICT, iSCSI targets on Linux are not one of those things. They don't care; they work the ~same whether backed by files or block devices. And on the performance benchmarks I find that compare performance of zvols-vs-files on ZFS, files usually win. > Why use zvols? Probably for the same reasons that people recommended separate disk partitions for /var, /usr, and such as was the case ~30 years ago when I got started with desktop *nix systems. That reason seemed to boil down to: "If it was good for a Sun/3 in 1986, then it must also be good for a Linux box in 1996." It was a dumb reason. tl;dr, folklore. :)
xeyownt 4mo ago
Nice writeup. This is this kind of exercise and information that really help you understand better how things work.
xk3 4mo ago
I've done something similar before with Btrfs https://gist.github.com/chapmanjacobd/bc6e31c8bc3647e0bcb0c43bc0464a9c https://gist.github.com/chapmanjacobd/bc6e31c8bc3647e0bcb0c4... pretty fun!

85 points by zdw 4mo ago

> The DVA was correct, the sector math was correct, the dd command was correct. The right place, the wrong mental model. God the intensity is tiresome. Whether or not it's AI slop, it's also bad writing. Things can be fun or interesting or worthwhile without being a harrowing battle of discovery!

calcifer 4mo ago
> Things can be fun or interesting or worthwhile without being a harrowing battle of discovery! The quoted sentences used "correct", "right" and "wrong". Hardly the sensationalist words you're implying.
- eigencoder 4mo ago
  I get what they're saying, it's more about the punchy tone than the word choice
- rcxdude 4mo ago
  It's not the word choice, it's the whole tone and structure of the sentence. It reads like a horror writer building up the tension before a big reveal but it just keeps drawing it out over a whole article and for something that isn't worth the build-up. It gets quite tiring to read IMO (LLM writing in general tends to have a grandiosity to it which really grates with something which is meant to be more informative, in my experience. They will explain a section of tax law like it's the second coming of Christ).

ralferoo 4mo ago

Hmmm, it's been a long long time since I actually had a failed drive (and also I don't use zfs), but from what I remember of my last failing drive 20 years ago, the drive was able to detect that sectors had been corrupted, and then failed the read rather than just returning silently corrupted data. If my memory is correct, replacing random bytes on disk wouldn't actually reflect the typical way data corruption manifests itself. I always thought that the reason zfs did its extensive CRC checks was primarily to detect data corruption while it was in RAM or over the network, with a side effect that in the rare cares that data on disk got corrupted without the drive detecting it because the CRC was still valid, it'd also be spotted. But anyway, it might be worth testing by replacing some of the disk images with actually truncated ones so that there are holes when reading, so that it returns an actual read error rather than junk data.

adrian_b 4mo ago
The error-correcting codes used by HDDs/SSDs correct or detect the most frequent errors, but sometimes, when there are too many erroneous bits in a sector, they can mis-correct the data and then the HDD/SSD returns a corrupted sector without signaling any error. I have seen this a few times on HDDs that had been used for the cold storage of archival data, for several years (around 5 years or even more). For each archive file, I had my own hash values that were used to detect corrupted files, which allowed me to detect all such cases. I had duplicates for all such HDDs. Sometimes both HDD copies had a few silent corrupted sectors, but they were not in the same locations, so in all cases I could recover the corrupted files from their duplicates. If I had stored the archival data without redundancy, I would have lost it. If you do not use hashes or other error-detecting codes for all your files, like I do, you may have had some failures in your HDDs without recognizing them, but such errors are much more likely to happen in files that have been stored for many years.
- ramses0 4mo ago
  And/Or: `*.par` files. https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive
  - wongarsu 4mo ago
    Or rar files with recovery records. Same concept, but in one self-contained file instead of a number of sidecar files
  - adrian_b 4mo ago
    Yes, already for many years, I have also used par2create/par2verify for adding redundancy to archive files and repairing any corrupted files. However, I use both par2create and duplicate storage media, because duplicates that are preferably stored in different geographic locations are the only solution that guards against incidents so serious that they would destroy partially or totally the storage device. By itself, when an adequate amount of added redundancy is chosen, par2create is sufficient to recover archive files that are only affected by a few sporadic corrupted sectors, like on a HDD that has been stored in good conditions for some years. It will not help if the entire HDD becomes unusable, due to some mechanical or electrical defect, which may happen in HDDs used for cold storage, instead of being used continuously.

lanycrost 4mo ago

I miss ZFS, only had a chance once to work with it in production and liked it very much. It's have performance overhead compared to journal filesystems but greatly designed.

igtztorrero 4mo ago

I always run my servers on zfs pool mirrored using raid1 on 2 nvme drives, because when nvme fails, fail completely. How can a File be corrupted on normal operations?

rincebrain 4mo ago
Various ways. Drives sometimes are worse at their internal error detection than you might hope, and might return incorrect bytes. You might have faulty hardware flipping bits between when you computed a checksum/parity/etc over your data and when you wrote it out, either in memory, or over the wire. You might have a software bug or an interaction with a hardware erratum that causes the CPU to misbehave and mangle your bits in certain cases, maybe around switching from running code in a VM to not and back. You might have had, say, the Samsung HD204UI hard drive, which loses data after it tells the OS that it's written because of a bug around its write cache, so you get no error back, but you go to read the data back later and it's actually whatever was there before you tried overwriting it. SSDs, NVMe and otherwise, _can_ fail in ways that aren't just vanishing from the bus, it's just much less common than with mechanical drives, IME. I have sometimes seen SSDs return incorrect bytes inconsistently or consistently, or start spitting up read/write errors rather than entirely vanishing from the bus. Each of the above examples is a real thing I saw happen. None of them is particularly likely, plenty of people never have dumb shit like any of that come up. But it's not never.

guardiangod 4mo ago

I ran 5 external USB + SMR hard disks in RAIDZ 5 for 10 years. The only thing I had to change was to use Highpoint's enterprise level USB controllers- commercial USB controllers from Realtek and Renasus are junk and will drop the drives after a while. Even then, I had multiple cases where files were corrupted, and once the whole array refused to be online due to corrupted metadata. I had to make ZFS to replay the journal log with undocumented commands. Sometimes it takes a few days of hair-rising recovery but I always manage to get the array back intact. The files that are corrupted are always extremely large files (>50 GB) with many small read/writes (eg. iSCSI image files.) It's pretty impressive how resilient ZFS is, really, given I had what likely to be the worst possible hardware combination.

BuildTheRobots 4mo ago
Out of curiosity, why were you using a >50gb file on a dataset as as iSCSI target rather than a zvol or did I misunderstand?
- oasisaimlessly 4mo ago
  Why use zvols? Aren't they essentially just single-file ZFS datasets (allowing e.g. independent snapshotting)?
  - ssl-3 4mo ago
    They are as you describe. Except zvols present as real-live block devices that can do block-device things instead of regular-file things, and that's important for some stuff. But AFAICT, iSCSI targets on Linux are not one of those things. They don't care; they work the ~same whether backed by files or block devices. And on the performance benchmarks I find that compare performance of zvols-vs-files on ZFS, files usually win. > Why use zvols? Probably for the same reasons that people recommended separate disk partitions for /var, /usr, and such as was the case ~30 years ago when I got started with desktop *nix systems. That reason seemed to boil down to: "If it was good for a Sun/3 in 1986, then it must also be good for a Linux box in 1996." It was a dumb reason. tl;dr, folklore. :)

xeyownt 4mo ago

Nice writeup. This is this kind of exercise and information that really help you understand better how things work.

xk3 4mo ago

I've done something similar before with Btrfs https://gist.github.com/chapmanjacobd/bc6e31c8bc3647e0bcb0c43bc0464a9c https://gist.github.com/chapmanjacobd/bc6e31c8bc3647e0bcb0c4... pretty fun!