4 ms·

Which part of the chain isn't protected by at least a packet CRC? I think the answer is its all covered.

nybble41 5y ago
Link-level CRCs don't protect against bit flips that happen during processing or while transitioning between links with different CRCs. For maximum end-to-end integrity you want to calculate the check value over the data in (ECC) RAM before writing it and the check value to storage, and verify the check value after reading all the data back in—this ensures that a bit-flip will be detected no matter where it occurs in the pipeline.
kevin_thibedeau 5y ago
No it isn't. At some point data is in transit in an IC without any error handling. CRCs aren't 100% reliable. Bit flips are guaranteed to happen with non-zero probability in all digital electronics. Bad data eventually makes it to persistent storage.
- StillBored 5y ago
  I don't know what parts of the storage stack you guys are working on, but I worked on enterprise storage systems we had 100% coverage in one form or another from the moment it left the RAM, via the intel interconnects, PCIe, the FC adapters which were either ECC or were using some form of data protection wrapping the entire transaction. So random bit flips in serdes/etc didn't cause problems because the higher level link protocols protected the data between the onboard ram and the endpoint, then there were the higher level protocol also providing their own data integrity. Looking at even SATA 1, if it is implemented _CORRECTLY_ you get the packet CRC protecting the link data from the point its formed to the point the endpoint verifies the result. So like ethernet sometimes it doesn't matter if some random piece of junk in the process doesn't do its own ECC validation, because its covered under a higher level of the stack. If your adapters are "desktop" grade then I might consider seeking another vendor if you care about data integrity, some vendors are definitely shipping crap, but there are vendors I can assure you will detect link/etc failures. And as a side, note, I've seen a lot of bad data, a huge percentage of it, were kernel/etc filesystem errors. We added a bunch of out of band extra metrics to track when/where writes were going and our own metadata layers/etc, and it uncovered a whole bunch of software errors.
- nix23 5y ago
  ZFS can easily protect you from that: >>it is designed with a focus on data integrity by protecting the user's data on disk against silent data corruption caused by data degradation, power surges (voltage spikes), bugs in disk firmware, phantom writes (the previous write did not make it to disk), misdirected reads/writes (the disk accesses the wrong block), DMA parity errors between the array and server memory or from the driver (since the checksum validates data inside the array), driver errors (data winds up in the wrong buffer inside the kernel), accidental overwrites (such as swapping to a live file system), etc. https://en.wikipedia.org/wiki/ZFS#Data_integrity https://en.wikipedia.org/wiki/ZFS#Data_integrity

Link-level CRCs don't protect against bit flips that happen during processing or while transitioning between links with different CRCs. For maximum end-to-end integrity you want to calculate the check value over the data in (ECC) RAM before writing it and the check value to storage, and verify the check value after reading all the data back in—this ensures that a bit-flip will be detected no matter where it occurs in the pipeline.

kevin_thibedeau 5y ago

No it isn't. At some point data is in transit in an IC without any error handling. CRCs aren't 100% reliable. Bit flips are guaranteed to happen with non-zero probability in all digital electronics. Bad data eventually makes it to persistent storage.

StillBored 5y ago
I don't know what parts of the storage stack you guys are working on, but I worked on enterprise storage systems we had 100% coverage in one form or another from the moment it left the RAM, via the intel interconnects, PCIe, the FC adapters which were either ECC or were using some form of data protection wrapping the entire transaction. So random bit flips in serdes/etc didn't cause problems because the higher level link protocols protected the data between the onboard ram and the endpoint, then there were the higher level protocol also providing their own data integrity. Looking at even SATA 1, if it is implemented _CORRECTLY_ you get the packet CRC protecting the link data from the point its formed to the point the endpoint verifies the result. So like ethernet sometimes it doesn't matter if some random piece of junk in the process doesn't do its own ECC validation, because its covered under a higher level of the stack. If your adapters are "desktop" grade then I might consider seeking another vendor if you care about data integrity, some vendors are definitely shipping crap, but there are vendors I can assure you will detect link/etc failures. And as a side, note, I've seen a lot of bad data, a huge percentage of it, were kernel/etc filesystem errors. We added a bunch of out of band extra metrics to track when/where writes were going and our own metadata layers/etc, and it uncovered a whole bunch of software errors.
nix23 5y ago
ZFS can easily protect you from that: >>it is designed with a focus on data integrity by protecting the user's data on disk against silent data corruption caused by data degradation, power surges (voltage spikes), bugs in disk firmware, phantom writes (the previous write did not make it to disk), misdirected reads/writes (the disk accesses the wrong block), DMA parity errors between the array and server memory or from the driver (since the checksum validates data inside the array), driver errors (data winds up in the wrong buffer inside the kernel), accidental overwrites (such as swapping to a live file system), etc. https://en.wikipedia.org/wiki/ZFS#Data_integrity https://en.wikipedia.org/wiki/ZFS#Data_integrity