6 ms·

For internal storage Apple has control over the hardware. They can just choose hardware with better internal error correction/detection, so block level checksum

For internal storage Apple has control over the hardware. They can just choose hardware with better internal error correction/detection, so block level checksumming on the software side doesn't make much sense in this case. For external storage it would be nice, though.

rhinoceraptor 7y ago
You can't know how many uncorrectable errors you have if you're not check summing. You can't see because there are no lights. And IMO it's just negligent to believe your flash memory is 100% perfect.
- dwaite 7y ago
  I thought the storage itself had the integrity check.
  - rhinoceraptor 7y ago
    I'm sure it has some special Apple magic, but I really think the hardware-only approach isn't the path forward. Hardware faults happen, and papering over that fact with un-debuggable, un-observable firmware is not a good choice.
- monocasa 7y ago
  Their flash translation layer has block level checksums. Most of the time that's not enough, because the SSD vendor doesn't give you a lot of introspection into their failures or how their F2L works. But this is something you gain from vertical integration like Apple's.
- bayindirh 7y ago
  However the flash memory knows the checksum on the hardware level, and running its own internal checks for data integrity. Flash controllers reserve a certain amount of blocks for CRC and check summing, so flash is much more intelligent and knowledgeable than it seems.
  - throw0101a 7y ago
    > * However the flash memory knows the checksum on the hardware level, and running its own internal checks for data integrity.* Assuming that the flash controller has no bugs and doesn't fetch the wrong sector/LBA. The OS asks for LBA X, but instead Y is fetched, and the sum(Y) passes so the data is sent back as "OK". If you don't have checksums at a layer above the hardware, how do you know if the right sector was sent to you? (I've seen reports of ZFS catching errors like this.)
    - bayindirh 7y ago
      Yes, that's a possibility. Flash level checksum only guarantees that the written or read LBA is intact. However, I've never seen an enterprise storage system to do a mistake like that. My archaic OCZ Vectors or Mac SSDs also didn't do something similar. The problem you mention is as sinister as bit-rot. It's always nice that the FS is self aware about its integrity, however it may be overkill for consumer class devices and installations. Maybe making the flash controller and the storage itself more reliable should be better in the long run for these type of devices IMHO.
      - monocasa 7y ago
        Flash level checksums also generally cover the flash translation layer (ie. the mapping of physical to logical sectors). It's probably not needed for a vertically integrated stack of APFS on a Apple SSD controller. It'd be nice to have fir external drives though.
        bayindirh 7y ago
        Flash level checksums are an integral part of wear-leveling and flash block allocation strategy. So they are also used for block migration/retirement controls. In other words, it's not trivial or desirable to remove flash level ECC or other checksums from physical flash management layer. I cannot find the article now, but the flash storage is way less reliable than it seems and it's correcting bit errors silently without reporting most of the time. This article [0] shows some of the things behind the curtain in flash storage configuration and lifetime. I'd edit or add the other article if I can find it. [0]: https://goughlui.com/2015/04/05/teardown-optimization-comsol-8gb-usb-flash-stick-au6989sn-gt-sdtnrcama-008g/ https://goughlui.com/2015/04/05/teardown-optimization-comsol...

rhinoceraptor 7y ago

You can't know how many uncorrectable errors you have if you're not check summing. You can't see because there are no lights. And IMO it's just negligent to believe your flash memory is 100% perfect.

dwaite 7y ago
I thought the storage itself had the integrity check.
- rhinoceraptor 7y ago
  I'm sure it has some special Apple magic, but I really think the hardware-only approach isn't the path forward. Hardware faults happen, and papering over that fact with un-debuggable, un-observable firmware is not a good choice.
monocasa 7y ago
Their flash translation layer has block level checksums. Most of the time that's not enough, because the SSD vendor doesn't give you a lot of introspection into their failures or how their F2L works. But this is something you gain from vertical integration like Apple's.
bayindirh 7y ago
However the flash memory knows the checksum on the hardware level, and running its own internal checks for data integrity. Flash controllers reserve a certain amount of blocks for CRC and check summing, so flash is much more intelligent and knowledgeable than it seems.
- throw0101a 7y ago
  > * However the flash memory knows the checksum on the hardware level, and running its own internal checks for data integrity.* Assuming that the flash controller has no bugs and doesn't fetch the wrong sector/LBA. The OS asks for LBA X, but instead Y is fetched, and the sum(Y) passes so the data is sent back as "OK". If you don't have checksums at a layer above the hardware, how do you know if the right sector was sent to you? (I've seen reports of ZFS catching errors like this.)
  - bayindirh 7y ago
    Yes, that's a possibility. Flash level checksum only guarantees that the written or read LBA is intact. However, I've never seen an enterprise storage system to do a mistake like that. My archaic OCZ Vectors or Mac SSDs also didn't do something similar. The problem you mention is as sinister as bit-rot. It's always nice that the FS is self aware about its integrity, however it may be overkill for consumer class devices and installations. Maybe making the flash controller and the storage itself more reliable should be better in the long run for these type of devices IMHO.
    - monocasa 7y ago
      Flash level checksums also generally cover the flash translation layer (ie. the mapping of physical to logical sectors). It's probably not needed for a vertically integrated stack of APFS on a Apple SSD controller. It'd be nice to have fir external drives though.
      - bayindirh 7y ago
        Flash level checksums are an integral part of wear-leveling and flash block allocation strategy. So they are also used for block migration/retirement controls. In other words, it's not trivial or desirable to remove flash level ECC or other checksums from physical flash management layer. I cannot find the article now, but the flash storage is way less reliable than it seems and it's correcting bit errors silently without reporting most of the time. This article [0] shows some of the things behind the curtain in flash storage configuration and lifetime. I'd edit or add the other article if I can find it. [0]: https://goughlui.com/2015/04/05/teardown-optimization-comsol-8gb-usb-flash-stick-au6989sn-gt-sdtnrcama-008g/ https://goughlui.com/2015/04/05/teardown-optimization-comsol...