4 ms·
Yes, that's a possibility. Flash level checksum only guarantees that the written or read LBA is intact. However, I've never seen an enterprise storage system t
by bayindirh 7y ago
Yes, that's a possibility. Flash level checksum only guarantees that the written or read LBA is intact.
However, I've never seen an enterprise storage system to do a mistake like that. My archaic OCZ Vectors or Mac SSDs also didn't do something similar. The problem you mention is as sinister as bit-rot.
It's always nice that the FS is self aware about its integrity, however it may be overkill for consumer class devices and installations. Maybe making the flash controller and the storage itself more reliable should be better in the long run for these type of devices IMHO.
- monocasa 7y agoFlash level checksums also generally cover the flash translation layer (ie. the mapping of physical to logical sectors). It's probably not needed for a vertically integrated stack of APFS on a Apple SSD controller. It'd be nice to have fir external drives though.
- bayindirh 7y agoFlash level checksums are an integral part of wear-leveling and flash block allocation strategy. So they are also used for block migration/retirement controls. In other words, it's not trivial or desirable to remove flash level ECC or other checksums from physical flash management layer. I cannot find the article now, but the flash storage is way less reliable than it seems and it's correcting bit errors silently without reporting most of the time. This article [0] shows some of the things behind the curtain in flash storage configuration and lifetime. I'd edit or add the other article if I can find it. [0]: https://goughlui.com/2015/04/05/teardown-optimization-comsol-8gb-usb-flash-stick-au6989sn-gt-sdtnrcama-008g/ https://goughlui.com/2015/04/05/teardown-optimization-comsol...
- monocasa 7y agoI'm saying that flash level checksums on a vertically integrated stack make checksumming again at the FS layer moot, assuming you have proper control and reporting on both sides of the block device abstraction boundary. Apple's about the only one who can pull this off with their vertically integrated stack.
- zzzcpan 7y agoAnyone can pull this off, because filesystem level checksumming without an additional place to recover data from is useless.
- bayindirh 7y agoExchange the checksumming with hamming codes or with another ECC system with >=2 bit detection/correction capability and you're all set.
- zzzcpan 7y agoNot really set, but even assuming you can make it work this is already way beyond checksumming the lack of which people complain about for some reason.
- bayindirh 7y agoMost of the big storage vendors are making it work with Erasure or Hamming Codes. They compensate damaged or missing disks in the arrays transparently and send you a small e-mail about the details of the disk you need to change. RAID-5 and RAID-6 is the most primitive and probably most deployed version of this checksum based compensation and recovery methods. Erasure/Hamming is used for bigger arrays where RAID-5/6 cannot build the disks in a reasonable amount time while they are under I/O load.
- monocasa 7y agoPretty much every NAND flash device out there uses full ECC rather than just a checksum. The whole industry relies on it for yield reasons.
- bayindirh 7y ago> ...flash level checksums on a vertically integrated stack make checksumming again at the FS layer moot... This is simply not true. First, flash is not always the place data ends at. Flash is prevalent in more points in the storage chain. Predominantly on RAID controllers' caches in last 3-5 years and hybrid drives used in laptops. So, flash-level checksumming can guarantee that the data is retrieved in the form it's written and allows neat tricks like wear-leveling and such. On the other hand data can get corrupted in other places. e.g. Data can get corrupted in RAM, bit-rot in hard drives, RAID controllers can add a thick layer of complexity which you cannot manage or the filesystem can just make mistakes due to bugs (I've seen it all). So flash-level checksum can protect the data on the flash, but cannot guarantee that the data is correct in the first place. So if you checksummed the data and wrote it differently, FS or RAID level checksumming will catch it. This will prevent domino effects which the corrupted data can create. Data management gets complicated fast when you need more than a few disk's aggregate capacity with high performance and reliability and, FS level checksumming is a powerful tool to ensure that. > ...assuming you have proper control and reporting on both sides of the block device abstraction boundary. Again this will only guarantee that the device reads the data it has written, but it won't guarantee that it had received the correct data in the first place. > Apple's about the only one who can pull this off with their vertically integrated stack. While Apple can do a lot of exotic stuff with the hardware, it won't be wise to create a custom flash controller and protocol to fuse FS and flash level checksums. It'll increase the software and integration cost, limit their options in hardware scene, slow down product development and plainly limit them in the long run. I had an Intel 4 channel RAID card which corrupted the array when it wakes up from the wrong side of the bed and, it spent hours to verify and rebuild the array from the block level checksums it created on the disks during array creation. I've never lost data, but decided that it didn't worth the effort to keep the array alive and well. So, I eventually ditched it. However its activity LEDs on the board was kinda nice.
- throw0101a 7y ago> It's probably not needed for a vertically integrated stack of APFS on a Apple SSD controller. Apple code has no bugs that could send back bad data? Apple has access to hardware that does not experience bit rot? Apple's gear is not susceptible to firmware hacking? * https://www.wired.com/2015/02/nsa-firmware-hacking/ https://www.wired.com/2015/02/nsa-firmware-hacking/ The various old school SAN vendors have all sorts of control for storage controllers and disk firmware, and I've seen ZFS checksum errors appear on scrubs with them.
- monocasa 7y ago> Apple code has no bugs that could send back bad data? Apple has access to hardware that does not experience bit rot? Apple's gear is not susceptible to firmware hacking? 'It' being checksums over data that they already know is checksummed by the F2L layer and they know the semantics of. If you're hacking the firmware, you know that APFS is running on top of it, so you would just fix-up it's checksums too. APFS having checksums doesn't get you anything there either. > The various old school SAN vendors have all sorts of control for storage controllers and disk firmware, and I've seen ZFS checksum errors appear on scrubs with them. You have next to zero introspection into how those drives actually work, or where the semantics of their F2L breaks down the easiest (they hide all of that out of patent concerns). For instance do those drives work around a write ahead log primarily, or is it a more traditional block renaming tree? How many generations are there? All the BS counters in the world are next to meaningless without some base implementation knowledge that's missing.
- zzzcpan 7y agoSay you have a bug that corrupts data which somehow happens after writes so filesystem level checksums can detect it and you run that filesystem on desktop/laptop/mobile without redundancy. Does it actually help you with anything since you can't correct that error on the same filesystem level? Of course not. Does it at least prevent inconsistency? No, it doesn't unless failed checksums stop filesystem from doing new reads and writes. So the only sensible place for checksumming is on the level where such errors can be automatically corrected. Without redundancy that's just your disk controller or applications designed with data recovery in mind that also can do checksumming, not filesystems though.
- throw0101a 7y ago> However, I've never seen an enterprise storage system to do a mistake like that. Unless you're using a checksumming file system (a la ZFS), how would you know? And even if you were using ZFS, it would probably just do a re-read from another part of the mirror/RAID-Z and fix it behind the scenes. That's the issue with people saying "things are fine": how do they actually know since we're not verifying things? Up until fairly recently, we haven't really been looking too closely, simply trusting the lower levels of the stack: * http://www.cs.toronto.edu/~bianca/papers/fast08.pdf http://www.cs.toronto.edu/~bianca/papers/fast08.pdf