2 ms·
> ...flash level checksums on a vertically integrated stack make checksumming again at the FS layer moot... This is simply not true. First, flash is not alway
by bayindirh 7y ago
> ...flash level checksums on a vertically integrated stack make checksumming again at the FS layer moot...
This is simply not true. First, flash is not always the place data ends at. Flash is prevalent in more points in the storage chain. Predominantly on RAID controllers' caches in last 3-5 years and hybrid drives used in laptops. So, flash-level checksumming can guarantee that the data is retrieved in the form it's written and allows neat tricks like wear-leveling and such. On the other hand data can get corrupted in other places.
e.g. Data can get corrupted in RAM, bit-rot in hard drives, RAID controllers can add a thick layer of complexity which you cannot manage or the filesystem can just make mistakes due to bugs (I've seen it all). So flash-level checksum can protect the data on the flash, but cannot guarantee that the data is correct in the first place. So if you checksummed the data and wrote it differently, FS or RAID level checksumming will catch it. This will prevent domino effects which the corrupted data can create.
Data management gets complicated fast when you need more than a few disk's aggregate capacity with high performance and reliability and, FS level checksumming is a powerful tool to ensure that.
> ...assuming you have proper control and reporting on both sides of the block device abstraction boundary.
Again this will only guarantee that the device reads the data it has written, but it won't guarantee that it had received the correct data in the first place.
> Apple's about the only one who can pull this off with their vertically integrated stack.
While Apple can do a lot of exotic stuff with the hardware, it won't be wise to create a custom flash controller and protocol to fuse FS and flash level checksums. It'll increase the software and integration cost, limit their options in hardware scene, slow down product development and plainly limit them in the long run.
I had an Intel 4 channel RAID card which corrupted the array when it wakes up from the wrong side of the bed and, it spent hours to verify and rebuild the array from the block level checksums it created on the disks during array creation. I've never lost data, but decided that it didn't worth the effort to keep the array alive and well. So, I eventually ditched it. However its activity LEDs on the board was kinda nice.
- monocasa 7y ago> While Apple can do a lot of exotic stuff with the hardware, it won't be wise to create a custom flash controller and protocol to fuse FS and flash level checksums. It'll increase the software and integration cost, limit their options in hardware scene, slow down product development and plainly limit them in the long run. I mean, tell that to them, because that's what they've done. The T2 chip's feature set is very married to APFS. AFAIK, Linux still doesn't have working drivers for it despite it sort of looking like an NVMe drive.