5 ms·

> If they're really interested with data integrity they should add checksums to APFS. Or you can spend half a billion dollars to solve the issue in hardware.

> If they're really interested with data integrity they should add checksums to APFS. Or you can spend half a billion dollars to solve the issue in hardware. As one of the creators of ZFS wrote when APFS was announced: > Explicitly not checksumming user data is a little more interesting. The APFS engineers I talked to cited strong ECC protection within Apple storage devices. Both NAND flash SSDs and magnetic media HDDs use redundant data to detect and correct errors. The Apple engineers contend that Apple devices basically don't return bogus data. https://arstechnica.com/gadgets/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/ https://arstechnica.com/gadgets/2016/06/a-zfs-developers-ana... APFS keeps redundant copies and checksums for metadata, but doesn't constantly checksum files looking for changes any more than NTFS does.

sitkack 1y ago
That is a weak excuse to rely on data integrity in the hardware. They most likely had that feature and removed it so they wouldn't be liable for a class action lawsuit when it turns out the NAND ages out due to bug in the retention algorithm. NTFS is what, 35 years old at this point? Odd comparison.
- GeekyBear 1y ago
  The point is that NTFS makes zero effort to maintain file integrity at any level. Handling file integrity at the hardware level is a big step up.
  - sitkack 1y ago
    NTFS is an ancient but well designed file system, a more apt comparison would be ZFS or BTRFS both of which checksum metadata and data.
protimewaster 1y ago
That solution doesn't help anyone who's using external storage, though, so it kinda feels like a half billion dollars spent on a limited solution.
- 1over137 1y ago
  Apple does not care about external storage at all, as in external disks. They offer iCloud for external storage. They don't sell external disks. They don't like cables. They make lots of money selling you a bigger internal disk.
  - realityking 1y ago
    Their store has a whole section dedicated to storage, most of it external. https://www.apple.com/shop/mac/accessories/storage https://www.apple.com/shop/mac/accessories/storage
- GeekyBear 1y ago
  There is nothing preventing you from running OpenZFS on external storage if you are worried that the hardware you purchased is less reliable.
  - protimewaster 1y ago
    That's my point, though, is that it seems weird to spend a half billion dollars just to solve the problem for an extremely common use case by saying "use OpenZFS". Why not come up with a solution that covers external storage too, instead of spending all that money and relying on external solutions? I just don't understand why they couldn't have optional checksums in APFS.
    - GeekyBear 1y ago
      It's far more weird that NTFS still makes zero effort to maintain file integrity on any level, on internal or external disks. ReFS exists, so Microsoft knew they needed to do something, but they have utterly failed to protect the vast majority of users.
      - protimewaster 1y ago
        To be fair, though, NTFS predates APFS by over 20 years. Don't get me wrong, there's no reason Microsoft can't transition to another filesystem (like offering ReFS outside of Server or whatever Windows variants support it currently), but I don't understand why a company would transition to a new filesystem in 2016 and not include a data checksums option. Hell, ReFS predates APFS, and I think it even has optional data checksums.
throw0101c 1y ago
> Or you can spend half a billion dollars to solve the issue in hardware. And hope that your hardware/firmware doesn't ever get bugs. Or you can do checksumming at the hardware layer and checksumming at the software/FS layer. Protection in depth. ZFS has caught issues from hardware, like when LBA 123 is requested but LBA 456 is delivered: the hardware-level checksum for LBA 456 was fine, and so it was passed up the stack, but it wasn't actually the data that was asked for. See Bryan Cantrill's talk "Zebras All the way Down": * https://www.youtube.com/watch?v=fE2KDzZaxvE https://www.youtube.com/watch?v=fE2KDzZaxvE And if checksums are not needed for a particular use-case, make them toggleable: even ZFS has a set checksums=off option. My problem is not having the option at all.
- GeekyBear 1y ago
  When the vast majority of the devices you sell run on battery power, it makes far more sense from a battery life perspective to handle issues in hardware as much as possible. For instance, try to find a processor aimed at mobile devices that doesn't handle video decoding in dedicated hardware instead of running it on a CPU core.
  - flkenosad 1y ago
    Good point.
  - throw0101c 1y ago
    > […] handle issues in hardware as much as possible. 1. There is hardware support for (e.g.) SHA in ARM: * https://developer.arm.com/documentation/ddi0514/g/introduction/about-the-cortex-a57-processor-cryptography-engine https://developer.arm.com/documentation/ddi0514/g/introducti... But given Apple designs their own CPUs they could add extensions for anything they need. Or use a simpler algorithm, like Fletcher (which ZFS uses): * https://en.wikipedia.org/wiki/Fletcher%27s_checksum https://en.wikipedia.org/wiki/Fletcher%27s_checksum 2. It does not have to be enabled by default for every device. The main problem is the lack of it even as an option. I wouldn't necessarily use ZFS checksums on a laptop, but ZFS has them for when I use it on a not-laptop.

sitkack 1y ago

That is a weak excuse to rely on data integrity in the hardware. They most likely had that feature and removed it so they wouldn't be liable for a class action lawsuit when it turns out the NAND ages out due to bug in the retention algorithm. NTFS is what, 35 years old at this point? Odd comparison.

GeekyBear 1y ago
The point is that NTFS makes zero effort to maintain file integrity at any level. Handling file integrity at the hardware level is a big step up.
- sitkack 1y ago
  NTFS is an ancient but well designed file system, a more apt comparison would be ZFS or BTRFS both of which checksum metadata and data.

protimewaster 1y ago

That solution doesn't help anyone who's using external storage, though, so it kinda feels like a half billion dollars spent on a limited solution.

1over137 1y ago
Apple does not care about external storage at all, as in external disks. They offer iCloud for external storage. They don't sell external disks. They don't like cables. They make lots of money selling you a bigger internal disk.
- realityking 1y ago
  Their store has a whole section dedicated to storage, most of it external. https://www.apple.com/shop/mac/accessories/storage https://www.apple.com/shop/mac/accessories/storage
GeekyBear 1y ago
There is nothing preventing you from running OpenZFS on external storage if you are worried that the hardware you purchased is less reliable.
- protimewaster 1y ago
  That's my point, though, is that it seems weird to spend a half billion dollars just to solve the problem for an extremely common use case by saying "use OpenZFS". Why not come up with a solution that covers external storage too, instead of spending all that money and relying on external solutions? I just don't understand why they couldn't have optional checksums in APFS.
  - GeekyBear 1y ago
    It's far more weird that NTFS still makes zero effort to maintain file integrity on any level, on internal or external disks. ReFS exists, so Microsoft knew they needed to do something, but they have utterly failed to protect the vast majority of users.
    - protimewaster 1y ago
      To be fair, though, NTFS predates APFS by over 20 years. Don't get me wrong, there's no reason Microsoft can't transition to another filesystem (like offering ReFS outside of Server or whatever Windows variants support it currently), but I don't understand why a company would transition to a new filesystem in 2016 and not include a data checksums option. Hell, ReFS predates APFS, and I think it even has optional data checksums.

throw0101c 1y ago

> Or you can spend half a billion dollars to solve the issue in hardware. And hope that your hardware/firmware doesn't ever get bugs. Or you can do checksumming at the hardware layer and checksumming at the software/FS layer. Protection in depth. ZFS has caught issues from hardware, like when LBA 123 is requested but LBA 456 is delivered: the hardware-level checksum for LBA 456 was fine, and so it was passed up the stack, but it wasn't actually the data that was asked for. See Bryan Cantrill's talk "Zebras All the way Down": * https://www.youtube.com/watch?v=fE2KDzZaxvE https://www.youtube.com/watch?v=fE2KDzZaxvE And if checksums are not needed for a particular use-case, make them toggleable: even ZFS has a set checksums=off option. My problem is not having the option at all.

GeekyBear 1y ago
When the vast majority of the devices you sell run on battery power, it makes far more sense from a battery life perspective to handle issues in hardware as much as possible. For instance, try to find a processor aimed at mobile devices that doesn't handle video decoding in dedicated hardware instead of running it on a CPU core.
- flkenosad 1y ago
  Good point.
- throw0101c 1y ago
  > […] handle issues in hardware as much as possible. 1. There is hardware support for (e.g.) SHA in ARM: * https://developer.arm.com/documentation/ddi0514/g/introduction/about-the-cortex-a57-processor-cryptography-engine https://developer.arm.com/documentation/ddi0514/g/introducti... But given Apple designs their own CPUs they could add extensions for anything they need. Or use a simpler algorithm, like Fletcher (which ZFS uses): * https://en.wikipedia.org/wiki/Fletcher%27s_checksum https://en.wikipedia.org/wiki/Fletcher%27s_checksum 2. It does not have to be enabled by default for every device. The main problem is the lack of it even as an option. I wouldn't necessarily use ZFS checksums on a laptop, but ZFS has them for when I use it on a not-laptop.