6 ms·

The article speculates on why Apple integrates the SSD controller onto the SOC for their A and M series chips, but misses one big reason, data integrity. About

The article speculates on why Apple integrates the SSD controller onto the SOC for their A and M series chips, but misses one big reason, data integrity. About a decade and a half ago, Apple paid half a billion dollars to acquire the patents of a company making enterprise SSD controllers. > Anobit appears to be applying a lot of signal processing techniques in addition to ECC to address the issue of NAND reliability and data retention. In its patents there are mentions of periodically refreshing cells whose voltages may have drifted, exploiting some of the behaviors of adjacent cells and generally trying to deal with the things that happen to NAND once it's been worn considerably. Through all of these efforts, Anobit is promising significant improvements in NAND longevity and reliability. https://www.anandtech.com/show/5258/apple-acquires-anobit-bringing-nand-endurance-technology-inhouse https://www.anandtech.com/show/5258/apple-acquires-anobit-br...

jeffbee 1y ago
> Through all of these efforts, Anobit is promising significant improvements in NAND longevity and reliability. Every flash controller does this. Modern NAND is just math on a stick. Lots and lots of math.
- dontlaugh 1y ago
  Presumably Apple want to be able to guarantee the quality of such logic. Still sucks that you can’t use standard parts.
  - jeffbee 1y ago
    Sure, and I agree with that goal. In fact I would like NVMe controllers to simply not exist. The operating system should manage raw flash, using host algorithms that I can study in the the source code.
    - mixmastamyk 1y ago
      Interesting… I see no reason to be downvoted.
    - nine_k 1y ago
      How do you think it would be electrically connected to the CPU? Same thing with DDR5: the electrical layer is a beast, it's a reason enough to require its own controller.
      - daneel_w 1y ago
        > How do you think it would be electrically connected to the CPU? On the CPU's PCIe bus. NVMe drives are PCIe devices, designed specifically to facilitate such interfacing. Edit: Pardon, misread the actual statement you responded to. Of course one shouldn't hook NAND directly to the CPU. I'll leave my response for whatever value the info has.
    - gruturo 1y ago
      I'm with you, but.... no. At the level where the controller is operating, things are no longer digital. Capacity (as in farads, not bytes), voltage, crosstalk, debouncing, traces behaving like antennas, terminations, what have you. Analog values, temperature dependencies, RF interference. Stuff best dealt with custom logic placed as close as possible to it.
      - jeffbee 1y ago
        The physical interface controller can exist to that extent, of course. But I think the command interface it should present to the host system should be a physical one, not a logical translation. The host should be totally aware of the layout of the flash devices, and should command the things that the devices are actually capable of doing: erase this, write that, read this. We already see the demand for this in the latest NVMe protocol spec that allows the host to give placement hints. But this is a half-measure that suggests what systems really want, which is not to vaguely influence the device but instead to tell it exactly what to do.
  - daneel_w 1y ago
    Contrary to popular belief, you can run many different off-the-shelf brand NVMe drives on all of the NVMe-fitted Intel Macs. All you need is a passive adapter. My 2017 MacBook Air has a 250GB WD Blue SN570 in it. https://forums.macrumors.com/threads/upgrading-2013-2015-macbook-pro-ssd-to-m-2-nvme.2034976/ https://forums.macrumors.com/threads/upgrading-2013-2015-mac...
    - Rohansi 1y ago
      And all of the Apple Silicon Macs have them soldered on or use a proprietary module. They make way too much from storage upgrades to include an M.2 slot.
      - daneel_w 1y ago
        The on-board storage of their MacBooks is as much, if not more, driven by their design vision. They didn't make these things record thin to get an excuse to use surface-mounted RAM and storage. I don't agree with it, but I understand it.
    - riobard 1y ago
      Not all Intel Macs. Only those without T2 chip, as T2 acts as the storage controller.
wpm 1y ago
Note that this isn't too long after Apple abandoned efforts to bring ZFS into Mac OS X as a potential default filesystem. Patents were probably a good reason, given the Oracle buyout of Sun, but also a bit of "skating to where the puck will be" and realizing that the spinning rust ZFS was built for probably wasn't going to be in their computers for much longer.
- GeekyBear 1y ago
  When Apple announced the creation of APFS they mentioned that their intent was to handle data integrity at the hardware level.
  - throw0101c 1y ago
    See: > Apple File System uses checksums to ensure data integrity for metadata but not for the actual user data, relying instead on error-correcting code (ECC) mechanisms in the storage hardware.[18] * https://en.wikipedia.org/wiki/Apple_File_System#Data_integrity https://en.wikipedia.org/wiki/Apple_File_System#Data_integri...
- kjkjadksj 1y ago
  More evidence they thought hdds were on their way out was the unibody macbook keynote. They made a big deal about how the user can access their hdd from the latch on the bottom without any tools as they said ssd was on the horizon.
- throw0101c 1y ago
  > Patents were probably a good reason, given the Oracle buyout of Sun There is no reason to speculate as the reason is know (as stated by Jeff Bonwick, one of the co-inventors of ZFS): >> Apple can currently just take the ZFS CDDL code and incorporate it (like they did with DTrace), but it may be that they wanted a "private license" from Sun (with appropriate technical support and indemnification), and the two entities couldn't come to mutually agreeable terms. > I cannot disclose details, but that is the essence of it. * https://archive.is/http://mail.opensolaris.org/pipermail/zfs-discuss/2009-October/033125.html https://archive.is/http://mail.opensolaris.org/pipermail/zfs... * https://web.archive.org/web/*/http://mail.opensolaris.org/pipermail/zfs-discuss/2009-October/033125.html https://web.archive.org/web/*/http://mail.opensolaris.org/pi...
throw0101c 1y ago
> The article speculates on why Apple integrates the SSD controller onto the SOC for their A and M series chips, but misses one big reason, data integrity. If they're really interested with data integrity they should add checksums to APFS. If you don't have RAID you can't rebuild corrupted data, but at least you know there's a problem and perhaps restore from Time Machine. For metadata, you may have multiple copies, so can use a known-good one (this is how ZFS works: some things have multiple copies 'inherently' because they're so important). Edit: > Apple File System uses checksums to ensure data integrity for metadata but not for the actual user data, relying instead on error-correcting code (ECC) mechanisms in the storage hardware.[18] * https://en.wikipedia.org/wiki/Apple_File_System#Data_integrity https://en.wikipedia.org/wiki/Apple_File_System#Data_integri...
- sneak 1y ago
  You can do this yourself in userspace if you really want it: https://git.eeqj.de/sneak/attrsum https://git.eeqj.de/sneak/attrsum I use zfs where I can (it has content checksums) but it sucks bad on macOS, so I wrote attrsum. It keeps the file content checksum in an xattr (which APFS (and ext3/4) supports). I use it to protect my photo library on a huge external SSD formatted with APFS (encrypted, natch) because I need to mount it on a mac laptop for Lightroom.
  - arm 1y ago
    A similar alternative is Howard Oakley’s Dintch/Fintch/cintch: https://eclecticlight.co/dintch/ https://eclecticlight.co/dintch/
- GeekyBear 1y ago
  > If they're really interested with data integrity they should add checksums to APFS. Or you can spend half a billion dollars to solve the issue in hardware. As one of the creators of ZFS wrote when APFS was announced: > Explicitly not checksumming user data is a little more interesting. The APFS engineers I talked to cited strong ECC protection within Apple storage devices. Both NAND flash SSDs and magnetic media HDDs use redundant data to detect and correct errors. The Apple engineers contend that Apple devices basically don't return bogus data. https://arstechnica.com/gadgets/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/ https://arstechnica.com/gadgets/2016/06/a-zfs-developers-ana... APFS keeps redundant copies and checksums for metadata, but doesn't constantly checksum files looking for changes any more than NTFS does.
vlovich123 1y ago
Not just durability. Performance too. Apple has a much better SSD controller that is vertically integrated into the stack.
pkaye 1y ago
Do Apple SSDs have a much longer longevity and reliability? I've not looked at the specific patents nor am I an expert on signal processing but I've worked on SSD controllers and NAND manufacturers in the past and they had their own similar ideas as this.
- fckgw 1y ago
  From my experience working on Mac laptops, yeah. SSD failures are incredibly rare but on the flip side when they do go out repairs are very costly. I know if my previous job at a large hard drive manufacturer we had special Apple drives that ran different parts and firmware than the regular PC drives. Their specs and tolerances where much different than the PC market at a whole.
rasz 1y ago
Main reason was capturing 100% of storage upsell/upgrade money. They did same thing with RAM.

jeffbee 1y ago

> Through all of these efforts, Anobit is promising significant improvements in NAND longevity and reliability. Every flash controller does this. Modern NAND is just math on a stick. Lots and lots of math.

dontlaugh 1y ago
Presumably Apple want to be able to guarantee the quality of such logic. Still sucks that you can’t use standard parts.
- jeffbee 1y ago
  Sure, and I agree with that goal. In fact I would like NVMe controllers to simply not exist. The operating system should manage raw flash, using host algorithms that I can study in the the source code.
  - mixmastamyk 1y ago
    Interesting… I see no reason to be downvoted.
  - nine_k 1y ago
    How do you think it would be electrically connected to the CPU? Same thing with DDR5: the electrical layer is a beast, it's a reason enough to require its own controller.
    - daneel_w 1y ago
      > How do you think it would be electrically connected to the CPU? On the CPU's PCIe bus. NVMe drives are PCIe devices, designed specifically to facilitate such interfacing. Edit: Pardon, misread the actual statement you responded to. Of course one shouldn't hook NAND directly to the CPU. I'll leave my response for whatever value the info has.
  - gruturo 1y ago
    I'm with you, but.... no. At the level where the controller is operating, things are no longer digital. Capacity (as in farads, not bytes), voltage, crosstalk, debouncing, traces behaving like antennas, terminations, what have you. Analog values, temperature dependencies, RF interference. Stuff best dealt with custom logic placed as close as possible to it.
    - jeffbee 1y ago
      The physical interface controller can exist to that extent, of course. But I think the command interface it should present to the host system should be a physical one, not a logical translation. The host should be totally aware of the layout of the flash devices, and should command the things that the devices are actually capable of doing: erase this, write that, read this. We already see the demand for this in the latest NVMe protocol spec that allows the host to give placement hints. But this is a half-measure that suggests what systems really want, which is not to vaguely influence the device but instead to tell it exactly what to do.
- daneel_w 1y ago
  Contrary to popular belief, you can run many different off-the-shelf brand NVMe drives on all of the NVMe-fitted Intel Macs. All you need is a passive adapter. My 2017 MacBook Air has a 250GB WD Blue SN570 in it. https://forums.macrumors.com/threads/upgrading-2013-2015-macbook-pro-ssd-to-m-2-nvme.2034976/ https://forums.macrumors.com/threads/upgrading-2013-2015-mac...
  - Rohansi 1y ago
    And all of the Apple Silicon Macs have them soldered on or use a proprietary module. They make way too much from storage upgrades to include an M.2 slot.
    - daneel_w 1y ago
      The on-board storage of their MacBooks is as much, if not more, driven by their design vision. They didn't make these things record thin to get an excuse to use surface-mounted RAM and storage. I don't agree with it, but I understand it.
  - riobard 1y ago
    Not all Intel Macs. Only those without T2 chip, as T2 acts as the storage controller.

wpm 1y ago

Note that this isn't too long after Apple abandoned efforts to bring ZFS into Mac OS X as a potential default filesystem. Patents were probably a good reason, given the Oracle buyout of Sun, but also a bit of "skating to where the puck will be" and realizing that the spinning rust ZFS was built for probably wasn't going to be in their computers for much longer.

GeekyBear 1y ago
When Apple announced the creation of APFS they mentioned that their intent was to handle data integrity at the hardware level.
- throw0101c 1y ago
  See: > Apple File System uses checksums to ensure data integrity for metadata but not for the actual user data, relying instead on error-correcting code (ECC) mechanisms in the storage hardware.[18] * https://en.wikipedia.org/wiki/Apple_File_System#Data_integrity https://en.wikipedia.org/wiki/Apple_File_System#Data_integri...
kjkjadksj 1y ago
More evidence they thought hdds were on their way out was the unibody macbook keynote. They made a big deal about how the user can access their hdd from the latch on the bottom without any tools as they said ssd was on the horizon.
throw0101c 1y ago
> Patents were probably a good reason, given the Oracle buyout of Sun There is no reason to speculate as the reason is know (as stated by Jeff Bonwick, one of the co-inventors of ZFS): >> Apple can currently just take the ZFS CDDL code and incorporate it (like they did with DTrace), but it may be that they wanted a "private license" from Sun (with appropriate technical support and indemnification), and the two entities couldn't come to mutually agreeable terms. > I cannot disclose details, but that is the essence of it. * https://archive.is/http://mail.opensolaris.org/pipermail/zfs-discuss/2009-October/033125.html https://archive.is/http://mail.opensolaris.org/pipermail/zfs... * https://web.archive.org/web/*/http://mail.opensolaris.org/pipermail/zfs-discuss/2009-October/033125.html https://web.archive.org/web/*/http://mail.opensolaris.org/pi...

throw0101c 1y ago

> The article speculates on why Apple integrates the SSD controller onto the SOC for their A and M series chips, but misses one big reason, data integrity. If they're really interested with data integrity they should add checksums to APFS. If you don't have RAID you can't rebuild corrupted data, but at least you know there's a problem and perhaps restore from Time Machine. For metadata, you may have multiple copies, so can use a known-good one (this is how ZFS works: some things have multiple copies 'inherently' because they're so important). Edit: > Apple File System uses checksums to ensure data integrity for metadata but not for the actual user data, relying instead on error-correcting code (ECC) mechanisms in the storage hardware.[18] * https://en.wikipedia.org/wiki/Apple_File_System#Data_integrity https://en.wikipedia.org/wiki/Apple_File_System#Data_integri...

sneak 1y ago
You can do this yourself in userspace if you really want it: https://git.eeqj.de/sneak/attrsum https://git.eeqj.de/sneak/attrsum I use zfs where I can (it has content checksums) but it sucks bad on macOS, so I wrote attrsum. It keeps the file content checksum in an xattr (which APFS (and ext3/4) supports). I use it to protect my photo library on a huge external SSD formatted with APFS (encrypted, natch) because I need to mount it on a mac laptop for Lightroom.
- arm 1y ago
  A similar alternative is Howard Oakley’s Dintch/Fintch/cintch: https://eclecticlight.co/dintch/ https://eclecticlight.co/dintch/
GeekyBear 1y ago
> If they're really interested with data integrity they should add checksums to APFS. Or you can spend half a billion dollars to solve the issue in hardware. As one of the creators of ZFS wrote when APFS was announced: > Explicitly not checksumming user data is a little more interesting. The APFS engineers I talked to cited strong ECC protection within Apple storage devices. Both NAND flash SSDs and magnetic media HDDs use redundant data to detect and correct errors. The Apple engineers contend that Apple devices basically don't return bogus data. https://arstechnica.com/gadgets/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/ https://arstechnica.com/gadgets/2016/06/a-zfs-developers-ana... APFS keeps redundant copies and checksums for metadata, but doesn't constantly checksum files looking for changes any more than NTFS does.

vlovich123 1y ago

Not just durability. Performance too. Apple has a much better SSD controller that is vertically integrated into the stack.

pkaye 1y ago

Do Apple SSDs have a much longer longevity and reliability? I've not looked at the specific patents nor am I an expert on signal processing but I've worked on SSD controllers and NAND manufacturers in the past and they had their own similar ideas as this.

fckgw 1y ago
From my experience working on Mac laptops, yeah. SSD failures are incredibly rare but on the flip side when they do go out repairs are very costly. I know if my previous job at a large hard drive manufacturer we had special Apple drives that ran different parts and firmware than the regular PC drives. Their specs and tolerances where much different than the PC market at a whole.

rasz 1y ago

Main reason was capturing 100% of storage upsell/upgrade money. They did same thing with RAM.