11 ms·

So containers with a new bi-directional hardlink "Firmlink" (intended to be invisible to applications, link between encrypted containers), with read only volume

So containers with a new bi-directional hardlink "Firmlink" (intended to be invisible to applications, link between encrypted containers), with read only volumes protecting system software. ZFS send/receive style volume serialization for replication backup, etc. File system snapshot support included. iOS or should I say iPadOS supports external USB media and SMB. What seems to be missing is block level checksumming and integrity checks, scrubbing, block level duplication to protect important files, etc. No whiff of deduplication either, but that might be too memory intensive for Apple's purposes.

eclipxe 7y ago
iOS on iPhone also supports external USB media and SMB.
- pwinnski 7y ago
  I was surprised to see this. From the original keynote, it seemed like an iPadOS-only option, but this video made it clear that it applies to both.
  - dwaite 7y ago
    There's still a lot of effort to keep subsystems and application code-bases the same - iPad OS I believe is mostly a sign that there may continue to be UX deviation (widgets on home screen, drag and drop between applications, split view multitasking, etc)
    - scarface74 7y ago
      Each supported device has always had its own image, so I assume it’s not exactly the same codebase. The only reason I know this is because the last time I had to upgrade iOS using iTunes - going from iOS 4 to 5, I had three separate devices that required separate downloads for my iPhone 4, iPod Touch, and iPad.
- Angostura 7y ago
  And Samba shares, which from my perspective is even more interesting.
- scarface74 7y ago
  Minor correction. Will support USB media in iOS 13.
jandrese 7y ago
My understanding is that deduplication has generally been not worth the effort for a general use filesystem. The cost in memory, cpu, and sheer complexity doesn't pay off in enough storage savings. Disk space is relatively cheap. Filesystem crashes/corruption are not.
- vardump 7y ago
  Yeah, block level deduplication is not useful for files on a typical consumer laptop. I can imagine it's even less useful for files on iOS devices. However it is often great for virtual machines and other software development related things, as long as all the underlying block sizes and alignments match. You really do want strong integrity guarantees though. One corrupted block can cause a lot of damage.
  - Scaevolus 7y ago
    The filesystem encryption used on iOS probably makes deduplication even less effective.
    - vardump 7y ago
      Depends on the layering order. You can do both dedup and encryption in the same time, if the encryption occurs on a lower layer.
      - eridius 7y ago
        Encryption in iOS is per-file. I'm not sure the OS really wants to go ahead and individual decrypt large numbers of files just to check if there are any duplicated blocks.
- FPGAhacker 7y ago
  Disk space isn’t cheap on a Mac laptop.
  - 7y ago
sneak 7y ago
It is a real bummer that there are still no content checksums available. One can work around this by doing sums and scrubs in userspace by storing mtime+hash in an xattr and periodically verifying and updating them. There is a utility to do this but I forgot the name.
- Hackbraten 7y ago
  I’m thinking of moving back from ZFS to APFS if I had such a workaround. Block- or file-level integrity is now the only thing that keeps me back.
  - privateSFacct 7y ago
    What are the use cases folks have with super high reliability needs? I'm generally more worried about an SSD falling over entirely - so focus is on duplication (raid with hot spare -> offsite S3 backup).
    - wang_li 7y ago
      From a user perspective it's effectively a zero cost feature. Apple should add it in.
      - privateSFacct 7y ago
        OK - so they do a block level checksum (no error correction but it would let you know that something went wrong). Now something goes wrong with checksum. Is this a "zero cost" feature to have every app updated to handle this checksum error. What is gained? If you need reliability do something like SSD or a raid option and a backup.
        wang_li 7y ago
        The apps don't have to deal with it. It's a read error. Presumably something they already deal with. There's no operational difference for the end user between a block that can't be read and a block than can be read but with corrupted data. And yes, storage would ideally be paired with another physical device to provide redundancy and a mechanism to correct the error on the erroring device.
tobias3 7y ago
For internal storage Apple has control over the hardware. They can just choose hardware with better internal error correction/detection, so block level checksumming on the software side doesn't make much sense in this case. For external storage it would be nice, though.
- rhinoceraptor 7y ago
  You can't know how many uncorrectable errors you have if you're not check summing. You can't see because there are no lights. And IMO it's just negligent to believe your flash memory is 100% perfect.
  - dwaite 7y ago
    I thought the storage itself had the integrity check.
    - rhinoceraptor 7y ago
      I'm sure it has some special Apple magic, but I really think the hardware-only approach isn't the path forward. Hardware faults happen, and papering over that fact with un-debuggable, un-observable firmware is not a good choice.
  - monocasa 7y ago
    Their flash translation layer has block level checksums. Most of the time that's not enough, because the SSD vendor doesn't give you a lot of introspection into their failures or how their F2L works. But this is something you gain from vertical integration like Apple's.
  - bayindirh 7y ago
    However the flash memory knows the checksum on the hardware level, and running its own internal checks for data integrity. Flash controllers reserve a certain amount of blocks for CRC and check summing, so flash is much more intelligent and knowledgeable than it seems.

eclipxe 7y ago

iOS on iPhone also supports external USB media and SMB.

pwinnski 7y ago
I was surprised to see this. From the original keynote, it seemed like an iPadOS-only option, but this video made it clear that it applies to both.
- dwaite 7y ago
  There's still a lot of effort to keep subsystems and application code-bases the same - iPad OS I believe is mostly a sign that there may continue to be UX deviation (widgets on home screen, drag and drop between applications, split view multitasking, etc)
  - scarface74 7y ago
    Each supported device has always had its own image, so I assume it’s not exactly the same codebase. The only reason I know this is because the last time I had to upgrade iOS using iTunes - going from iOS 4 to 5, I had three separate devices that required separate downloads for my iPhone 4, iPod Touch, and iPad.
Angostura 7y ago
And Samba shares, which from my perspective is even more interesting.
scarface74 7y ago
Minor correction. Will support USB media in iOS 13.

jandrese 7y ago

My understanding is that deduplication has generally been not worth the effort for a general use filesystem. The cost in memory, cpu, and sheer complexity doesn't pay off in enough storage savings. Disk space is relatively cheap. Filesystem crashes/corruption are not.

vardump 7y ago
Yeah, block level deduplication is not useful for files on a typical consumer laptop. I can imagine it's even less useful for files on iOS devices. However it is often great for virtual machines and other software development related things, as long as all the underlying block sizes and alignments match. You really do want strong integrity guarantees though. One corrupted block can cause a lot of damage.
- Scaevolus 7y ago
  The filesystem encryption used on iOS probably makes deduplication even less effective.
  - vardump 7y ago
    Depends on the layering order. You can do both dedup and encryption in the same time, if the encryption occurs on a lower layer.
    - eridius 7y ago
      Encryption in iOS is per-file. I'm not sure the OS really wants to go ahead and individual decrypt large numbers of files just to check if there are any duplicated blocks.
FPGAhacker 7y ago
Disk space isn’t cheap on a Mac laptop.
- 7y ago

sneak 7y ago

It is a real bummer that there are still no content checksums available. One can work around this by doing sums and scrubs in userspace by storing mtime+hash in an xattr and periodically verifying and updating them. There is a utility to do this but I forgot the name.

Hackbraten 7y ago
I’m thinking of moving back from ZFS to APFS if I had such a workaround. Block- or file-level integrity is now the only thing that keeps me back.
- privateSFacct 7y ago
  What are the use cases folks have with super high reliability needs? I'm generally more worried about an SSD falling over entirely - so focus is on duplication (raid with hot spare -> offsite S3 backup).
  - wang_li 7y ago
    From a user perspective it's effectively a zero cost feature. Apple should add it in.
    - privateSFacct 7y ago
      OK - so they do a block level checksum (no error correction but it would let you know that something went wrong). Now something goes wrong with checksum. Is this a "zero cost" feature to have every app updated to handle this checksum error. What is gained? If you need reliability do something like SSD or a raid option and a backup.
      - wang_li 7y ago
        The apps don't have to deal with it. It's a read error. Presumably something they already deal with. There's no operational difference for the end user between a block that can't be read and a block than can be read but with corrupted data. And yes, storage would ideally be paired with another physical device to provide redundancy and a mechanism to correct the error on the erroring device.

tobias3 7y ago

For internal storage Apple has control over the hardware. They can just choose hardware with better internal error correction/detection, so block level checksumming on the software side doesn't make much sense in this case. For external storage it would be nice, though.

rhinoceraptor 7y ago
You can't know how many uncorrectable errors you have if you're not check summing. You can't see because there are no lights. And IMO it's just negligent to believe your flash memory is 100% perfect.
- dwaite 7y ago
  I thought the storage itself had the integrity check.
  - rhinoceraptor 7y ago
    I'm sure it has some special Apple magic, but I really think the hardware-only approach isn't the path forward. Hardware faults happen, and papering over that fact with un-debuggable, un-observable firmware is not a good choice.
- monocasa 7y ago
  Their flash translation layer has block level checksums. Most of the time that's not enough, because the SSD vendor doesn't give you a lot of introspection into their failures or how their F2L works. But this is something you gain from vertical integration like Apple's.
- bayindirh 7y ago
  However the flash memory knows the checksum on the hardware level, and running its own internal checks for data integrity. Flash controllers reserve a certain amount of blocks for CRC and check summing, so flash is much more intelligent and knowledgeable than it seems.