6 ms·

The author worked very hard to determine the root cause of the problem but he was stymied because the APFS source code is not available. Why not? Apple has open

The author worked very hard to determine the root cause of the problem but he was stymied because the APFS source code is not available. Why not? Apple has open-sourced the Swift compiler with great success, but there seems to be no movement within Apple to open up other system software components.

jarym 8y ago
Pretty sure the code would expose just how much of a rush-job apple did with APFS. It may be released one day..
wmf 8y ago
Apple releases most of the macOS kernel but I guess it doesn't include APFS: https://opensource.apple.com/source/xnu/ https://opensource.apple.com/source/xnu/
- NegativeLatency 8y ago
  That's missing a good chunk of the interesting stuff related to how users interact with the os though. (app and framework source code)
  - wmf 8y ago
    Yeah, apps and frameworks aren't part of the kernel. But APFS is.
    - Phrodo_00 8y ago
      Kind of talking out of my ass, but Darwin can act as a mirokernel, so APFS could be implemented as a user mode service (no idea if it's the case, though).
      - yjftsjthsd-h 8y ago
        Darwin's XNU is not a microkernel, in spite of there being a microkernel version of XNU. While Darwin does support FUSE, performance would almost certainly be inadequate, given the constant transitions between user and kernel space.
        Phrodo_00 8y ago
        I thought it was a mixture of OSFMK and BSD, which is why I said it can act like a microkernel. Don't know more than what wikipedia says, though.
        rayiner 8y ago
        Yes, it is, but the Mach layer and BSD layer were fused into a monolithic kernel before NeXT Step turned into OS X. In OS X, HFS (and presumably APFS) are in the BSD layer.
    - comex 8y ago
      It's loaded into the kernel, but it's a kext (kernel extension) rather than being part of the xnu source tree itself. Only a subset of kexts are open source. (By comparison, HFS used to be in the xnu tree, but it was moved into a kext as well a few years back; that one is open source.)
mikhailt 8y ago
They do plan to document it fully later: > Is APFS open source? > An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format specification. Source: https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html https://developer.apple.com/library/archive/documentation/Fi... My theory is that Apple rushed APFS and its implementation isn't in a great shape for the public review. APFS doesn't feel any faster on my SSDs on macOS. I suspect they'll add it in a few years, after a few iterations.
- twotwotwo 8y ago
  They released the format spec here: https://developer.apple.com/support/apple-file-system/Apple-File-System-Reference.pdf https://developer.apple.com/support/apple-file-system/Apple-...
- JadeNB 8y ago
  > They do plan to document it fully later: > > Is APFS open source? > > An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format specification. They claim to plan to document it fully later.
  - monocasa 8y ago
    The spec has already been released.
    - loeg 8y ago
      It's a partial spec.
- MBCook 8y ago
  What makes you think it’s a rush job? Perhaps it’s just the lawyer reviews holding it up.
  - asveikau 8y ago
saagarjha 8y ago
The APFS source code isn't available, but you're free to disassemble apfs_vnop_readdir in apfs.kext. I'm seeing a couple of calls to lck_rw_lock_shared, so it's entirely possible that there's a lock here.
- userbinator 8y ago
  This. It's what everyone I know who works in the Windows space does when they have a deep problem to investigate. Lack of source doesn't mean lack of ability to investigate, and sometimes the source doesn't tell the whole story either. (Yes, I know about the anti-RE clauses in EULAs. If they were actually enforced as strictly as they claim, people like Mark Russinovich and Matt Pietrek would've been sued out of existence long ago, along with just about every Windows security researcher.)
  - pcwalton 8y ago
    The Wine source is often helpful too.

jarym 8y ago

Pretty sure the code would expose just how much of a rush-job apple did with APFS. It may be released one day..

wmf 8y ago

Apple releases most of the macOS kernel but I guess it doesn't include APFS: https://opensource.apple.com/source/xnu/ https://opensource.apple.com/source/xnu/

NegativeLatency 8y ago
That's missing a good chunk of the interesting stuff related to how users interact with the os though. (app and framework source code)
- wmf 8y ago
  Yeah, apps and frameworks aren't part of the kernel. But APFS is.
  - Phrodo_00 8y ago
    Kind of talking out of my ass, but Darwin can act as a mirokernel, so APFS could be implemented as a user mode service (no idea if it's the case, though).
    - yjftsjthsd-h 8y ago
      Darwin's XNU is not a microkernel, in spite of there being a microkernel version of XNU. While Darwin does support FUSE, performance would almost certainly be inadequate, given the constant transitions between user and kernel space.
      - Phrodo_00 8y ago
        I thought it was a mixture of OSFMK and BSD, which is why I said it can act like a microkernel. Don't know more than what wikipedia says, though.
        rayiner 8y ago
        Yes, it is, but the Mach layer and BSD layer were fused into a monolithic kernel before NeXT Step turned into OS X. In OS X, HFS (and presumably APFS) are in the BSD layer.
  - comex 8y ago
    It's loaded into the kernel, but it's a kext (kernel extension) rather than being part of the xnu source tree itself. Only a subset of kexts are open source. (By comparison, HFS used to be in the xnu tree, but it was moved into a kext as well a few years back; that one is open source.)

mikhailt 8y ago

They do plan to document it fully later: > Is APFS open source? > An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format specification. Source: https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html https://developer.apple.com/library/archive/documentation/Fi... My theory is that Apple rushed APFS and its implementation isn't in a great shape for the public review. APFS doesn't feel any faster on my SSDs on macOS. I suspect they'll add it in a few years, after a few iterations.

twotwotwo 8y ago
They released the format spec here: https://developer.apple.com/support/apple-file-system/Apple-File-System-Reference.pdf https://developer.apple.com/support/apple-file-system/Apple-...
JadeNB 8y ago
> They do plan to document it fully later: > > Is APFS open source? > > An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format specification. They claim to plan to document it fully later.
- monocasa 8y ago
  The spec has already been released.
  - loeg 8y ago
    It's a partial spec.
MBCook 8y ago
What makes you think it’s a rush job? Perhaps it’s just the lawyer reviews holding it up.
- asveikau 8y ago

saagarjha 8y ago

The APFS source code isn't available, but you're free to disassemble apfs_vnop_readdir in apfs.kext. I'm seeing a couple of calls to lck_rw_lock_shared, so it's entirely possible that there's a lock here.

userbinator 8y ago
This. It's what everyone I know who works in the Windows space does when they have a deep problem to investigate. Lack of source doesn't mean lack of ability to investigate, and sometimes the source doesn't tell the whole story either. (Yes, I know about the anti-RE clauses in EULAs. If they were actually enforced as strictly as they claim, people like Mark Russinovich and Matt Pietrek would've been sued out of existence long ago, along with just about every Windows security researcher.)
- pcwalton 8y ago
  The Wine source is often helpful too.