22 ms·

Apple File System

1094 points by sirn 10y ago

drinchev 10y ago
This should've been in the keynote.
- waterphone 10y ago
  The keynote was for consumers and the press. The 2 PM PT Platforms State of the Union is for developers and will likely go into this more, and if not there is a session tomorrow specifically about it.
  - drinchev 10y ago
    Still... Marketing fully-encrypted snapshot-featured filesystem with focus on SSD / Flash, would make much more impact than having "native-tabs" for apps on an event like that. Anyway I'm not a specialist, just another IT guy, so maybe I'm too opinionated to argue more.
    - geodel 10y ago
      As another IT guy I found quote from Deepak Chopra a low point in Keynote :-)
    - waterphone 10y ago
      I don't think this is going to be a part of macOS Sierra as a standard feature, given that it has a release date in 2017 while Sierra will be out later this year. So it makes sense to not include it in a Sierra announcement.
    - msbarnett 10y ago
      The press understands "apps" and "tabs". The press doesn't understand "filesystem snapshots". Part of the reason Apple has historically gotten much better press from WWDC than most tech companies do from their big developer events is that they laser-focused their keynote on being Press-digestible and not full of incomprehensible tech terminology that bores non-geeks to tears. As evidenced by this thread, the people "filesystem snapshots" can be marketed to didn't need it to be mentioned in the keynote to get them talking about it. Hence, using precious keynote time for it would have been a waste of marketing resources.
      - zeckalpha 10y ago
        > The press understands "apps" and "tabs". The press doesn't understand "filesystem snapshots". That's a limited view of the press.
- pwthornton 10y ago
  My No. 1 guess as to why it wasn't in the keynote was that Apple isn't sure when this will ship. I think they are hoping for a year from now, but a new filesystem is hard to pin down.
0xCMP 10y ago
What about case sensitivity?
- killercup 10y ago
  > Current Limitations […] Case Sensitivity: Filenames are currently case-sensitive only.
cpr 10y ago
Siracusa will finally be happy. ;-) Sounds an awful like like ZFS (zero-cost clones, read-only snapshots) but could it be? I would imagine they'd start from scratch to due IP issues. Clearly this is immature technology they want to get out for testing/evaluation before it's fully adopted even into their own products. (See below.) - - - from the release notes - - - As a developer preview of this technology, there are currently several limitations: Startup Disk: APFS volumes cannot currently be used as a startup disk. Case Sensitivity: Filenames are currently case-sensitive only. Time Machine: Time Machine backups are not currently supported. FileVault: APFS volumes cannot currently be encrypted using FileVault. Fusion Drive: Fusion Drives cannot currently use APFS.
- DannyBee 10y ago
  "FileVault: APFS volumes cannot currently be encrypted using FileVault. " This one is confusing, because this is a logical volume feature. Not sure how or why APFS would ever care that some layer above it is encrypting stuff.
  - Someone 10y ago
    On the one hand, that may be an artificial limitation. If this turns out to have some bug that overwrites the encryption keys, they could of course say "we warned you", but it would not be good PR. Also, if beta developers report intermittent smaller data-losing bugs, they might want to study the affected drives to see what went wrong with it. Not having encryption enabled on them will make that a tiny bit easier. On the other hand, if the new ability to partition drives with flexible partition sizes includes separate encryption keys per partition, and encryption/decryption is done by the block driver, they may have work to do to keep that block driver informed about what blocks should get encrypted with what key.
  - leonroy 10y ago
    Could be for performance reasons rather than a technical blocker. That said AES-NI instructions do make encryption pretty fast so it's anyone's guess at this stage.
  - tptacek 10y ago
mailslot 10y ago
Another proprietary filesystem? I was really hoping they'd adopt ZFS fully.
- jasonjei 10y ago
  Apple says they intend to open-source the file system in the docs. Open Source: "An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format when Apple File System is released in 2017."
  - sirn 10y ago
    Source of the quote is this page: https://developer.apple.com/library/prerelease/content/documentation/FileManagement/Conceptual/APFS_Guide/UsingtheAppleFileSystem/UsingtheAppleFileSystem.html#//apple_ref/doc/uid/TP40016999-CH4-SW1 https://developer.apple.com/library/prerelease/content/docum... Releasing a new filesystem is already surprising. And they plan to open-source it. Huh.
  - anc84 10y ago
    Free software and patent-free or whatever some lawyers call "open"?
    - skrause 10y ago
      They said "publish the APFS volume format". So that can mean just documentation and not code at all.
      - Retra 10y ago
        Published code doesn't mean 'open' either. You need rights to the code, not merely access to it.
  - zingplex 10y ago
    Just like they promised to open source FaceTime and iMessages? Apple has a history of making false promises in regards to open sourcing products.
partiallogic 10y ago
Paging @siracusa
- oneeyedpigeon 10y ago
  And he said it would never happen!
  - noblethrasher 10y ago
    Actually, he said he thought that it would happen next year (2017).
    - cmsj 10y ago
      It's not happening until next year, the docs say it's coming in 2017, so I guess a macOS 12 point release.
- Nickoladze 10y ago
  https://twitter.com/siracusa/status/742435975079419905 https://twitter.com/siracusa/status/742435975079419905
- rvanmil 10y ago
  Can't wait for the next ATP :)
xd1936 10y ago
> It is optimized for Flash/SSD storage and features strong encryption, copy-on-write metadata, space sharing, cloning for files and directories, snapshots, fast directory sizing, atomic safe-save primitives, and improved file system fundamentals. \o/ Hallelujah, something modern!
- quantumhobbit 10y ago
  I was excited to until I read the limitations section. Can't be used as the startup disk and doesn't work with time machine. Hopefully this with change in the next macOs release.
  - xd1936 10y ago
    Sure, this is a developer preview. Final release in 2017 will surely make this FS the default.
    - lewisl9029 10y ago
      That's what people were hoping for with ReFS from Microsoft, which released years ago, but that still hasn't happened yet.
      - whorleater 10y ago
        Apple is far more willing to switch to more newer technologies than Microsoft is.
        mevile 10y ago
        One of the benefits of being vertically integrated.
        drewda 10y ago
        ... and Microsoft is far more willing than Apple to put effort into backwards compatibility.
ulfw 10y ago
Shipping in 2017? So it won't even be ready for Sierra?
- deleted 10y ago
  [deleted]
- JonathonW 10y ago
  It's present as a developer preview in Sierra, but isn't the default filesystem and can't be used as the boot drive. Which is probably how it should be. I'd rather not lose data to a new filesystem because Apple rushed the thing out without proper testing.
  - jasonjei 10y ago
    Seconded. File systems are hard and specs need to be finalized. Plus the grandparent post reminds me of the old Slashdot meme: ``No wireless. Less space than a nomad. Lame.''
deleted 10y ago
[deleted]
jxy 10y ago
Just saw a comment in another thread, stating Apple had slipped in improving their Unix layer, and here comes this. New file system is not a joking matter: Microsoft failed to deliver their new FS; Linux took years to go from ext2 to ext3 to ext4, and btrfs is in forever testing; most of *BSD still use their old ones; zfs took decade to become mainstream... The information currently is very scarce on this one, but I hope they would at least test it REALLY WELL.
- izacus 10y ago
  Didn't Microsoft introduce ReFS with many modern FS features?
  - lossolo 10y ago
    They did, it's more fault tolerant than NTFS, i am using it on my Windows machine for my sensitive data with storage spaces.
  - spikengineer 10y ago
    They don't consider it primetime yet.
  - wtallis 10y ago
    Yes, and the list of NTFS features it dropped was just as long.
- skykooler 10y ago
  SailfishOS uses btrfs.
- malcolmgreaves 10y ago
  Sounds like a useless Apple project that is going to die pretty soon :(
- rhinoceraptor 10y ago
  And Linux has the blessing/curse of having a million distros, each implementing their installer a little differently, exposing subtle bugs and inconsistencies in the FS/bootloader department. And AFAIK, GNU Grub hasn't done a release in about four years now, and all the distros are using their own, custom beta build of it. It's a bit of a mess.
lossolo 10y ago
Could anyone outline advantages and disadvantages comparing to ext4 or other file systems?
- 2bitencryption 10y ago
  It's not HFS. That in itself is a pretty huge advantage.
  - amock 10y ago
    That's not really a useful statement. HFS+ has worked pretty well for many years and is really well tested. This will be a completely new filesystem, which means it will have lots of new code that can have bugs or incompatibilities with current apps. APFS sounds like it's going to be great, but I think not being HFS is a disadvantage rather than an advantage.
    - hyperpape 10y ago
      It's tested, so there's less risk of new and unexpected bugs, but the known problems are significant: https://blog.barthe.ph/2014/06/10/hfs-plus-bit-rot/ https://blog.barthe.ph/2014/06/10/hfs-plus-bit-rot/.
      - astrange 10y ago
        HFS+ didn't lose any files, his hard drive did. He wouldn't necessarily be able to get them back with filesystem-level checksum either, but he would know they're gone faster. BTW, disk images already provide checksumming, and so does authenticated encryption.
    - MBCook 10y ago
      > HFS+ has worked pretty well for many years and is really well tested Nope. HFS+ is REALLY GOOD at losing data or having silent corruption. I'm thrilled to have a new filesystem just because it's not HFS based. I'd celebrate having NTFS as the official filesystem. It's much more modern than HFS+.
skrause 10y ago
No checksumming?
- amock 10y ago
  I also didn't see any mention of compression. Hopefully those are at least being worked on and just didn't make it into the docs yet.
  - tambourine_man 10y ago
    HFS+ already supports compression and it says APFS would support nearly all of the features of HFS+, so there's hope.
    - amock 10y ago
      I had forgotten about that. I don't really know how HFS+ compression works, so I'll wait and see APFS turns out.
      - ryao 10y ago
        HFS+ uses zlib (gzip).
        dolken 10y ago
        As far as I know only for read-only volumes...
- kylec 10y ago
  Checksumming is the thing that I want the most from a new Apple FS. I really hope they include it at some point.
- derefr 10y ago
  Heck, if someone's going whole-hog on file integrity, I'd really like them to support the creation of files that are stored "expanded" by a fountain-code. Rather than having to RAID-mirror every block on my disk, I'd like to be able to pick just some files and say "please store those ones slightly more redundantly, such that they're protected from bit-level disk errors—they're important."
lathiat 10y ago
Much like other modern filesystems such as ZFS and BTRFS it supports multiple filesystems in a shared storage space, snapshots, copy-on-write and also has encryption built-in as a first class feature including multiple keys and per-file keys. There are many limitations in the developer beta so this is clearly still very much a work in progress. Getting these file-systems right is traditionally difficult and can take years (see ZFS, BTRFS) so it will be interesting to see how well it does.
- pmlnr 10y ago
  It's the transparent compression I'm using btrfs for. Only that, ZFS and NTFS support it, so not much choices for that scenario.
cjbprime 10y ago
I hope they don't patent it and go after Linux implementations, as Microsoft did with exfat.
- baldfat 10y ago
  Well the issue was that was a closed file system owned by Microsoft and used on Linux for Flash Drives and SD Cards. If you sue closed technology you get burned. I didn't have a problem with Microsoft doing what they did. We should have used an open file system for those devices. I personally have used ext2/rfs on my drives and it is a pretty light security system ;)
  - ryao 10y ago
    I would expect Apple's driver to be open source, so someone could just port it. Their HFS+ driver is open source.
  - dchest 10y ago
    There's no such thing as "closed" or "open": something can be open source (copyright law), but patented (patent law), and it can be closed sourced, but not patented.
jbergstroem 10y ago
No snapshotting yet, can't boot, no migration tools for existing setups (time machine, filevault, fusion). It'd be interesting to see whether their permission changes introduced in 10.11 will travel further down the fs. Also, I don't see any mention of compression like lz4. Lots of work until release in 2017!
- deleted 10y ago
  [deleted]
- MBCook 10y ago
  Is compression really that big a deal any more?
  - entee 10y ago
    I heard there was this Pied Piper startup, it's making big waves in compression technology ;)
  - STRML 10y ago
    While they're still shipping machines with < 1TB SSDs (read: for a long time), yes - why not take the relatively free space and I/O savings?
    - bluegreyred 10y ago
      Because you can upsell a larger iCloud subscription to your customers using the new built in automatic cloud archival function. Then use compression server-side. $$$
    - brigade 10y ago
      On the other hand, people (sample size: my family and friends) tend to fill up their drives with already-compressed video and photos, which lz4 and similar cannot really compress further.
  - feld 10y ago
    Yes, it can make I/O faster
pieterr 10y ago
> You can share APFS formatted volumes using the SMB network file sharing protocol. The AFP protocol is deprecated and cannot be used to share APFS formatted volumes. AFP deprecated too.
killercup 10y ago
> Flash / SSD Optimization I.e., a "unique copy-on-write design" > Space Sharing Basically, ZFS datasets. > Snapshots If those can be sent: Finally Time Machine done right. > The AFP protocol is deprecated and cannot be used to share APFS formatted volumes. Interesting. > An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format when Apple File System is released in 2017. Yay! Limitations: https://developer.apple.com/library/prerelease/content/documentation/FileManagement/Conceptual/APFS_Guide/UsingtheAppleFileSystem/UsingtheAppleFileSystem.html#//apple_ref/doc/uid/TP40016999-CH4-DontLinkElementID_24 https://developer.apple.com/library/prerelease/content/docum... Edit: > encryption models for each volume in a container: no encryption, single-key encryption, or multi-key encryption with per-file keys for file data and a separate key for sensitive metadata Nice. I hope they also include checksums for each block. Famously missing, but not the hardest thing to add considering all the features above: Compression (which HFS+ supports!)
- PinguTS 10y ago
  > The AFP protocol is deprecated and cannot be used to share APFS formatted volumes. Interesting. That is really interesting. I recognised with El Capitan that it already defaults to SMB instead of AFP when you don't give a scheme.
  - duskwuff 10y ago
    AFP depends on having persistent, globally addressable IDs for files (CNIDs). Perhaps this is no longer available under APFS? While I imagine this might be retrofittable (netatalk manages it, after all), I can't blame Apple for wanting to ditch AFP. It's an ancient protocol at this point, and I'm frankly just amazed it still works at all.
  - secalex 10y ago
    AFP is also a security disaster. Check out the spec for "DHX2" https://developer.apple.com/library/mac/documentation/Networking/Conceptual/AFP/AFPSecurity/AFPSecurity.html https://developer.apple.com/library/mac/documentation/Networ...
geodel 10y ago
Any guess if AFS is written in Swift?
- ysleepy 10y ago
  Probably C++ like the rest of IOKit. https://en.wikipedia.org/wiki/I/O_Kit https://en.wikipedia.org/wiki/I/O_Kit http://opensource.apple.com/source/xnu/xnu-3248.20.55/iokit/ http://opensource.apple.com/source/xnu/xnu-3248.20.55/iokit/
rbanffy 10y ago
> Case Sensitivity: Filenames are currently case-sensitive only. Sanity, at last! (even if temporary)
- zymhan 10y ago
  But Photoshop!
  - spikengineer 10y ago
    Adobe has to get their shit right.
DannyB2 10y ago
Complain if you want, but you seem to forget that it is from Apple. And that makes it somehow better with magical goodness that nobody else could attain. Apple's engineers are not mere humans like every other 'ordinary' tech company would hire. /s
- enraged_camel 10y ago
  >>And that makes it somehow better with magical goodness that nobody else could attain. I don't know about that, but there is definitely something magical about these keynotes that makes Apple haters come out of the woodwork.
- dang 10y ago
  Please don't do this here. Comments like this make the discussion quite a bit worse in their own right. We detached this subthread from https://news.ycombinator.com/item?id=11897039 https://news.ycombinator.com/item?id=11897039 and marked it off-topic.
  - mpweiher 10y ago
    Former Apple engineer here. Yes, it sounds snide on the surface, but there is a lot of truth to it. I actually remember those of us on the inside being rather a- and bemused by the "magic" qualities we and our output had suddenly acquired.
    - dang 10y ago
      The trouble with snidely expressed truth is that the snideness ends up dominating the discussion more than the truth does, and these effects compound over time. Therefore it's important to focus on simply expressing the truth as one understands it, as indeed you did in your comment here. This is one point where the discourse on a large internet forum differs from that of smaller, more cohesive groups. We have to be vigilant about this on HN because we're more vulnerable to those compounding effects. The discussion doesn't naturally right itself and return to interesting things.
Philipp__ 10y ago
Wow! This makes me feel good, maybe Apple gets it that many of it's users use it's devices because they present proper UNIX desktop environment. And that environment needs to be cherished and it needs to evolve. This is not a small thing. We had nice visual overhaul 2 years ago, now Apple needs to pick it up on under the hood level.
tdicola 10y ago
I'm going to be very cynical and say that based on their track record of major OS overhauls I am _not_ looking forward to the bugs and issues that will sneak past their QA. I still have nightmares of all the bugs with their wifi & USB stack changes in the last couple OSX releases. Please Apple, for your own good don't 'move fast and break things' with the filesystem. At the very least it's time for everyone to make sure they have a good backup system in place before touching this thing.
- killercup 10y ago
  I think they'll only call this 'stable' once they enable it on new iPhones by default. Which might be 2017, 2018, …
- MBCook 10y ago
  Compared to how risky and problematic HFS+ is? It may still be better.
  - cmurf 10y ago
    Exchanging problems you know for problems you don't know.
entee 10y ago
It would be really nice is there was a modern file system that "just worked" regardless of device. I'm tired of having exFAT/FAT being the only filesystem I can reliably use on multiple different OS-es painlessly, and even then I can't use it for all the functions of those OS-es (No time machine). Hopefully this will be open enough to enable that, though who knows how it will shake out.
- regularfry 10y ago
  https://en.wikipedia.org/wiki/Universal_Disk_Format https://en.wikipedia.org/wiki/Universal_Disk_Format ? That looked hopeful last I was juggling with this sort of thing, but that was long enough ago that WinXP support was the killer.
  - duskwuff 10y ago
    From what I understand, UDF is primarily targeted at one-time-recordable media. (It's used in DVDs, for example.) It's unclear how well it works for primary storage, but I suspect it's clumsy at best.
    - regularfry 10y ago
      It's only got to beat FAT. That's a pretty low bar.
      - setpatchaddress 10y ago
        One that it doesn't meet. Try to read and understand the UDF spec sometime.
  - Sanddancer 10y ago
    Getting UDF to work with rewritable media, like usb sticks, is an unholy pain in the arse. It involves trying to remember an arcane combination of versions and feature flags to get the universal format to actually be universal. I managed it once, but after that, realized that the network was fast enough for most of what I needed and I tend to not have the four gig files that cause problems with FAT filesystems.
busterarm 10y ago
Give me case sensitivity and file dates that go past Feb 6th 2040 please. :D
notacoward 10y ago
I'm notoriously skeptical of anything from the local-filesystem world, and I'm no Apple fanboy, but this looks pretty cool. I look forward to hearing how some of it works, especially what's new about their version of COW and why end-to-end checksumming wasn't mentioned.
ysleepy 10y ago
Uh interesting thought: With Space sharing and multiple logical volumes there will probably be a shift to encrypting users home directories separately. Probably even an unencrypted System Base, which is read-only and protected by rootless anyway, so the system can boot without user interaction. This also finally allows the most user friendly implementation of file encryption: Encrypted Folders, supported by the OS. Sounds cool, Id be willing to help with a rust port once the source is available :) (case sensitive only - naturally)
- derefr 10y ago
  The point of encrypting the base system image is mostly to make it part of the Secure Boot chain. rootless is a policy control to stop processes from modifying the OS from inside. An encrypted system image stops things with hardware access (but not the unlock key) from modifying the OS from the outside. You can trust any disk whose blocks you can decrypt with key X, to have been only written to by someone with key X.
  - majewsky 10y ago
    Why would you need encryption for the Secure Boot chain? Would you not rather sign the base system image for that purpose?
    - derefr 10y ago
      You'd have to sit there and read the entire base system image before booting to verify that signature. But only have to decrypt a block when it comes time to read that block—with the ability to kernel-abort right then during the boot process if the block doesn't decrypt. (Or were you suggesting individually signing every block?)
      - nwf 10y ago
        You don't sign the whole image as a stream, and you don't sign every block. Recursion is your friend! You sign the Merkle tree root, check it once, and then check O(log n) hashes per block access. You can, of course, amortize the checking of the first several of those hashes as a further optimization that ties in easily with your caching layer. There's no such thing as "the block doesn't decrypt" absent MACs/MICs or AEAD schemes -- encryption and decryption are just maps from N bytes to N bytes.
crest 10y ago
I wonder if Apples acquisition of acunu.com is finally paying off. Is this file system based on stratified B-trees?
pducks32 10y ago
I read the link and have a pretty good understanding about how computers and file systems work but can someone "explain like I'm a decently intelligent programmer" what is different about this file system? Thanks you.
- cpr 10y ago
  They simply started from scratch with an aim to * best support modern hardware technologies, * start from the state of the art in file systems (like ZFS and btrfs, old as they may be), and also * emphasize security.
  - astrodust 10y ago
    Sounds a lot like what Apple did with Swift.
- josteink 10y ago
  It's apple reinventing the zfs and btrfs wheels. Which are already mature, useable and open source. Google them and see what they do well, and you might get some ideas.
  - tracewpearson 10y ago
    Both filesystems have copyleft licenses so that may factor into Apples decision.
- Eric_WVGG 10y ago
  This bit from 2012 by John Siracusa outlines everything “wrong” with the old file system, HFS+, and doubles as a sort of guide to what this new file system fixes. http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/ http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/
bluegreyred 10y ago
I didn't see any mentions of check-summing. Please tell me this brand new filesystem won't be susceptible to bitrot?
franciscop 10y ago
I cannot see anything on my Android mobile S:
Bytes 10y ago
This is a huge feature. I have no idea why Apple didn't include this in their keynote.
- methyl 10y ago
  Probably too technical for a typical keynote watcher
  - kristofferR 10y ago
    Typical World Wide Developer Conference watcher? If so, that's a shame!
    - Razengan 10y ago
      Well, have you seen the comments on Reddit and MacRumors and the like? You'd think anything above "dark mode" and a new MacBook model would be over their heads. Most people do seem to assume that the WWDC is an hour~ long commercial for everything Apple is going to do for the rest of the year.
    - huxley 10y ago
      Opening Keynote is really for the press, it'll probably be mentioned more prominently in the State of the Union keynotes and in its own session.
- Jerry2 10y ago
  It was featured in Platforms State of the Union... which is the "real" keynote targeted at developers. You will be able to see it here shortly: https://developer.apple.com/videos/play/wwdc2016/102/ https://developer.apple.com/videos/play/wwdc2016/102/
VeejayRampay 10y ago
Website doesn't work at all on my Nexus 5x. Any other company would get lambasted for that kind of stuff.
deleted 10y ago
[deleted]
amelius 10y ago
I hope finally a filesystem that allows me to unplug a USB drive (while it is still mounted) without potentially corrupting it.
- userbinator 10y ago
  Any filesystem that doesn't cache writes will basically have that property, unless you're talking about unplugging while write operations are happening, in which case all bets are off.
  - wmf 10y ago
    The point of journaling/soft updates/COW/etc. is that all bets are not off; they're intended to prevent the fs from being corrupted in those sorts of situations.
ahl 10y ago
The spartan description of APFS certainly sounds like the (partial) feature list for ZFS--the comparisons made in the comments here are on-point. ZFS though took around 5 years to ship and, arguably, another 5-10 to get right. I say this having shipped multiple products based on ZFS, writing code in ZFS, and diagnosing production problems with it. On-disk consistency ("crash protection"), snapshots, encryption, and transactional interfaces ("atomic safe-save") will no doubt be incredibly valuable. I don't think though though that APFS will dramatically improve upon the time it took ZFS to mature from a first product to world-class storage. Some commenters have opined that (despite Apple distributing ZFS for Mac OS X at WWDC nearly a decade ago) that ZFS would never be appropriate for the desktop, phone, or watch. True ZFS was designed for servers and storage servers, but I don't think there's anything that makes it innately untenable in those environments--even its default, but not essential, use of lots of RAM. Who knows... maybe Apple have spent the decade since killing their internal ZFS port taking this new filesystem though the paces. Its level of completeness though would suggest otherwise.
- watersb 10y ago
  Yes. I was a heavy user of ZFS on Mac OS X back then. Was on the developer mailing list for that implentation, submitting bugs and talking to Apple developers working like mad to ship it. Then Steve Jobs' buddy Larry bought Sun, and licensing of ZFS became basically impossible to sort out on time. So they dropped it. Was that 2007 or 2009?
  - ahl 10y ago
    The Sun acquisition was announced in 2009 and closed in 2010. It may have been related, but I have my own suspicions about about what caused Apple to jettison their port.
    - rafaelferreira 10y ago
      Do tell.
    - astrodust 10y ago
lai 10y ago
Oh I forgot, you can't read Apple documentation on a mobile chrome browser.
- watersb 10y ago
  Last week, Apple modified their formatting for the User Support Forums, so I can actually see readable text on an iOS device. That only took... a very very long time. (Wonder if Radar has a new look...)
- nullymcnull 10y ago
  Even on Chrome desktop, if you turn on Mobile emulation and switch it to appear to be any Android device w/ phone display size, it fails in the same way - yet switch it to 'iPhone 5' or 'iPhone 6' and the same compact layout works fine. They're pointlessly sniffing and special-casing 'Android' user-agents and botching it completely.
rleigh 10y ago
While I am happy that Apple have at last committed to replacing HFS+, I'm wondering why they didn't use ZFS rather than reinventing the wheel. It's not like it's a particularly easy wheel to reinvent either; the amount of effort which goes into a filesystem like ZFS is non-trivial. Why not build on top of that? I would have greatly appreciated being able to use ZFS with MacOS X, for datasets, snapshots, sending them to remote pools for backup etc. It would have made it directly interoperable with a lot of pre-existing and cross-platform infrastructure. (I couldn't care less if it didn't scale down to the "watch". Filesystems are not a one-size-fits-all affair.) I find it great that I can take a set of disks from e.g. Linux, run "zpool export", pull them out, and then shovel them into a FreeBSD system, run "zpool import" and have the pool and datasets reassembled and automatically mounted. Perfectly transparent interoperability and portability. While Apple like to do their own thing, this is one place I would have definitely appreciated some down to earth pragmatism and re-use of existing battle-tested and widely used technology.
- wcummings 10y ago
  How does AAPL feel about the CDDL?
  - wstrange 10y ago
    Given that there are other companies shipping ZFS without Oracle's blessing, I am curious why Apple could not do the same? (honest question) Is it that Apple is a much bigger target? Patent issues?
    - zaphar 10y ago
      Many of those companies probably started using ZFS before Oracle acquired Sun. After Oracle acquired Sun the risks of using ZFS skyrocketed legally and Apple had the option of just not taking on that risk since they hadn't shipped it yet. Given what we know about Oracle that was probably the right call.
  - jdub 10y ago
    They ship dtrace.
CoachRufus87 10y ago
Forgive my ignorance, but what benefits would an average user experience with this new file system?
- nisa 10y ago
  In theory (the practical side is tricky to get right): - More resilience against data corruption for one due to copy on write and likely checksumming - so in theory power failure should not harm the filesystem and you can roll back to a last known state and only lose data for the last few seconds if at all. - Snapshots, before installing updates you can snapshot the filesystem and roll back instantly. You could also create a readonly snapshot to have consistent backups. This should also allow seamless rollback of OS updates. There are also clones supported e.g. multiple diverging versions of a file tree - think something similar to git branches on a filesystem level. - Native encryption at the filesystem level - Possible better handling of metadata. Faster access to directories with lot's of files, more files on a volume without slowdown. - Sparse files - you can create huge files instantly and fill them later, guess it's important for various types of software e.g. virtual machine images. But as nice as these properties are in theory history showed that that in practice a lot can go wrong.
  - CoachRufus87 10y ago
    Thank you for the explainer.
watersb 10y ago
If you create a ZFS filesystem on a non-macOS, you have to be careful about Unicode mapping in order to use it safely on MacOS. I have no idea if Unicode is an issue for Apple File System. Emoji file names, you guys.
- cynix 10y ago
  > Emoji file names, you guys. But are they 3x bigger?
0x0 10y ago
Hopefully this will be the end of .DS_Store and the crazy unicode normalization issue that causes mixups when rsync-roundtripping a directory structure between ext4 and HFS! :)
- nneonneo 10y ago
  Unicode support on filesystems is just an ugly mess, and APFS is unlikely to fix this (the brief documentation doesn't mention it at all). Let's recap: - On OS X HFS+, filenames are stored using a "variant" of NFD, where some characters are precomposed "for compatibility with old Mac text encodings" (https://developer.apple.com/library/mac/qa/qa1173/_index.html https://developer.apple.com/library/mac/qa/qa1173/_index.htm...). - On Windows NTFS, filenames are "opaque sequences of WCHARs", and are thus "kind of" UTF-16 with no formally required normalization format. Windows itself tries to use NFC, but applications are free to use the Windows APIs to create a filename with anything they like. Since filenames are just sequences of 16-bit WCHARs, dangling surrogate pairs are allowed (and can break all sorts of code!) - On Linux, filenames are opaque sequences of 8-bit characters. The only requirement is that a filename not contain either a slash or NUL character. No other formal specification exists, although "most" users these days use UTF-8. (However, you can and will find loads of filesystems with invalid UTF-8, usually because filenames are in one of the ISO encodings instead). Multiple programming languages have been bitten by the possibility of invalid Unicode in filenames (see for example: rust (https://github.com/rust-lang/rust/issues/12056 https://github.com/rust-lang/rust/issues/12056), Python (https://www.python.org/dev/peps/pep-0383/) https://www.python.org/dev/peps/pep-0383/)). This mess is pretty much never going to go away, either, because filesystems are extremely durable and long-lasting.
  - 0x0 10y ago
    So the main thing I've observed is rsync'ing web files from a linux server to an osx laptop, and back to the linux server, ends up with a bunch of duplicate decomposed utf8 filenames on the linux server. It can be avoided with careful use of rysnc's "--iconv=utf-8-mac,utf-8" etcetera, but it feels super-unnecessary. Fix it already :D More details here http://serverfault.com/a/427200 http://serverfault.com/a/427200 PS: I've been using LC_CTYPE="whatever.ISO8859-1" and an ISO-8859-1 Terminal.app locale forever since I seem to keep dragging a bunch of legacy filenames around (having started on MS-DOS and FreeBSD 2.2) and ISO-8859-1 still seems to be the only locale that lets me "see the bytes" matrix-style instead of a random amount of "?" chars. Curiously, Finder.app seems to keep up very very well despite the odd encoding. Crossing my fingers the new APFS will act more like Linux. PPS: Java is especially hilarious when launched with -Dfile.encoding=utf-8 as it is literally impossible to access some files from there. And don't get me started on .DS_Store :P
comex 10y ago
I had been nervous that if and when Apple finally got around to making a new file system, they would follow the trend and neither open source it nor document the on-disk format. HFS+ itself has always been open by virtue of being included in the open-sourced core xnu kernel, but Apple's CoreStorage volume manager (the basis for FileVault and Fusion Drive) and DMG disk image format have always lived in closed-source kernel extensions and are not officially documented; Microsoft has acted similarly with its NTFS, ReFS, and exFAT file systems and Storage Spaces volume manager. The result, of course, is that nothing can access anything else's file system, at least well. Most of the formats I mentioned have been reverse engineered to some extent, but there are limits, especially when you're scared of corrupting data by misinterpreting the format and therefore limit your implementation to read access. If you want read-write access to NTFS from Linux or macOS, a filesystem which is old enough to drink, you have to rely on either a relatively slow FUSE implementation or some proprietary software. If you dual boot Linux on your Mac and want access to the Mac partition, make sure to turn off disk encryption (and hope you don't run into any evil maids) unless you want to use a read-only FUSE tool, and I hope you don't have a Fusion Drive because I don't think anything supports that at all (could be wrong). To be fair, this situation cannot be blamed entirely on lack of openness. Lack of interest matters too; as I said, HFS+ has an open source implementation, yet Linux's hfsplus driver doesn't support journaling, a feature added in 2002. But if anyone tries to improve the situation, that implementation makes their job a lot easier: for one thing, since filesystem support is in the BSD-based portion of the xnu kernel, it may be possible to port the HFS implementation to other BSDs with relative ease - like the NetBSD rump kernel, which can run in userspace, from which FUSE support wouldn't be that hard... Alternately, if they chose the path of enhancing the native Linux driver, at least they could consult Apple's code to be sure they weren't missing any obscure corners of the format that could cause their driver to corrupt data. Anyway, what we got with Apple File System is this: > An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format when Apple File System is released in 2017. That's pretty good! Open source plus documentation would be ideal, but documentation is much better than nothing[1], and the text of that paragraph doesn't exactly rule out the possibility of the final release being open source too. (I haven't installed the beta yet, but am I right in guessing that APFS is implemented in a kext?) While there's no guarantee, I hope that that documentation will also come with information about the Core Storage format APFS will usually be wrapped in, so that proper interoperability can be achieved. Rather than my fear coming to pass of one of the last vestiges of openness in proprietary OS storage formats disappearing, the situation may actually be improved compared to today. Maybe in a few years I'll be able to install Ubuntu on my Mac and have full access to the OS X partition out of the box. I can hope... [1] whether it's better or worse than open source alone is debatable - source can be harder to understand, but it's also never wrong, unlike documentation...
cm3 10y ago
Does anyone know if Dominic Giampaolo was involved with the design?
- gyc 10y ago
  He's still at Apple so I would assume he had a major role.
  - cm3 10y ago
    I hope so, though I kinda miss the pervasive use of metadata as in BeFS. That was before its time and such a nice concept. Coupled with saved queries, it's unmatched to this day in mainstream, general purpose file system semantics.
DeepYogurt 10y ago
It seems to be a b-tree based file system that isn't zfs or btrfs. Neat.
- rleigh 10y ago
  Note that ZFS doesn't use b-trees; it uses Merkle hashes.
PakG1 10y ago
This is interesting because just yesterday I saw a discussion here on HN about how sad people were that Apple was doing nothing to improve OSX and was putting all the focus on iOS instead. Heh.
FunctionalMan 10y ago
Finally catching up to NTFS/ReFS! Congratulations, Apple.
jackjeff 10y ago
I don't see any mention of data integrity... Unless you use multi-key encryption. I find that a bit odd.
jackjeff 10y ago
I don't see any mention of data integrity... Unless you use multi-key encryption. I find that a bit odd.
Black-Plaid 10y ago
> Case Sensitivity: Filenames are currently case-sensitive only. Dominic, lol.
Chyzwar 10y ago
Arrogance... Instead using ZFS they build new file system from scratch (APFS) Instead using Vulkan they build new Graphics API from scratch (Metal) Keep this up and they will end like MS.
- chucknelson 10y ago
  I'd be more interested in understanding why they chose to implement their own file system instead of adopting something like ZFS. No doubt there is reasoning in there besides the "we are an evil company" logic - Apple developers are people too...
  - Chyzwar 10y ago
    I just do not understand rationale. File systems is hard stuff, instead taking something that is probably best file system around they decided for long and painful road. Note that they worked before on OpenZFS in Apple ....
- taharvey 10y ago
  ummmm... metal cames out 2 years before vulkan. Vulkan was inspired by metal.
  - Chyzwar 10y ago
    nope, both are inspired by AMD Mantle. Apple not only ignored first major consensus in Games industry it also blocked support on its devices. It also stuck with old OpenGL. This will make gaming on Apple to suck for many years to come.
sigzero 10y ago
I hope this comes about unlike the ZFS fiasco.
kevinSuttle 10y ago
I wonder how APFS compares to IPFS. https://ipfs.io/ https://ipfs.io/

Apple File System

1094 points by sirn 10y ago

drinchev 10y ago

This should've been in the keynote.

waterphone 10y ago
The keynote was for consumers and the press. The 2 PM PT Platforms State of the Union is for developers and will likely go into this more, and if not there is a session tomorrow specifically about it.
- drinchev 10y ago
  Still... Marketing fully-encrypted snapshot-featured filesystem with focus on SSD / Flash, would make much more impact than having "native-tabs" for apps on an event like that. Anyway I'm not a specialist, just another IT guy, so maybe I'm too opinionated to argue more.
  - geodel 10y ago
    As another IT guy I found quote from Deepak Chopra a low point in Keynote :-)
  - waterphone 10y ago
    I don't think this is going to be a part of macOS Sierra as a standard feature, given that it has a release date in 2017 while Sierra will be out later this year. So it makes sense to not include it in a Sierra announcement.
  - msbarnett 10y ago
    The press understands "apps" and "tabs". The press doesn't understand "filesystem snapshots". Part of the reason Apple has historically gotten much better press from WWDC than most tech companies do from their big developer events is that they laser-focused their keynote on being Press-digestible and not full of incomprehensible tech terminology that bores non-geeks to tears. As evidenced by this thread, the people "filesystem snapshots" can be marketed to didn't need it to be mentioned in the keynote to get them talking about it. Hence, using precious keynote time for it would have been a waste of marketing resources.
    - zeckalpha 10y ago
      > The press understands "apps" and "tabs". The press doesn't understand "filesystem snapshots". That's a limited view of the press.
pwthornton 10y ago
My No. 1 guess as to why it wasn't in the keynote was that Apple isn't sure when this will ship. I think they are hoping for a year from now, but a new filesystem is hard to pin down.

0xCMP 10y ago

What about case sensitivity?

killercup 10y ago
> Current Limitations […] Case Sensitivity: Filenames are currently case-sensitive only.

cpr 10y ago

Siracusa will finally be happy. ;-) Sounds an awful like like ZFS (zero-cost clones, read-only snapshots) but could it be? I would imagine they'd start from scratch to due IP issues. Clearly this is immature technology they want to get out for testing/evaluation before it's fully adopted even into their own products. (See below.) - - - from the release notes - - - As a developer preview of this technology, there are currently several limitations: Startup Disk: APFS volumes cannot currently be used as a startup disk. Case Sensitivity: Filenames are currently case-sensitive only. Time Machine: Time Machine backups are not currently supported. FileVault: APFS volumes cannot currently be encrypted using FileVault. Fusion Drive: Fusion Drives cannot currently use APFS.

DannyBee 10y ago
"FileVault: APFS volumes cannot currently be encrypted using FileVault. " This one is confusing, because this is a logical volume feature. Not sure how or why APFS would ever care that some layer above it is encrypting stuff.
- Someone 10y ago
  On the one hand, that may be an artificial limitation. If this turns out to have some bug that overwrites the encryption keys, they could of course say "we warned you", but it would not be good PR. Also, if beta developers report intermittent smaller data-losing bugs, they might want to study the affected drives to see what went wrong with it. Not having encryption enabled on them will make that a tiny bit easier. On the other hand, if the new ability to partition drives with flexible partition sizes includes separate encryption keys per partition, and encryption/decryption is done by the block driver, they may have work to do to keep that block driver informed about what blocks should get encrypted with what key.
- leonroy 10y ago
  Could be for performance reasons rather than a technical blocker. That said AES-NI instructions do make encryption pretty fast so it's anyone's guess at this stage.
- tptacek 10y ago

mailslot 10y ago

Another proprietary filesystem? I was really hoping they'd adopt ZFS fully.

jasonjei 10y ago
Apple says they intend to open-source the file system in the docs. Open Source: "An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format when Apple File System is released in 2017."
- sirn 10y ago
  Source of the quote is this page: https://developer.apple.com/library/prerelease/content/documentation/FileManagement/Conceptual/APFS_Guide/UsingtheAppleFileSystem/UsingtheAppleFileSystem.html#//apple_ref/doc/uid/TP40016999-CH4-SW1 https://developer.apple.com/library/prerelease/content/docum... Releasing a new filesystem is already surprising. And they plan to open-source it. Huh.
- anc84 10y ago
  Free software and patent-free or whatever some lawyers call "open"?
  - skrause 10y ago
    They said "publish the APFS volume format". So that can mean just documentation and not code at all.
    - Retra 10y ago
      Published code doesn't mean 'open' either. You need rights to the code, not merely access to it.
- zingplex 10y ago
  Just like they promised to open source FaceTime and iMessages? Apple has a history of making false promises in regards to open sourcing products.

partiallogic 10y ago

Paging @siracusa

oneeyedpigeon 10y ago
And he said it would never happen!
- noblethrasher 10y ago
  Actually, he said he thought that it would happen next year (2017).
  - cmsj 10y ago
    It's not happening until next year, the docs say it's coming in 2017, so I guess a macOS 12 point release.
Nickoladze 10y ago
https://twitter.com/siracusa/status/742435975079419905 https://twitter.com/siracusa/status/742435975079419905
rvanmil 10y ago
Can't wait for the next ATP :)

xd1936 10y ago

> It is optimized for Flash/SSD storage and features strong encryption, copy-on-write metadata, space sharing, cloning for files and directories, snapshots, fast directory sizing, atomic safe-save primitives, and improved file system fundamentals. \o/ Hallelujah, something modern!

quantumhobbit 10y ago
I was excited to until I read the limitations section. Can't be used as the startup disk and doesn't work with time machine. Hopefully this with change in the next macOs release.
- xd1936 10y ago
  Sure, this is a developer preview. Final release in 2017 will surely make this FS the default.
  - lewisl9029 10y ago
    That's what people were hoping for with ReFS from Microsoft, which released years ago, but that still hasn't happened yet.
    - whorleater 10y ago
      Apple is far more willing to switch to more newer technologies than Microsoft is.
      - mevile 10y ago
        One of the benefits of being vertically integrated.
      - drewda 10y ago
        ... and Microsoft is far more willing than Apple to put effort into backwards compatibility.

ulfw 10y ago

Shipping in 2017? So it won't even be ready for Sierra?

deleted 10y ago
[deleted]
JonathonW 10y ago
It's present as a developer preview in Sierra, but isn't the default filesystem and can't be used as the boot drive. Which is probably how it should be. I'd rather not lose data to a new filesystem because Apple rushed the thing out without proper testing.
- jasonjei 10y ago
  Seconded. File systems are hard and specs need to be finalized. Plus the grandparent post reminds me of the old Slashdot meme: ``No wireless. Less space than a nomad. Lame.''

deleted 10y ago

[deleted]

jxy 10y ago

Just saw a comment in another thread, stating Apple had slipped in improving their Unix layer, and here comes this. New file system is not a joking matter: Microsoft failed to deliver their new FS; Linux took years to go from ext2 to ext3 to ext4, and btrfs is in forever testing; most of *BSD still use their old ones; zfs took decade to become mainstream... The information currently is very scarce on this one, but I hope they would at least test it REALLY WELL.

izacus 10y ago
Didn't Microsoft introduce ReFS with many modern FS features?
- lossolo 10y ago
  They did, it's more fault tolerant than NTFS, i am using it on my Windows machine for my sensitive data with storage spaces.
- spikengineer 10y ago
  They don't consider it primetime yet.
- wtallis 10y ago
  Yes, and the list of NTFS features it dropped was just as long.
skykooler 10y ago
SailfishOS uses btrfs.
malcolmgreaves 10y ago
Sounds like a useless Apple project that is going to die pretty soon :(
rhinoceraptor 10y ago
And Linux has the blessing/curse of having a million distros, each implementing their installer a little differently, exposing subtle bugs and inconsistencies in the FS/bootloader department. And AFAIK, GNU Grub hasn't done a release in about four years now, and all the distros are using their own, custom beta build of it. It's a bit of a mess.

lossolo 10y ago

Could anyone outline advantages and disadvantages comparing to ext4 or other file systems?

2bitencryption 10y ago
It's not HFS. That in itself is a pretty huge advantage.
- amock 10y ago
  That's not really a useful statement. HFS+ has worked pretty well for many years and is really well tested. This will be a completely new filesystem, which means it will have lots of new code that can have bugs or incompatibilities with current apps. APFS sounds like it's going to be great, but I think not being HFS is a disadvantage rather than an advantage.
  - hyperpape 10y ago
    It's tested, so there's less risk of new and unexpected bugs, but the known problems are significant: https://blog.barthe.ph/2014/06/10/hfs-plus-bit-rot/ https://blog.barthe.ph/2014/06/10/hfs-plus-bit-rot/.
    - astrange 10y ago
      HFS+ didn't lose any files, his hard drive did. He wouldn't necessarily be able to get them back with filesystem-level checksum either, but he would know they're gone faster. BTW, disk images already provide checksumming, and so does authenticated encryption.
  - MBCook 10y ago
    > HFS+ has worked pretty well for many years and is really well tested Nope. HFS+ is REALLY GOOD at losing data or having silent corruption. I'm thrilled to have a new filesystem just because it's not HFS based. I'd celebrate having NTFS as the official filesystem. It's much more modern than HFS+.

skrause 10y ago

No checksumming?

amock 10y ago
I also didn't see any mention of compression. Hopefully those are at least being worked on and just didn't make it into the docs yet.
- tambourine_man 10y ago
  HFS+ already supports compression and it says APFS would support nearly all of the features of HFS+, so there's hope.
  - amock 10y ago
    I had forgotten about that. I don't really know how HFS+ compression works, so I'll wait and see APFS turns out.
    - ryao 10y ago
      HFS+ uses zlib (gzip).
      - dolken 10y ago
        As far as I know only for read-only volumes...
kylec 10y ago
Checksumming is the thing that I want the most from a new Apple FS. I really hope they include it at some point.
derefr 10y ago
Heck, if someone's going whole-hog on file integrity, I'd really like them to support the creation of files that are stored "expanded" by a fountain-code. Rather than having to RAID-mirror every block on my disk, I'd like to be able to pick just some files and say "please store those ones slightly more redundantly, such that they're protected from bit-level disk errors—they're important."

lathiat 10y ago

Much like other modern filesystems such as ZFS and BTRFS it supports multiple filesystems in a shared storage space, snapshots, copy-on-write and also has encryption built-in as a first class feature including multiple keys and per-file keys. There are many limitations in the developer beta so this is clearly still very much a work in progress. Getting these file-systems right is traditionally difficult and can take years (see ZFS, BTRFS) so it will be interesting to see how well it does.

pmlnr 10y ago
It's the transparent compression I'm using btrfs for. Only that, ZFS and NTFS support it, so not much choices for that scenario.

cjbprime 10y ago

I hope they don't patent it and go after Linux implementations, as Microsoft did with exfat.

baldfat 10y ago
Well the issue was that was a closed file system owned by Microsoft and used on Linux for Flash Drives and SD Cards. If you sue closed technology you get burned. I didn't have a problem with Microsoft doing what they did. We should have used an open file system for those devices. I personally have used ext2/rfs on my drives and it is a pretty light security system ;)
- ryao 10y ago
  I would expect Apple's driver to be open source, so someone could just port it. Their HFS+ driver is open source.
- dchest 10y ago
  There's no such thing as "closed" or "open": something can be open source (copyright law), but patented (patent law), and it can be closed sourced, but not patented.

jbergstroem 10y ago

No snapshotting yet, can't boot, no migration tools for existing setups (time machine, filevault, fusion). It'd be interesting to see whether their permission changes introduced in 10.11 will travel further down the fs. Also, I don't see any mention of compression like lz4. Lots of work until release in 2017!

deleted 10y ago
[deleted]
MBCook 10y ago
Is compression really that big a deal any more?
- entee 10y ago
  I heard there was this Pied Piper startup, it's making big waves in compression technology ;)
- STRML 10y ago
  While they're still shipping machines with < 1TB SSDs (read: for a long time), yes - why not take the relatively free space and I/O savings?
  - bluegreyred 10y ago
    Because you can upsell a larger iCloud subscription to your customers using the new built in automatic cloud archival function. Then use compression server-side. $$$
  - brigade 10y ago
    On the other hand, people (sample size: my family and friends) tend to fill up their drives with already-compressed video and photos, which lz4 and similar cannot really compress further.
- feld 10y ago
  Yes, it can make I/O faster

pieterr 10y ago

> You can share APFS formatted volumes using the SMB network file sharing protocol. The AFP protocol is deprecated and cannot be used to share APFS formatted volumes. AFP deprecated too.

killercup 10y ago

> Flash / SSD Optimization I.e., a "unique copy-on-write design" > Space Sharing Basically, ZFS datasets. > Snapshots If those can be sent: Finally Time Machine done right. > The AFP protocol is deprecated and cannot be used to share APFS formatted volumes. Interesting. > An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format when Apple File System is released in 2017. Yay! Limitations: https://developer.apple.com/library/prerelease/content/documentation/FileManagement/Conceptual/APFS_Guide/UsingtheAppleFileSystem/UsingtheAppleFileSystem.html#//apple_ref/doc/uid/TP40016999-CH4-DontLinkElementID_24 https://developer.apple.com/library/prerelease/content/docum... Edit: > encryption models for each volume in a container: no encryption, single-key encryption, or multi-key encryption with per-file keys for file data and a separate key for sensitive metadata Nice. I hope they also include checksums for each block. Famously missing, but not the hardest thing to add considering all the features above: Compression (which HFS+ supports!)

PinguTS 10y ago
> The AFP protocol is deprecated and cannot be used to share APFS formatted volumes. Interesting. That is really interesting. I recognised with El Capitan that it already defaults to SMB instead of AFP when you don't give a scheme.
- duskwuff 10y ago
  AFP depends on having persistent, globally addressable IDs for files (CNIDs). Perhaps this is no longer available under APFS? While I imagine this might be retrofittable (netatalk manages it, after all), I can't blame Apple for wanting to ditch AFP. It's an ancient protocol at this point, and I'm frankly just amazed it still works at all.
- secalex 10y ago
  AFP is also a security disaster. Check out the spec for "DHX2" https://developer.apple.com/library/mac/documentation/Networking/Conceptual/AFP/AFPSecurity/AFPSecurity.html https://developer.apple.com/library/mac/documentation/Networ...

geodel 10y ago

Any guess if AFS is written in Swift?

ysleepy 10y ago
Probably C++ like the rest of IOKit. https://en.wikipedia.org/wiki/I/O_Kit https://en.wikipedia.org/wiki/I/O_Kit http://opensource.apple.com/source/xnu/xnu-3248.20.55/iokit/ http://opensource.apple.com/source/xnu/xnu-3248.20.55/iokit/

rbanffy 10y ago

> Case Sensitivity: Filenames are currently case-sensitive only. Sanity, at last! (even if temporary)

zymhan 10y ago
But Photoshop!
- spikengineer 10y ago
  Adobe has to get their shit right.

DannyB2 10y ago

Complain if you want, but you seem to forget that it is from Apple. And that makes it somehow better with magical goodness that nobody else could attain. Apple's engineers are not mere humans like every other 'ordinary' tech company would hire. /s

enraged_camel 10y ago
>>And that makes it somehow better with magical goodness that nobody else could attain. I don't know about that, but there is definitely something magical about these keynotes that makes Apple haters come out of the woodwork.
dang 10y ago
Please don't do this here. Comments like this make the discussion quite a bit worse in their own right. We detached this subthread from https://news.ycombinator.com/item?id=11897039 https://news.ycombinator.com/item?id=11897039 and marked it off-topic.
- mpweiher 10y ago
  Former Apple engineer here. Yes, it sounds snide on the surface, but there is a lot of truth to it. I actually remember those of us on the inside being rather a- and bemused by the "magic" qualities we and our output had suddenly acquired.
  - dang 10y ago
    The trouble with snidely expressed truth is that the snideness ends up dominating the discussion more than the truth does, and these effects compound over time. Therefore it's important to focus on simply expressing the truth as one understands it, as indeed you did in your comment here. This is one point where the discourse on a large internet forum differs from that of smaller, more cohesive groups. We have to be vigilant about this on HN because we're more vulnerable to those compounding effects. The discussion doesn't naturally right itself and return to interesting things.

Philipp__ 10y ago

Wow! This makes me feel good, maybe Apple gets it that many of it's users use it's devices because they present proper UNIX desktop environment. And that environment needs to be cherished and it needs to evolve. This is not a small thing. We had nice visual overhaul 2 years ago, now Apple needs to pick it up on under the hood level.

tdicola 10y ago

I'm going to be very cynical and say that based on their track record of major OS overhauls I am _not_ looking forward to the bugs and issues that will sneak past their QA. I still have nightmares of all the bugs with their wifi & USB stack changes in the last couple OSX releases. Please Apple, for your own good don't 'move fast and break things' with the filesystem. At the very least it's time for everyone to make sure they have a good backup system in place before touching this thing.

killercup 10y ago
I think they'll only call this 'stable' once they enable it on new iPhones by default. Which might be 2017, 2018, …
MBCook 10y ago
Compared to how risky and problematic HFS+ is? It may still be better.
- cmurf 10y ago
  Exchanging problems you know for problems you don't know.

entee 10y ago

It would be really nice is there was a modern file system that "just worked" regardless of device. I'm tired of having exFAT/FAT being the only filesystem I can reliably use on multiple different OS-es painlessly, and even then I can't use it for all the functions of those OS-es (No time machine). Hopefully this will be open enough to enable that, though who knows how it will shake out.

regularfry 10y ago
https://en.wikipedia.org/wiki/Universal_Disk_Format https://en.wikipedia.org/wiki/Universal_Disk_Format ? That looked hopeful last I was juggling with this sort of thing, but that was long enough ago that WinXP support was the killer.
- duskwuff 10y ago
  From what I understand, UDF is primarily targeted at one-time-recordable media. (It's used in DVDs, for example.) It's unclear how well it works for primary storage, but I suspect it's clumsy at best.
  - regularfry 10y ago
    It's only got to beat FAT. That's a pretty low bar.
    - setpatchaddress 10y ago
      One that it doesn't meet. Try to read and understand the UDF spec sometime.
- Sanddancer 10y ago
  Getting UDF to work with rewritable media, like usb sticks, is an unholy pain in the arse. It involves trying to remember an arcane combination of versions and feature flags to get the universal format to actually be universal. I managed it once, but after that, realized that the network was fast enough for most of what I needed and I tend to not have the four gig files that cause problems with FAT filesystems.

busterarm 10y ago

Give me case sensitivity and file dates that go past Feb 6th 2040 please. :D

notacoward 10y ago

I'm notoriously skeptical of anything from the local-filesystem world, and I'm no Apple fanboy, but this looks pretty cool. I look forward to hearing how some of it works, especially what's new about their version of COW and why end-to-end checksumming wasn't mentioned.

ysleepy 10y ago

Uh interesting thought: With Space sharing and multiple logical volumes there will probably be a shift to encrypting users home directories separately. Probably even an unencrypted System Base, which is read-only and protected by rootless anyway, so the system can boot without user interaction. This also finally allows the most user friendly implementation of file encryption: Encrypted Folders, supported by the OS. Sounds cool, Id be willing to help with a rust port once the source is available :) (case sensitive only - naturally)

derefr 10y ago
The point of encrypting the base system image is mostly to make it part of the Secure Boot chain. rootless is a policy control to stop processes from modifying the OS from inside. An encrypted system image stops things with hardware access (but not the unlock key) from modifying the OS from the outside. You can trust any disk whose blocks you can decrypt with key X, to have been only written to by someone with key X.
- majewsky 10y ago
  Why would you need encryption for the Secure Boot chain? Would you not rather sign the base system image for that purpose?
  - derefr 10y ago
    You'd have to sit there and read the entire base system image before booting to verify that signature. But only have to decrypt a block when it comes time to read that block—with the ability to kernel-abort right then during the boot process if the block doesn't decrypt. (Or were you suggesting individually signing every block?)
    - nwf 10y ago
      You don't sign the whole image as a stream, and you don't sign every block. Recursion is your friend! You sign the Merkle tree root, check it once, and then check O(log n) hashes per block access. You can, of course, amortize the checking of the first several of those hashes as a further optimization that ties in easily with your caching layer. There's no such thing as "the block doesn't decrypt" absent MACs/MICs or AEAD schemes -- encryption and decryption are just maps from N bytes to N bytes.

crest 10y ago

I wonder if Apples acquisition of acunu.com is finally paying off. Is this file system based on stratified B-trees?

pducks32 10y ago

I read the link and have a pretty good understanding about how computers and file systems work but can someone "explain like I'm a decently intelligent programmer" what is different about this file system? Thanks you.

cpr 10y ago
They simply started from scratch with an aim to * best support modern hardware technologies, * start from the state of the art in file systems (like ZFS and btrfs, old as they may be), and also * emphasize security.
- astrodust 10y ago
  Sounds a lot like what Apple did with Swift.
josteink 10y ago
It's apple reinventing the zfs and btrfs wheels. Which are already mature, useable and open source. Google them and see what they do well, and you might get some ideas.
- tracewpearson 10y ago
  Both filesystems have copyleft licenses so that may factor into Apples decision.
Eric_WVGG 10y ago
This bit from 2012 by John Siracusa outlines everything “wrong” with the old file system, HFS+, and doubles as a sort of guide to what this new file system fixes. http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/ http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/

bluegreyred 10y ago

I didn't see any mentions of check-summing. Please tell me this brand new filesystem won't be susceptible to bitrot?

franciscop 10y ago

I cannot see anything on my Android mobile S:

Bytes 10y ago

This is a huge feature. I have no idea why Apple didn't include this in their keynote.

methyl 10y ago
Probably too technical for a typical keynote watcher
- kristofferR 10y ago
  Typical World Wide Developer Conference watcher? If so, that's a shame!
  - Razengan 10y ago
    Well, have you seen the comments on Reddit and MacRumors and the like? You'd think anything above "dark mode" and a new MacBook model would be over their heads. Most people do seem to assume that the WWDC is an hour~ long commercial for everything Apple is going to do for the rest of the year.
  - huxley 10y ago
    Opening Keynote is really for the press, it'll probably be mentioned more prominently in the State of the Union keynotes and in its own session.
Jerry2 10y ago
It was featured in Platforms State of the Union... which is the "real" keynote targeted at developers. You will be able to see it here shortly: https://developer.apple.com/videos/play/wwdc2016/102/ https://developer.apple.com/videos/play/wwdc2016/102/

VeejayRampay 10y ago

Website doesn't work at all on my Nexus 5x. Any other company would get lambasted for that kind of stuff.

deleted 10y ago

[deleted]

amelius 10y ago

I hope finally a filesystem that allows me to unplug a USB drive (while it is still mounted) without potentially corrupting it.

userbinator 10y ago
Any filesystem that doesn't cache writes will basically have that property, unless you're talking about unplugging while write operations are happening, in which case all bets are off.
- wmf 10y ago
  The point of journaling/soft updates/COW/etc. is that all bets are not off; they're intended to prevent the fs from being corrupted in those sorts of situations.

ahl 10y ago

The spartan description of APFS certainly sounds like the (partial) feature list for ZFS--the comparisons made in the comments here are on-point. ZFS though took around 5 years to ship and, arguably, another 5-10 to get right. I say this having shipped multiple products based on ZFS, writing code in ZFS, and diagnosing production problems with it. On-disk consistency ("crash protection"), snapshots, encryption, and transactional interfaces ("atomic safe-save") will no doubt be incredibly valuable. I don't think though though that APFS will dramatically improve upon the time it took ZFS to mature from a first product to world-class storage. Some commenters have opined that (despite Apple distributing ZFS for Mac OS X at WWDC nearly a decade ago) that ZFS would never be appropriate for the desktop, phone, or watch. True ZFS was designed for servers and storage servers, but I don't think there's anything that makes it innately untenable in those environments--even its default, but not essential, use of lots of RAM. Who knows... maybe Apple have spent the decade since killing their internal ZFS port taking this new filesystem though the paces. Its level of completeness though would suggest otherwise.

watersb 10y ago
Yes. I was a heavy user of ZFS on Mac OS X back then. Was on the developer mailing list for that implentation, submitting bugs and talking to Apple developers working like mad to ship it. Then Steve Jobs' buddy Larry bought Sun, and licensing of ZFS became basically impossible to sort out on time. So they dropped it. Was that 2007 or 2009?
- ahl 10y ago
  The Sun acquisition was announced in 2009 and closed in 2010. It may have been related, but I have my own suspicions about about what caused Apple to jettison their port.
  - rafaelferreira 10y ago
    Do tell.
  - astrodust 10y ago

lai 10y ago

Oh I forgot, you can't read Apple documentation on a mobile chrome browser.

watersb 10y ago
Last week, Apple modified their formatting for the User Support Forums, so I can actually see readable text on an iOS device. That only took... a very very long time. (Wonder if Radar has a new look...)
nullymcnull 10y ago
Even on Chrome desktop, if you turn on Mobile emulation and switch it to appear to be any Android device w/ phone display size, it fails in the same way - yet switch it to 'iPhone 5' or 'iPhone 6' and the same compact layout works fine. They're pointlessly sniffing and special-casing 'Android' user-agents and botching it completely.

rleigh 10y ago

While I am happy that Apple have at last committed to replacing HFS+, I'm wondering why they didn't use ZFS rather than reinventing the wheel. It's not like it's a particularly easy wheel to reinvent either; the amount of effort which goes into a filesystem like ZFS is non-trivial. Why not build on top of that? I would have greatly appreciated being able to use ZFS with MacOS X, for datasets, snapshots, sending them to remote pools for backup etc. It would have made it directly interoperable with a lot of pre-existing and cross-platform infrastructure. (I couldn't care less if it didn't scale down to the "watch". Filesystems are not a one-size-fits-all affair.) I find it great that I can take a set of disks from e.g. Linux, run "zpool export", pull them out, and then shovel them into a FreeBSD system, run "zpool import" and have the pool and datasets reassembled and automatically mounted. Perfectly transparent interoperability and portability. While Apple like to do their own thing, this is one place I would have definitely appreciated some down to earth pragmatism and re-use of existing battle-tested and widely used technology.

wcummings 10y ago
How does AAPL feel about the CDDL?
- wstrange 10y ago
  Given that there are other companies shipping ZFS without Oracle's blessing, I am curious why Apple could not do the same? (honest question) Is it that Apple is a much bigger target? Patent issues?
  - zaphar 10y ago
    Many of those companies probably started using ZFS before Oracle acquired Sun. After Oracle acquired Sun the risks of using ZFS skyrocketed legally and Apple had the option of just not taking on that risk since they hadn't shipped it yet. Given what we know about Oracle that was probably the right call.
- jdub 10y ago
  They ship dtrace.

CoachRufus87 10y ago

Forgive my ignorance, but what benefits would an average user experience with this new file system?

nisa 10y ago
In theory (the practical side is tricky to get right): - More resilience against data corruption for one due to copy on write and likely checksumming - so in theory power failure should not harm the filesystem and you can roll back to a last known state and only lose data for the last few seconds if at all. - Snapshots, before installing updates you can snapshot the filesystem and roll back instantly. You could also create a readonly snapshot to have consistent backups. This should also allow seamless rollback of OS updates. There are also clones supported e.g. multiple diverging versions of a file tree - think something similar to git branches on a filesystem level. - Native encryption at the filesystem level - Possible better handling of metadata. Faster access to directories with lot's of files, more files on a volume without slowdown. - Sparse files - you can create huge files instantly and fill them later, guess it's important for various types of software e.g. virtual machine images. But as nice as these properties are in theory history showed that that in practice a lot can go wrong.
- CoachRufus87 10y ago
  Thank you for the explainer.

watersb 10y ago

If you create a ZFS filesystem on a non-macOS, you have to be careful about Unicode mapping in order to use it safely on MacOS. I have no idea if Unicode is an issue for Apple File System. Emoji file names, you guys.

cynix 10y ago
> Emoji file names, you guys. But are they 3x bigger?

0x0 10y ago

Hopefully this will be the end of .DS_Store and the crazy unicode normalization issue that causes mixups when rsync-roundtripping a directory structure between ext4 and HFS! :)

nneonneo 10y ago
Unicode support on filesystems is just an ugly mess, and APFS is unlikely to fix this (the brief documentation doesn't mention it at all). Let's recap: - On OS X HFS+, filenames are stored using a "variant" of NFD, where some characters are precomposed "for compatibility with old Mac text encodings" (https://developer.apple.com/library/mac/qa/qa1173/_index.html https://developer.apple.com/library/mac/qa/qa1173/_index.htm...). - On Windows NTFS, filenames are "opaque sequences of WCHARs", and are thus "kind of" UTF-16 with no formally required normalization format. Windows itself tries to use NFC, but applications are free to use the Windows APIs to create a filename with anything they like. Since filenames are just sequences of 16-bit WCHARs, dangling surrogate pairs are allowed (and can break all sorts of code!) - On Linux, filenames are opaque sequences of 8-bit characters. The only requirement is that a filename not contain either a slash or NUL character. No other formal specification exists, although "most" users these days use UTF-8. (However, you can and will find loads of filesystems with invalid UTF-8, usually because filenames are in one of the ISO encodings instead). Multiple programming languages have been bitten by the possibility of invalid Unicode in filenames (see for example: rust (https://github.com/rust-lang/rust/issues/12056 https://github.com/rust-lang/rust/issues/12056), Python (https://www.python.org/dev/peps/pep-0383/) https://www.python.org/dev/peps/pep-0383/)). This mess is pretty much never going to go away, either, because filesystems are extremely durable and long-lasting.
- 0x0 10y ago
  So the main thing I've observed is rsync'ing web files from a linux server to an osx laptop, and back to the linux server, ends up with a bunch of duplicate decomposed utf8 filenames on the linux server. It can be avoided with careful use of rysnc's "--iconv=utf-8-mac,utf-8" etcetera, but it feels super-unnecessary. Fix it already :D More details here http://serverfault.com/a/427200 http://serverfault.com/a/427200 PS: I've been using LC_CTYPE="whatever.ISO8859-1" and an ISO-8859-1 Terminal.app locale forever since I seem to keep dragging a bunch of legacy filenames around (having started on MS-DOS and FreeBSD 2.2) and ISO-8859-1 still seems to be the only locale that lets me "see the bytes" matrix-style instead of a random amount of "?" chars. Curiously, Finder.app seems to keep up very very well despite the odd encoding. Crossing my fingers the new APFS will act more like Linux. PPS: Java is especially hilarious when launched with -Dfile.encoding=utf-8 as it is literally impossible to access some files from there. And don't get me started on .DS_Store :P

comex 10y ago

I had been nervous that if and when Apple finally got around to making a new file system, they would follow the trend and neither open source it nor document the on-disk format. HFS+ itself has always been open by virtue of being included in the open-sourced core xnu kernel, but Apple's CoreStorage volume manager (the basis for FileVault and Fusion Drive) and DMG disk image format have always lived in closed-source kernel extensions and are not officially documented; Microsoft has acted similarly with its NTFS, ReFS, and exFAT file systems and Storage Spaces volume manager. The result, of course, is that nothing can access anything else's file system, at least well. Most of the formats I mentioned have been reverse engineered to some extent, but there are limits, especially when you're scared of corrupting data by misinterpreting the format and therefore limit your implementation to read access. If you want read-write access to NTFS from Linux or macOS, a filesystem which is old enough to drink, you have to rely on either a relatively slow FUSE implementation or some proprietary software. If you dual boot Linux on your Mac and want access to the Mac partition, make sure to turn off disk encryption (and hope you don't run into any evil maids) unless you want to use a read-only FUSE tool, and I hope you don't have a Fusion Drive because I don't think anything supports that at all (could be wrong). To be fair, this situation cannot be blamed entirely on lack of openness. Lack of interest matters too; as I said, HFS+ has an open source implementation, yet Linux's hfsplus driver doesn't support journaling, a feature added in 2002. But if anyone tries to improve the situation, that implementation makes their job a lot easier: for one thing, since filesystem support is in the BSD-based portion of the xnu kernel, it may be possible to port the HFS implementation to other BSDs with relative ease - like the NetBSD rump kernel, which can run in userspace, from which FUSE support wouldn't be that hard... Alternately, if they chose the path of enhancing the native Linux driver, at least they could consult Apple's code to be sure they weren't missing any obscure corners of the format that could cause their driver to corrupt data. Anyway, what we got with Apple File System is this: > An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format when Apple File System is released in 2017. That's pretty good! Open source plus documentation would be ideal, but documentation is much better than nothing[1], and the text of that paragraph doesn't exactly rule out the possibility of the final release being open source too. (I haven't installed the beta yet, but am I right in guessing that APFS is implemented in a kext?) While there's no guarantee, I hope that that documentation will also come with information about the Core Storage format APFS will usually be wrapped in, so that proper interoperability can be achieved. Rather than my fear coming to pass of one of the last vestiges of openness in proprietary OS storage formats disappearing, the situation may actually be improved compared to today. Maybe in a few years I'll be able to install Ubuntu on my Mac and have full access to the OS X partition out of the box. I can hope... [1] whether it's better or worse than open source alone is debatable - source can be harder to understand, but it's also never wrong, unlike documentation...

cm3 10y ago

Does anyone know if Dominic Giampaolo was involved with the design?

gyc 10y ago
He's still at Apple so I would assume he had a major role.
- cm3 10y ago
  I hope so, though I kinda miss the pervasive use of metadata as in BeFS. That was before its time and such a nice concept. Coupled with saved queries, it's unmatched to this day in mainstream, general purpose file system semantics.

DeepYogurt 10y ago

It seems to be a b-tree based file system that isn't zfs or btrfs. Neat.

rleigh 10y ago
Note that ZFS doesn't use b-trees; it uses Merkle hashes.

PakG1 10y ago

This is interesting because just yesterday I saw a discussion here on HN about how sad people were that Apple was doing nothing to improve OSX and was putting all the focus on iOS instead. Heh.

FunctionalMan 10y ago

Finally catching up to NTFS/ReFS! Congratulations, Apple.

jackjeff 10y ago

I don't see any mention of data integrity... Unless you use multi-key encryption. I find that a bit odd.

jackjeff 10y ago

I don't see any mention of data integrity... Unless you use multi-key encryption. I find that a bit odd.

Black-Plaid 10y ago

> Case Sensitivity: Filenames are currently case-sensitive only. Dominic, lol.

Chyzwar 10y ago

Arrogance... Instead using ZFS they build new file system from scratch (APFS) Instead using Vulkan they build new Graphics API from scratch (Metal) Keep this up and they will end like MS.

chucknelson 10y ago
I'd be more interested in understanding why they chose to implement their own file system instead of adopting something like ZFS. No doubt there is reasoning in there besides the "we are an evil company" logic - Apple developers are people too...
- Chyzwar 10y ago
  I just do not understand rationale. File systems is hard stuff, instead taking something that is probably best file system around they decided for long and painful road. Note that they worked before on OpenZFS in Apple ....
taharvey 10y ago
ummmm... metal cames out 2 years before vulkan. Vulkan was inspired by metal.
- Chyzwar 10y ago
  nope, both are inspired by AMD Mantle. Apple not only ignored first major consensus in Games industry it also blocked support on its devices. It also stuck with old OpenGL. This will make gaming on Apple to suck for many years to come.

sigzero 10y ago

I hope this comes about unlike the ZFS fiasco.

kevinSuttle 10y ago

I wonder how APFS compares to IPFS. https://ipfs.io/ https://ipfs.io/