19 ms·
What’s New in Apple Filesystems [pdf]
- sebiw 7y agoHere's the video: https://developer.apple.com/videos/play/wwdc2019/710/ https://developer.apple.com/videos/play/wwdc2019/710/
- pwinnski 7y agoThis is the key. The PDF fails to make sense in parts without the video.
- sebiw 7y agoInitially only found the PDF, then saw the video link at the end.
- DarkStar851 7y agoFor anyone else wondering about WWDC19, all of these PDFs are accompanied by a video. I guess they were separate in-person workshops.
- msla 7y agoAren't PDFs automatically marked anymore?
- sebiw 7y agoTitle got renamed
- kccqzy 7y agoThe read-only system volume really seems like a neat idea. It reminds me of the Transactional Server role in openSUSE where they make system upgrades as atomic as possible: https://news.opensuse.org/2018/05/15/transactional-updates-in-opensuse-leap-15/ https://news.opensuse.org/2018/05/15/transactional-updates-i... That said, I don't understand the use case for firm links. Seems like the problem it's trying to solve can just be solved with simple mount points: mounting read-write /Users and /usr/local. Am I missing something?
- pwinnski 7y agoIn the diagram, there are just two directories, and mount points could easily handle that case. In the video he describes how the scale of things makes using mount points here unwieldy.
- dunham 7y agoA few more details that I've managed to suss out. The Data volume is mounted on /System/Volumes/Data. Currently, there are 21 firm links into it. They are enumerated in /usr/share/firmlinks (included below). /AppleInternal Device/AppleInternal /Applications Device/Applications /Library Device/Library /System/Library/Caches Device/System/Library/Caches /System/Library/Assets Device/System/Library/Assets /System/Library/PreinstalledAssets Device/System/Library/PreinstalledAssets /System/Library/AssetsV2 Device/System/Library/AssetsV2 /System/Library/PreinstalledAssetsV2 Device/System/Library/PreinstalledAssetsV2 /System/Library/CoreServices/CoreTypes.bundle/Contents/Library Device/System/Library/CoreServices/CoreTypes.bundle/Contents/Library /System/Library/Speech Device/System/Library/Speech /Users Users /Volumes Device/Volumes /cores Device/cores /home Device/home /mnt Device/mnt /opt Device/opt /private Device/private /sw Device/sw /usr/local Device/usr/local /usr/libexec/cups Device/usr/libexec/cups /usr/share/snmp Device/usr/share/snmp
- matthewbauer 7y agoIt's interesting that /usr/X11 isn't in that list!
- klodolph 7y agoWell, X11 hasn't been provided by Apple since 10.8, about 7 years ago. So it wouldn't be touched by OS upgrades anyway (it's a separate product).
- matthewbauer 7y ago/sw isn't provided by Apple either, but they provide it so that Fink will work.
- Stubb 7y agoCan you back to an APFS volume yet, or does it need to be HFS+?
- xemoka 7y agoI was really surprised when APFS came out that it didn't support time machine. From the looks of this, it appears they're going to be able to write a new-style time machine on-top of APFS with the additions here. Interesting, over-due IMHO.
- dwaite 7y agoI suspect they are waiting until they can migrate a time machine volume in one go (e.g. migrate the filesystem from HPFS with directory hard links to snapshots and a potential non-backup volume for other files). They have a lot of deep time machine integration in other parts of the system, so I suspect the staging of the projects means it is pushed off to sometime next year.
- chrisfinazzo 7y agoHearing about how snapshots work over the years during APFS sessions made me immediately think of 2007/2008 when they were describing the tech that eventually became Time Machine, but it wasn't a one click operation yet. As a perverse experiment, do we know what happens if you take an APFS SSD and try to format it as a TM volume? Guessing it's still not pretty despite the fact that HDD's under APFS are PNG, but directory hardlinks are gone. Does it work, but just fill up the disk - no deduping?
- Stubb 7y agoTime Machine refuses to back up to APFS volumes at present.
- vardump 7y agoSo containers with a new bi-directional hardlink "Firmlink" (intended to be invisible to applications, link between encrypted containers), with read only volumes protecting system software. ZFS send/receive style volume serialization for replication backup, etc. File system snapshot support included. iOS or should I say iPadOS supports external USB media and SMB. What seems to be missing is block level checksumming and integrity checks, scrubbing, block level duplication to protect important files, etc. No whiff of deduplication either, but that might be too memory intensive for Apple's purposes.
- eclipxe 7y agoiOS on iPhone also supports external USB media and SMB.
- pwinnski 7y agoI was surprised to see this. From the original keynote, it seemed like an iPadOS-only option, but this video made it clear that it applies to both.
- dwaite 7y agoThere's still a lot of effort to keep subsystems and application code-bases the same - iPad OS I believe is mostly a sign that there may continue to be UX deviation (widgets on home screen, drag and drop between applications, split view multitasking, etc)
- scarface74 7y agoEach supported device has always had its own image, so I assume it’s not exactly the same codebase. The only reason I know this is because the last time I had to upgrade iOS using iTunes - going from iOS 4 to 5, I had three separate devices that required separate downloads for my iPhone 4, iPod Touch, and iPad.
- Angostura 7y agoAnd Samba shares, which from my perspective is even more interesting.
- theWheez 7y agoAny idea how this might affect (or help) projects like iSH, or similar? I've been fighting to be productive on my iPad, and currently that takes the form of a remote terminal. I've been craving the ability to run a proper Unix shell locally.
- lykr0n 7y agoI highly doubt that you're going to get a proper shell on the iPad.
- derefr 7y agoI'm assuming you mean a shell into the iPad's own Unix environment. You could build something for iOS that works like Crouton does on ChromeOS just fine. (Just, nobody has done it yet, for some reason. The iPad Pro is plenty powerful enough to run VMs!)
- matthewbauer 7y agoI don't think it's as easy as crouton. There's no chroot or user namespace equivalent for XNU.
- saagarjha 7y agoiSH does this by emulating Linux rather than virtualizing it.
- BlackRing 7y agoBoth Pythonista and OpenTerm on iOS are "good enough" for a lot of what I need, but it would be nice to have a proper shell. Even if it's jailed/locked to /private/var/mobile
- larkost 7y agoNo effect at all. Apple is unlikely to make this possible, and given the sandboxing of Apps on iOS a meaningful version of this for local resources outside of the App are prohibited.
- Wowfunhappy 7y ago> Read-only state of the system volume can be disabled but not persistently, will revert to read-only after a reboot Wait, what? Even with SIP off? Can it be disabled from within the OS? Can I add "command-to-make-system-rw" to a launchd plist that runs at load on my machine?
- nomel 7y agoI think having persistent access is the file system equivalent of logging in and working entirely from root. The default should be a restricted system, with intentional, temporary, jumps into unrestricted access for making whatever infrequent system configuration changes. What’s the use case for constant RW access? What changes at this level so frequently?
- Wowfunhappy 7y ago> I think having persistent access is the file system equivalent of logging in and working entirely from root. Except that those system files are already protected behind root privileges, so they're only read-write if I have root. And, one reason the root privilege system works is because temporarily gaining root when necessary isn't painful. I indeed don't change root files on a daily basis, but when I do, I don't want to have to go through an extended rigamarole every single time. Typing in my password should be enough. I totally respect that Apple wants to keep normal users out of this stuff. I just want a way to remove the safety wheels one time, instead of again and again.
- pwinnski 7y agoI've shot myself in the foot enough to appreciate an ever-lengthening list of hurdles I need to jump to do so again. I would be protesting if there were no way to make the rootfs read-write, but as it is I'm happy with having to perform a command or two when I want to write. I suspect you can do that in a launchd script, but however much more involved it is than typing my password, I'm probably fine with leaving it as-is.
- chrisfinazzo 7y ago
- fnord77 7y agonothing to address the docker mounted-volume slowness issue?
- sigjuice 7y agoIt is very unlikely that docker issues are a priority for Apple.
- microtherion 7y agoHave you tried using delegated mounts? I got the impression these helped performance, though I have not measured details.
- bni 7y agoThats probably very neat. Got rid of .DS_Store files yet?
- snazz 7y agoAnd the .Trashes, which always manage to make my camera SD card full even after “deleting” everything. I’m not sure that there is a good solution to this other than to always use permanent deletion on removable volumes, like I think Windows does.
- LeoPanthera 7y ago> And the .Trashes, which always manage to make my camera SD card full even after “deleting” everything. The trash isn't magic. You delete stuff, it goes into the Trash. The Trash is represented by a ".Trashes" directory on removable media. You can delete it manually or just empty the trash.
- snazz 7y agoMy point is: Joe Average takes some photos with his camera, finds that he’s out of space on his SD card, pops it into his MacBook to transfer over a couple of good photos and delete the rest, and when he plugs it back into the camera it’s still full. Finder doesn’t show anything, and neither does the camera’s image review function. I don’t think that looking to empty the Trash is an intuitive next step, since it seems to me that the Trash wouldn’t be on the SD card.
- LeoPanthera 7y agoJoe Average uses the Photos app, or the Image Capture app, which moves the files and doesn't trash them in the first place. Joe Average probably doesn't even know what a "file" is.
- stephenr 7y ago> Finder doesn’t show anything Finder shows a full "trash can" in the dock.
- sdan 7y agoAnyone know how they made these slides? I'm speculating it's just Keynote with San Fransico Bold or something.
- znep 7y agoHopefully somehow maybe while they are in there they might do something to fix the nasty system lockups I have seen when starting both time machine and carbon copy cloner backups since switching to APFS. I'm assuming it is related to having >5 million files, but haven't figured that out for sure. The symptoms are the system becomes almost totally unresponsive for up to 5 minutes to the point where you can't even drag and drop an existing window sometimes.
- dwaite 7y agoWhat percentage of that do you think is Disk I/O vs lock contention?
- znep 7y agoI wouldn't expect that level of disk IO to cause issues dragging existing windows around without some deeper lock contention or something, it does not behave like just IO bound. Stuff being slower, totally, but not to this level. It was never an issue before the filesystem upgrade.
- sebiw 7y agoWhat physical medium do you use (HDD, SSD)? I had a faulty HDD act up on me like that.
- znep 7y agoSSD on a 2016 macbook pro. The timing coincides exactly with the FS (and OS...) upgrade.
- jhack 7y agoStill nothing on transparent compression? It's one of the main reasons why I keep a ZFS partition on the Macbook.
- lcnmrn 7y agoThere is afsctool, git clone and build from https://github.com/RJVB/afsctool https://github.com/RJVB/afsctool or brew install afsctool (an older version).
- alpb 7y agoMods: Can someone add [pdf] suffix to this please?
- sctb 7y agoAdded. Thank you!
- PaulHoule 7y agoI am left with the feeling that in 2019 the filesystem is the weak part of the operating system. * mmap doesn't really work well on 64 bit systems * mmap will block if it has to page data in from a file * there are ten different backends to handle layering on Docker and that simple fact implies that none of them are good; if you could compose layers arbitrarily the speed and scalability of Docker would be in a different league than it is now, but you can't. So Docker is just as easily something that slows you down as speeds you up. * Filesystem metadata scans are slow, shockingly so on Windows. I know I like developing with S3-style object stores. To some extent this can replace the traditional filesystem, in other ways it can't.
- mappu 7y ago> there are ten different backends to handle layering on Docker and that simple fact implies that none of them are good overlay2 is fast, it's in the mainline kernel, and it's now the default for new Docker installs. I expect all the devicemapper / aufs mess to be history soon.
- Hello71 7y ago"you can now COPY FILES!" applause
- dang 7y agoPlease don't post unsubstantive comments here.
- Hello71 7y agoApple has spent so long oppressing their users and claiming that missing features are because they're looking out for users' best interests that it's ironic that the introduction of basic features that competitors have had for years if not decades is worthy of applause.
- dang 7y agoThere's probably a good point there but to be within the site guidelines you need to express it substantively and without snark. This comment is better than the GP that way. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html