13 ms·
macOS extensions are moving away from the kernel
- nemothekid 5y agoThis article does a good job of explaining how kernel extensions worked under Intel and how they will work under M1, and the blessed replacement for kernel extensions by Apple under M1. What it's missing, and what I assume everyone's questions is: Are the capabilities of Apple's user-level APIs sufficient to replace most legitimate uses of kernel extensions?
- pjmlp 5y agoThe roadmap as presented by Apple at WWDC 2019 was that after an userspace replacement for a kernel subsystem gets presented, there is one year deprecation until it gets removed from the kernel on the following OS release. So there is one year to try out what might still be missing and give respective feedback. The long term roadmap is that macOS will eventually be much more microkernel like.
- cpach 5y agoCool! Very interesting development given how bad reputation microkernels had for a while. (I’m thinking of Linux vs Minix/Hurd.)
- kzrdude 5y agomacOS (Mac OS X lineage) was always based on mach (a microkernel) which made it a hybrid kernel: https://en.wikipedia.org/wiki/XNU https://en.wikipedia.org/wiki/XNU
- zibzab 5y agoFYI, Linux userspace drivers were introduced in 2006-2007 and improved multiple times over the years. Some of my filesystem & networking drivers run in usersapce. The monolithic vs microkernel debate was pretty clueless at times. Core Linux is smaller than dito hurd when configured with similar functionality. OSX mach has always been a hybrid kernel. In a sense Linux is too but you can also configure it as a mostly-monolithic thing, and most seem to prefer that.
- creshal 5y agoSome of the userspace drivers are even shared between OSX and Linux, via FUSE.
- roblabla 5y agoAnother example would be userspace network drivers that directly talk to the network card. Those tend to be much more performant than using the kernel driver due to avoiding the need to context switch from userspace to kernelspace.
- zibzab 5y agoIs that really true in pratice (excluding unikernels, of course)?
- MayeulC 5y agoAren't we making userspace/kernel space communication a lot more efficient with io-uring, though? Likewise, eBPF allows userspace programs to perform some work inside kernel space to reduce syscalls. I wonder if io-uring won't be a starting point for a "microkernelization" of Linux. It seems really effective as a message-passing interface.
- pjmlp 5y agoLook at Android' Project Treble, that is what Google has done, just with Android IPC instead of io-uring.
- pjmlp 5y agoSince Android 8, Linux is a kind of pseudo-microkernel, as all drivers are required to be Treble based, and traditional Linux drivers are considered legacy on the platform. https://source.android.com/devices/architecture/hal-types https://source.android.com/devices/architecture/hal-types The Switch (yes the games console) uses a microkernel. QNX an OS for human critical scenarios is a microkernel. Finally, when arguing Linux vs Minix/Hurd, think of the irony of running Linux on top of a cloud hypervisor, with an endless load of Kubernetes or Docker loads.
- SlowRobotAhead 5y ago> QNX an OS for human critical scenarios is a microkernel. Hmm, Never thought about a stated OS focus like that. Off to look it up. Good mention
- pjmlp 5y agoSearch for high integrity computing, that is the industry term.
- tzs 5y agoAt the time of the Linux vs. Minix debate, on pretty much every performance measure that Linus said monolithic kernels were better at the microkernel QNX was totally wiping the floor with Linux.
- AshamedCaptain 5y agoI am quite sure that we have been other this several times in HN. The fact that they're no longer going to allow loading of 3rd party kernel modules (a pure political decision) does not a make a microkernel (which is an architectural style). The kernel is still going to be big and monolithic (it contains the entire BSD kernel in the same address space!), and they're still going to be loading Apple-authorized kernel modules (500MiB of them, according to the article) in the same address space.
- pjmlp 5y agoAs you wish, I am no longer going to debate the pedantic meaning of pure microkernel ideology. I rather be happy for the improvements in the long term roadmap and overall outcome for the eco-system.
- AshamedCaptain 5y ago"Pedantic"? If you just consider disallowing 3rd party modules to be a "microkernel" all while the kernel itself still loads most if not all (approved) drivers such as PCI bus, storage devices or filesystems in the same address space, then _all_ current major operating systems are "microkernels".
- pjmlp 5y agoFrom all pratical purposes for a third party driver writer persona, the only path forward are user space drivers. What you call a kernel like that?
- webmobdev 5y ago> Are the capabilities of Apple's user-level APIs sufficient to replace most legitimate uses of kernel extensions? From what I remember reading from some macOS Application Firewall makers, no. (And that's ofcourse, intentional.) While it is true that poorly coded Kernel extensions can make an OS unstable, this is just another example of Apple taking away more control from its users and further crippling macOS to make it more like ios.
- least 5y agoAfter feedback from aforementioned firewall developers, changes were made to network extensions that allowed them to do what they wanted with their applications (for the most part, at least). I still think that it’s a worse in-practice system than kexts but they are responding to feedback at least. You can still load kernel extensions in Big Sur by disabling SIP, though.
- no_time 5y agoI wonder how far away are we from governments mandating SIP and equivalents to be turned on to connect to the internet.
- IfOnlyYouKnew 5y agoWell, I'm glad to see even the cynics can't find an evil motive here, and has to resort to just assuming stupidity. That, however, isn't easy to do, considering Apple has a rather impressive track record of creating products people enjoy and buy. And to jump two replies ahead: them trying to make it "easy" and "save" doesn't imply their users are idiots. Quite a few developers and scientists use Macs, and the lawyers aren't exactly dumb either. Spending hours trying to somehow get both sound and bluetooth to work at the same time, a favourite pastime on Linux, means lifetime wasted for something that shouldn't need doing. And considering nobody reads the source, there is no reason to believe some "expert" has some ability to avoid installing that one extension that soon starts encrypting their files. At least they tend to have better backups...
- Hackbraten 5y ago
- AshamedCaptain 5y agoThe entire "ring 1" thing is highly likely false. It's been decades since I last saw anyone using the x86 ring levels and I seriously doubt XNU is using them. In any case, extensions run in the same address space as the rest of the kernel making any such "rings" pointless.
- als0 5y agoRings 1 and 2 don't even exist on AMD64/x86-64.
- zvrba 5y agoYes they do, but segmentation is practically non-existent and page tables distinguish only between rings 0 and rings 1-3. So no, they aren't very useful, but they do exist and, AFAIK, they're still checked in "far" control transfers.
- tzs 5y agoIt's too bad they have been made useless. I'm not fully sure about this, but I think that if you could have two rings for user code so that applications that want to run untrusted code in a sandbox could run the main application in ring 2 and the sandboxed code in ring 3, you could make the application safe from Spectre attacks from the sandboxed code without giving up performance.
- bonzini 5y agoThe point of Spectre attacks is that they use side channel to infer the target's data from communication between the target and the attacking userspace process. This is possible even over the network, so ring 2 wouldn't help much.
- zinekeller 5y agoI mean, recent developments in the Windows/NT architecture uses virtualization to create a three-level "ring" system, so while it's not strictly architectural "rings", it's effectively a tri-ring system. But then, I haven't delved into how macOS in M1 precisely works.
- deregulateMed 5y ago"blessed" replacement Why did you use the word Blessed? I'm curious what made your brain choose that word instead of something neutral. Reminds me of "Walled garden" marketing speak instead of "Walled Prison". Edit- is there some sensitive nerve I hit?
- stefanfisk 5y agoI can't speak for the author, but personally I'm enough of an atheist to not read "blessed" as more than "approved by the powers that be".
- andyzweb 5y agoI thought it might have been some kind of reference to the bless(8) command.
- cyberge99 5y agoIn context this is how it’s typically used.
- zepto 5y ago> "Walled Prison" I'm curious what made your brain choose that word instead of something neutral.
- andrekandre 5y agowhat would be a good term?
- zepto 5y agoWhy do you ask?
- andrekandre 5y agoi was just curious people use metaphors with a lot of conotation (walled prison) so if there is a more neutral term, i was curious how to state it in either non-positive or non-negative way
- smoldesu 5y ago> Are the capabilities of Apple's user-level APIs sufficient to replace most legitimate uses of kernel extensions? I would have trusted literally anyone else to do a good job at this. Apple's in the business of brokering power though, I see no reason why they'd be motivated to make those APIs as rich as kexts.
- the_other 5y agoI've love to hear what this means for realtime multi-channel audio applications. Would someone close to this give us a summary, or recommend some resources for the only-barely-knowledgeable?
- tomduncalf 5y agoI can’t comment on the internals of drivers etc as it’s not a domain I know much about, but I dabble with music production and can tell you that all my audio gear which requires drivers has been updated for Big Sur, which AFAIK is the new model. I’ve not noticed any change in performance at all, so I assume Apple have ensured the API covers the ground required for real time audio - it is one of their specialities as an OS/platform after all. This does mean that some older gear which requires bespoke drivers and is no longer maintained will never work with new MacOS - one example that comes to mind is Allen and Heath’s Xone DB2 DJ mixer. Quite a lot of newer gear is USB class compliant, which means it doesn’t need specific drivers and so should be future proof.
- ace2358 5y agoI’m running a PreSonus StudioLive 16.4.2 (1st gen) and am stuck on 10.13.6 because I’m pretty sure they didn’t release drivers for 10.14+. It’s a FireWire 400 device. I absolutely love it but because of it, I’m probably never going to upgrade my studio mac. I do wonder if it’s a class a compliant device and works without drives (I think I read somewhere that it might work) but I legitimately use the software that comes with it for controlling the mixer and saving patches and things so I’m not too enthused by it. The new Bitwig 4 even dropped support for 10.13 which was a bit unexpected as it supports Windows 7. Old gear getting dropped because of driver issues is really annoying, but I have come to terms with it. All music produces and studios deal with it. Some mixing rooms are on 20 year old hardware and that’s just how it is.
- tomduncalf 5y ago> Old gear getting dropped because of driver issues is really annoying, but I have come to terms with it. Yeah, certainly on macOS. Not much you can do about it sadly!
- deleted 5y ago[deleted]
- tagersenim 5y agoMaybe not the point here, but what a beautiful way to style the website, with a drop-caps (the big initial letter, I had to look it up). It strikes a middle ground between the old-timey drop-caps and the 'new' web.
- jannes 5y agoI read it on mobile and had no idea what you were talking about. The style seems to be absent on smaller viewport sizes.
- solarkraft 5y agosystemextensionsctl list yields 0 extensions on my system. Are there any users of these extensions at this point? It does seem a bit weird that installing these extensions supposedly doesn't require any user consent, even though stuff like screen recording or folder access does.
- danieldk 5y agoLittle Snitch: % systemextensionsctl list 1 extension(s) --- com.apple.system_extension.network_extension enabled active teamID bundleID (version) name [state] * * MLZF7K7B5R at.obdev.littlesnitch.networkextension (5.2.2/6209) Little Snitch Network Extension [activated enabled]
- smolyeet 5y agomore common in enterprise environment, but vmware fusion, global protect, and lulu/little snitch come to mind from my experience. Big Sur still technically supports Kernel Extensions, and companies will wait until the last minute before giving in.
- jmull 5y agoI checked my system... I have only a Cisco AnyConnect extension. I have VMware fusion running (latest version), so maybe they've finished the transition?
- neilalexander 5y agoVMware Fusion uses the macOS Hypervisor.framework now, so it no longer needs extensions.
- amatix 5y agoTripMode (https://tripmode.ch/ https://tripmode.ch/) for filtering application Internet access when you're running on mobile data uses a kernel extension.
- 5y ago
- GeekyBear 5y agoThe initial announcement from WWDC 2019. https://developer.apple.com/videos/play/wwdc2019/702/ https://developer.apple.com/videos/play/wwdc2019/702/
- cowsandmilk 5y agoIs there any movement on providing the apis needed by osxfuse? Or just giving us fuse apis natively?
- smasher164 5y agoThere's the File Provider API [1], but I don't think it's a proper replacement for osxfuse yet. [1]: https://developer.apple.com/documentation/fileprovider https://developer.apple.com/documentation/fileprovider
- mpweiher 5y agoI've filed a Radar^H^H^H^HFeedback...
- submeta 5y agoI hope this won't effect tools like "Karabiner Elements"[1]? It's an essential tool I use to re-program the whole keyboard. [1] https://karabiner-elements.pqrs.org/ https://karabiner-elements.pqrs.org/
- jscheel 5y agoYeah, definitely going to be problematic if that happens.
- Turing_Machine 5y agoFor sure. That would mean I couldn't use a decent keyboard, and that's gonna be a show-stopper. I don't know what happened there... Apple used to make wonderful keyboards, but it seems like every model they make now just plain sucks.
- tgv 5y agoPerhaps you can contact the macOS team that develops this and ask them to provide some kind of hook for keyboard drivers. From experience, I can tell that the Logic team listens to its users, but it's a smaller, and highly technical user group, of course. I wouldn't know if the kernel/extensions team reacts in the same way.
- zepto 5y ago> ask them to provide some kind of hook for keyboard drivers They already have.
- submeta 5y agoJust got an answer from the dev of Karabiner Elements[1]. It won‘t be affected. [1] https://github.com/pqrs-org/Karabiner-Elements/issues/2760 https://github.com/pqrs-org/Karabiner-Elements/issues/2760
- rrdharan 5y agoKarabiner Elements switched to DriverKit already. I think it will survive the next transition but it’s surprisingly hard to find confirmation. [EDIT: see sibling comment - developer confirmed the above.]
- qalmakka 5y agoTo me, this sounds kinda like the following: - XNU was born from a fork of the Mach microkernel - When the microkernel hype simmered down due to them being way too slow for '90s machines, NeXT bolted on lots of chunks taken from BSD to speed up XNU - After 20+ years, computers are fast enough to run true microkernels, so Apple is bringing them back again So it's finally time for microkernels to shine again?
- pjmlp 5y agoOutside UNIX land they were always kind of shining, even if the light wasn't like footbal stadium. QNX, Symbian, Nintendo Switch, type 1 hypervisors,....
- pram 5y agoBSD was actually grafted into Mach early on, it wasn't something NeXT did.
- qalmakka 5y agoWasn't Mach a re-implementation of the original 4.2BSD completely based on IPC in place of syscalls? I distinctly recall some documents saying that XNU achieved much better performances than plain OSFMK thanks to NeXT reworking the BSD parts as components of the kernel instead of being binaries in user space, like in the case of plain Mach.
- pram 5y agoOSFMK is basically Mach 3 onward. That is the actual “microkernel” version. NeXT started with the versions before that, which still had memory management and a litany of other things living in the kernel and not user space. It was basically a gradual evolution of migrating things out. So to your point: all NeXT was essentially doing was turning Mach 3 back into how Mach 2 worked lol Thus XNU is a novel implementation of Mach 2: an up to date OSFMK kernel and all its IPC, plus all the BSD parts that used to live in it
- 5y ago
- sharikous 5y agoYou can still find the official documents about how to make changes in the MacOS X kernel itself (dated 2004 or so...). It's really sad that we have come to this. The platform became more and more closed off in the name of security and no serious law or regulation is preventing that. And if at least it was bug free it would be worth it, maybe. But my kernel extensions are all for circumventing bugs or stuff that does not work properly, e.g. Karabiner for remapping the ctrl character of my keyboard to command, necessarily since I work with a lot of OSes at a time.
- jen20 5y agoYou don’t need to use Karabiner for that: keyboard preferences has a modifier keys setting (per keyboard) that achieves this exact remapping with no kernel extensions.
- wlesieutre 5y agoAnd even if it were required, Karabiner-Elements is already running as a userspace DriverKit device instead of a kernel extension: > Changed the virtual keyboard and mouse implementation to DriverKit from deprecated kernel extension. https://karabiner-elements.pqrs.org/docs/releasenotes/#karabiner-elements-1300 https://karabiner-elements.pqrs.org/docs/releasenotes/#karab...
- spicybright 5y agoSurprisingly not. You can't re-bind the FN key's function using normal settings. On a 2019 mac pro, catalina, btw, looking at Settings > Keyboard > Modifier Keys...
- jen20 5y agoSame Mac, Big Sur, the Fn key binding is there. This article [1] suggests that it was the case at least in Catalina given the screenshots - but I don't have it easily available to test. [1]: https://howchoo.com/mac/mac-remap-fn-to-ctrl https://howchoo.com/mac/mac-remap-fn-to-ctrl
- 5y ago
- ossman 5y agoThey already can: extension authors can specify that their extension only operates on specific URL's. The problem is that most extensions are designed to work on all web sites, so you have to choose between security and convenience. https://yvanced.com/ https://yvanced.com/ Most users pick the latter and trust the former.
- oauea 5y agoVery cool, more control lost.
- spicybright 5y agoI think the reason of security is a bit fishy too. I've never heard of wide-spread exploits from kernel extensions before. Seems they just want to lock it down to squash developers making kernel level features for mac... I'm sure someone will find a back door to allow extensions somehow, but we can't keep fighting forever :(
- 1123581321 5y agoMy understanding is that narrowly targeted exploits are more the concern.
- beermonster 5y agoDoes this limit the effectiveness of apps like LuLu and Little Snitch in any real way?
- EricE 5y agoNope - latest version of Little Snitch has already moved to user space: https://obdev.at/products/littlesnitch-big-sur/index.html https://obdev.at/products/littlesnitch-big-sur/index.html
- beermonster 5y agoI’ll have a read. Was wondering if any capabilities are lost besides performance