6 ms·

I think you really need to understand that the term micro kernel refers to a very specific thing. Please go read the wikipedia definition because it does a good

I think you really need to understand that the term micro kernel refers to a very specific thing. Please go read the wikipedia definition because it does a good job of summarizing it. A microkernel handles only the basic kernel primitives in the highest privilege level and critically provides an rpc like communication mechanism for authenticating and delegating everything else. eBPF does not do that, it does not enable doing that. It does the inverse. It adds code to the kernel in the exact same way a kernel module adds code to the kernel, but in very specific places to allow extending the kernel at points of policy. It does not run kernel code in userspace. It does not run userspace code in the kernel. It loads code from userspace into the kernel and runs it as kernel code.

rektide 4y ago
No, sorry, I am pretty well versed I think. On the topic of eBPF, you keep talking about eBPF adding code to the kernel. Yes: that is NOT the microkernel model. But you keep not acknowledging that it's not code shipped by the kernel. If we make the kernel programmable, yes, it means running some code in the kernel. But tons and tons of people are using this to make really really exceptional userland tools that take over kernel-like responsibilities, often running faster than or with more/newer/interesting features the kernel never could do. Adding kernel user-programmability lets us move stuff out of the kernel. And that keeps being what we see happening. It does allow other models too! It allows a whole raft of different things to happen. Some of the eBPF uses are to push more responsibility into the kernel, to make more stuff happen in ring0 or to take what would have been userland & make it ring0. It's a very general capability that breaks us out of the low-dimensionality world we've lived in. It makes a more more exploration possible. Some eBPF possibilities will in some ways resemble microkernel-y things. Some will look exactly the opposite. Right now there's no clear plan to moving most driver code into userland (the microkernel model), but I also would not rule it out & I think there's a number of clearly visible shifts (and my tea leaves tell me increasingly ongoingly) already well under way in that direction. An example of this is some of the Fuse + eBPF work, extfuse. File Systems in Userland is a recognizably microkernely idea, I hope we can kind of agree: let's have a driver for file-systems that runs in userland. But typically FUSE has a pretty big performance hit that can make it unviable, just too much cost of syscall'ing to the kernel. Well, what did some intrepid blokes do? They wrote a eBPF powered Fuse system that hyper-optimized the IPC layer, ignoring a lot of the normal communication bottlenecks & inventing their own new eBPF powered control/communication system. Which let them both expand speed, but also grow a raft of new features for their userland driver, some of which are utterly unique and new. It allows custom permissions checks, differing caching strategies, and creating io redirection to underlying FSes. All from the userland. So, userland greatly greatly grew, & assumed many roles the kernel either did (& some it couldn't do), by having a more malleable eBPF interface. (But yes, that still required talking to the kernel/having some new code in the kernel.) https://extfuse.github.io/ https://extfuse.github.io/ It's a bit less micro-kernel-y an example, but certainly it's an amazing story watching networking grow via eBPF. Yes the kernel is doing the work, but it's such a different situation that it's not not kernel code: it's user-code, running in ring0. We can focus on that code. But notably, that code while heavily trafficed, is often much smaller than the kernel code it replaces. The simple data-plane eBPF code replaces of dozens of fairly complex kernel modules making a lot of the decisions, figuring out how to prioritize or open tunnels or do any of a dozen other deeds. And that eBPF code is all built by a new Master Control Process, a ring -1, the userland process that wrote & injected it all. This userland code doesn't have privileged access to other processes (so still a micro-kernel-y win!), but it sure as heck has a very powerful position over the kernel, is calling the shots/making the decisions, & programming the (again much smaller) kernel according to it's userland desires. Is there still a kernel? Yes. But it's clear the power dynamic is greatly different in this case. There's so many more isolation barriers & complex who-controls-who power relationships here that all seem so much more interesting than the monolithic kernel world, so many specific assignments of responsibilities, done with so much more isolation than before. Seems like a win. I'd still be more hesitant to push this as an overtly micro-service-y case, where-as extfuse I think is more obviously in-the-mold, but reducing kernel responsibilities and giving them to userland at such huge scale with such critical success should make most Microkernelites cheer & feel vindicated over. I hope, even if it's not exactly 100% the purist they'd hoped for. We can keep trying to say, but oh! Gotcha! There's stuff happening in the kernel. I think the topic deserves a much broader view than that. We probably are not going to get rid of the kernel (maybe Fuschia's Zircon microkernel, maybe Genode, maybe whomever will some day make a real break-out win though! nothings decided!). Giving processes much faster ways to batch talk to others/the kernel (io_uring) and giving processes the ability to program/modify the kernel & inject some tendrils ther are- I would agree- not inherently micro-kernel-y. But they open up so many possible spaces that before had felt settled. And in quite a number of examples we're seeing core responsibilities move from kernel to userland. The idea that a more IPC centric network-of-processes userland-driver world emerges seems much more possible to me, thanks to io_uring and ebpf.
- softirq 4y ago
  Micro kernel-ish behavior would be features that allow for drivers/modules in userspace, which Linux does provide (and I do agree that Fuse is an example of this). eBPF is not sort of, or kind of, or remotely related to micro kernels and I'm not being pedantic. It really, really isn't related to microkernels because it is doing the exact opposite of what a microkernel does. Pluggable policy in kernelspace is not the same as running something in userspace that's implementing a system function like a filesystem, a networking stack, etc. which would be analogous to FUSE, DPDK, etc. eBPF is no different from runtime loadable kernel modules, which has never made Linux a microkernel. Modules allow for runtime extension, but they are still running in the kernel. The point of eBPF is that kernel modules are dangerous and error prone to write, and eBPF allows non-kernel programmers to write reasonably safe modules that can also be ported to different kernels via CO-RE. They are pluggable policy points or probes. The Fuse developers just figured out the same thing microkernel developers figured out in the late 80s/early 90s. Performance is much better if you move more processing to the kernel.
  - rektide 4y ago
    I don't feel like you read my post at all or are replying to anything I've said. You're repeating the same assertions without moving the argument along or talking details. Rather than recognize some similarity, rather than taking any kind of parallax view, rather than making friends, I feel like you've adopted a combatative and quite limited stance, and I think it's a shame you're not willing to chalk up some wins where a monolithic kernel has loosened up & allowed some new microkernel or microkernel-ish either offloading or sharing of responsibilities. I provided two pretty amazing examples already of how really amazing new userland capabilities were made by having a configurable kernel. You keep ignoring that eBPF may be code running in the kernel, but it also enables not running a dump-truck load of kernel modules that a user would need, while being more flexible, while letting a non-privileged isolated process either have control (ring -1) as the networking Calico/Cilium example shows off, or even more radically by moving the entire task into userland (ExtFUSE) by using eBPF as an io-port with the kernel to let the external process do more of the tasks than it could have before. You seem to be very rooted in a very precise & narrow minded view of what's afoot here. And fixated on any kernel code being disqualifying, even though it keeps being examples of less & less kernel code running. I think I've really given you a ton of great evidence already, & tried to help you break free of such a narrow conception. My examples really show how eBPF has helped move a ton of work from kernel into userland. I don't think it should be so hard to communicate this, but I'll say it yet again: programmable kernels sometimes will be programmed to do less in the kernel. That's what we've seen. (But they can also be used to add more to the kernel too. To repeat again: eBPF is not inherently MK nor inherently monolithic; it depends on what it's used for!) You talk about Fuse's downside, again boosterizing microkernels, but again, it's like you haven't read my post: where I mention how eBPF was used to greatly speed up & to add brand new novel capabilities neither kernel nor FUSE could do before enhance & Extend FUSE. That's what I keep trying to emphasize. If you have a fixed all-encompassing monolithic kernel, the boundaries are fixed. By introducing eBPF programmability, there's far more flexibility to calve off tasks, to pull them out of the kernel. eBPF is sometimes just a communication tool, not a processing tool, to get the data for the task out of the kernel, into userland, and that 110% qualifies as a microkernel-y thing. Again, to your points, I said I'd be less interested in trying to claim the Cilium/Calico model is microkernel-ish, because as you say, the data-plane remains in the kernel (although I think it's not hard to recognize that even though it's a different architectural pattern than MK it's still a revolution in moving huge responsibiliites out of the kernel & into userland, & getting the lions share of MK benefits). But the ExtFUSE model shows that that's not the only thing one can do with eBPF (allow a "policy agent", which still seems like a colossal MK-ish win to me!), it shows how it can also help to move work itself outside the kernel in new novel ways. Rather than focusing on eBPF being something "in the kernel", I think everyone needs to step back & re-ask themselves what eBPF is. It's a malleability system. It's a way to make the kernel programmable. As I've said again and again, that isn't inherently microkernely. But in a huge number of examples, users do exactly that: take things that the kernel would be doing and they make a small eBPF shim to replace complex kernel code with a small port, that sends the task out to userland, where userland does the work. This is quite obviously microkernely.

rektide 4y ago

No, sorry, I am pretty well versed I think. On the topic of eBPF, you keep talking about eBPF adding code to the kernel. Yes: that is NOT the microkernel model. But you keep not acknowledging that it's not code shipped by the kernel. If we make the kernel programmable, yes, it means running some code in the kernel. But tons and tons of people are using this to make really really exceptional userland tools that take over kernel-like responsibilities, often running faster than or with more/newer/interesting features the kernel never could do. Adding kernel user-programmability lets us move stuff out of the kernel. And that keeps being what we see happening. It does allow other models too! It allows a whole raft of different things to happen. Some of the eBPF uses are to push more responsibility into the kernel, to make more stuff happen in ring0 or to take what would have been userland & make it ring0. It's a very general capability that breaks us out of the low-dimensionality world we've lived in. It makes a more more exploration possible. Some eBPF possibilities will in some ways resemble microkernel-y things. Some will look exactly the opposite. Right now there's no clear plan to moving most driver code into userland (the microkernel model), but I also would not rule it out & I think there's a number of clearly visible shifts (and my tea leaves tell me increasingly ongoingly) already well under way in that direction. An example of this is some of the Fuse + eBPF work, extfuse. File Systems in Userland is a recognizably microkernely idea, I hope we can kind of agree: let's have a driver for file-systems that runs in userland. But typically FUSE has a pretty big performance hit that can make it unviable, just too much cost of syscall'ing to the kernel. Well, what did some intrepid blokes do? They wrote a eBPF powered Fuse system that hyper-optimized the IPC layer, ignoring a lot of the normal communication bottlenecks & inventing their own new eBPF powered control/communication system. Which let them both expand speed, but also grow a raft of new features for their userland driver, some of which are utterly unique and new. It allows custom permissions checks, differing caching strategies, and creating io redirection to underlying FSes. All from the userland. So, userland greatly greatly grew, & assumed many roles the kernel either did (& some it couldn't do), by having a more malleable eBPF interface. (But yes, that still required talking to the kernel/having some new code in the kernel.) https://extfuse.github.io/ https://extfuse.github.io/ It's a bit less micro-kernel-y an example, but certainly it's an amazing story watching networking grow via eBPF. Yes the kernel is doing the work, but it's such a different situation that it's not not kernel code: it's user-code, running in ring0. We can focus on that code. But notably, that code while heavily trafficed, is often much smaller than the kernel code it replaces. The simple data-plane eBPF code replaces of dozens of fairly complex kernel modules making a lot of the decisions, figuring out how to prioritize or open tunnels or do any of a dozen other deeds. And that eBPF code is all built by a new Master Control Process, a ring -1, the userland process that wrote & injected it all. This userland code doesn't have privileged access to other processes (so still a micro-kernel-y win!), but it sure as heck has a very powerful position over the kernel, is calling the shots/making the decisions, & programming the (again much smaller) kernel according to it's userland desires. Is there still a kernel? Yes. But it's clear the power dynamic is greatly different in this case. There's so many more isolation barriers & complex who-controls-who power relationships here that all seem so much more interesting than the monolithic kernel world, so many specific assignments of responsibilities, done with so much more isolation than before. Seems like a win. I'd still be more hesitant to push this as an overtly micro-service-y case, where-as extfuse I think is more obviously in-the-mold, but reducing kernel responsibilities and giving them to userland at such huge scale with such critical success should make most Microkernelites cheer & feel vindicated over. I hope, even if it's not exactly 100% the purist they'd hoped for. We can keep trying to say, but oh! Gotcha! There's stuff happening in the kernel. I think the topic deserves a much broader view than that. We probably are not going to get rid of the kernel (maybe Fuschia's Zircon microkernel, maybe Genode, maybe whomever will some day make a real break-out win though! nothings decided!). Giving processes much faster ways to batch talk to others/the kernel (io_uring) and giving processes the ability to program/modify the kernel & inject some tendrils ther are- I would agree- not inherently micro-kernel-y. But they open up so many possible spaces that before had felt settled. And in quite a number of examples we're seeing core responsibilities move from kernel to userland. The idea that a more IPC centric network-of-processes userland-driver world emerges seems much more possible to me, thanks to io_uring and ebpf.

softirq 4y ago
Micro kernel-ish behavior would be features that allow for drivers/modules in userspace, which Linux does provide (and I do agree that Fuse is an example of this). eBPF is not sort of, or kind of, or remotely related to micro kernels and I'm not being pedantic. It really, really isn't related to microkernels because it is doing the exact opposite of what a microkernel does. Pluggable policy in kernelspace is not the same as running something in userspace that's implementing a system function like a filesystem, a networking stack, etc. which would be analogous to FUSE, DPDK, etc. eBPF is no different from runtime loadable kernel modules, which has never made Linux a microkernel. Modules allow for runtime extension, but they are still running in the kernel. The point of eBPF is that kernel modules are dangerous and error prone to write, and eBPF allows non-kernel programmers to write reasonably safe modules that can also be ported to different kernels via CO-RE. They are pluggable policy points or probes. The Fuse developers just figured out the same thing microkernel developers figured out in the late 80s/early 90s. Performance is much better if you move more processing to the kernel.
- rektide 4y ago
  I don't feel like you read my post at all or are replying to anything I've said. You're repeating the same assertions without moving the argument along or talking details. Rather than recognize some similarity, rather than taking any kind of parallax view, rather than making friends, I feel like you've adopted a combatative and quite limited stance, and I think it's a shame you're not willing to chalk up some wins where a monolithic kernel has loosened up & allowed some new microkernel or microkernel-ish either offloading or sharing of responsibilities. I provided two pretty amazing examples already of how really amazing new userland capabilities were made by having a configurable kernel. You keep ignoring that eBPF may be code running in the kernel, but it also enables not running a dump-truck load of kernel modules that a user would need, while being more flexible, while letting a non-privileged isolated process either have control (ring -1) as the networking Calico/Cilium example shows off, or even more radically by moving the entire task into userland (ExtFUSE) by using eBPF as an io-port with the kernel to let the external process do more of the tasks than it could have before. You seem to be very rooted in a very precise & narrow minded view of what's afoot here. And fixated on any kernel code being disqualifying, even though it keeps being examples of less & less kernel code running. I think I've really given you a ton of great evidence already, & tried to help you break free of such a narrow conception. My examples really show how eBPF has helped move a ton of work from kernel into userland. I don't think it should be so hard to communicate this, but I'll say it yet again: programmable kernels sometimes will be programmed to do less in the kernel. That's what we've seen. (But they can also be used to add more to the kernel too. To repeat again: eBPF is not inherently MK nor inherently monolithic; it depends on what it's used for!) You talk about Fuse's downside, again boosterizing microkernels, but again, it's like you haven't read my post: where I mention how eBPF was used to greatly speed up & to add brand new novel capabilities neither kernel nor FUSE could do before enhance & Extend FUSE. That's what I keep trying to emphasize. If you have a fixed all-encompassing monolithic kernel, the boundaries are fixed. By introducing eBPF programmability, there's far more flexibility to calve off tasks, to pull them out of the kernel. eBPF is sometimes just a communication tool, not a processing tool, to get the data for the task out of the kernel, into userland, and that 110% qualifies as a microkernel-y thing. Again, to your points, I said I'd be less interested in trying to claim the Cilium/Calico model is microkernel-ish, because as you say, the data-plane remains in the kernel (although I think it's not hard to recognize that even though it's a different architectural pattern than MK it's still a revolution in moving huge responsibiliites out of the kernel & into userland, & getting the lions share of MK benefits). But the ExtFUSE model shows that that's not the only thing one can do with eBPF (allow a "policy agent", which still seems like a colossal MK-ish win to me!), it shows how it can also help to move work itself outside the kernel in new novel ways. Rather than focusing on eBPF being something "in the kernel", I think everyone needs to step back & re-ask themselves what eBPF is. It's a malleability system. It's a way to make the kernel programmable. As I've said again and again, that isn't inherently microkernely. But in a huge number of examples, users do exactly that: take things that the kernel would be doing and they make a small eBPF shim to replace complex kernel code with a small port, that sends the task out to userland, where userland does the work. This is quite obviously microkernely.