5 ms·

Not to mention Minix running on ~every recent Intel system: https://en.wikipedia.org/wiki/Intel_Management_Engine https://en.wikipedia.org/wiki/Intel_Management

by wzdd 2y ago

Not to mention Minix running on ~every recent Intel system: https://en.wikipedia.org/wiki/Intel_Management_Engine https://en.wikipedia.org/wiki/Intel_Management_Engine Microkernels still have lower performance than their monolithic counterparts. It makes sense really: the major difference between the two is protection barriers between components. All else being equal, a system without those barriers is faster than one with them unless crossing those barriers has zero cost -- which it only really does in the case of proof-carrying code and similar software solutions, where the "barrier" is more of a guarantee that the subsystem behaves as it should. A lot of microkernel research in the past focused on the "all else being equal" aspect, attempting to come up with innovative ways to reduce protection domain crossings in microkernels. For example, mapping graphics card memory or network device buffers directly into the address space of the client means that the hardware can deliver data directly to the client. However, monolithic kernels can also benefit from these sorts of innovations! As to why consumer OSes are all monoliths (yes, this includes Darwin and NT -- "hybrid" is a marketing term as all the important stuff runs in a single kernel address space; see https://en.wikipedia.org/wiki/Hybrid_kernel https://en.wikipedia.org/wiki/Hybrid_kernel): in consumer OSes, the 5% (or lower) performance hit of a microkernel is a big deal. In other spaces, such as embedded, the benefits of microkernels (reliability, security guarantees) are sometimes more important than raw performance.

area51org 2y ago
> Microkernels still have lower performance than their monolithic counterparts This is what I've understood, that the idea sounds good but in practice isn't as good as it seems like it would be.
- pclmulqdq 2y ago
  You necessarily cross from userland (end program) to OS to userland (driver) for most tasks that monokernels just do with the first userspace->OS transition.
  - Findecanor 2y ago
    There have been architectures specifically designed to support some form of IPC or other directly between processes without going through the kernel, and thus without having more overhead. But those are a bit esoteric. The Mill is one, that has been in development for a while, but its company hasn't published much lately and its current status is a bit unclear. It would have had something very similar to Spring/Solaris "Doors" in hardware, where the caller could have passed temporary permissions to memory ranges. A "kernel" would have been a process like any other, just with more permissions.
deleted 2y ago
[deleted]
microtonal 2y ago
Not to mention Minix running on ~every recent Intel system And L4 on every Apple device with a secure enclave: https://support.apple.com/nl-nl/guide/security/sec59b0b31ff/web#:~:text=The%20Secure%20Enclave%20Processor%20runs,against%20clock%20and%20power%20attacks https://support.apple.com/nl-nl/guide/security/sec59b0b31ff/....
AnimalMuppet 2y ago
>Not to mention Minix running on ~every recent Intel system: https://en.wikipedia.org/wiki/Intel_Management_Engine https://en.wikipedia.org/wiki/Intel_Management_Engine But not as the actual system OS. Minix is doing a very little, running on a little side processor that also does very little. It's not running on the main CPU, nor having to interact with the insane number of possible peripherals that real CPUs and OSes have to interact with, nor with real system loads, nor with many of the other constraints that real OSes have to deal with. Yes, it's there. Yes, it's there in really massive numbers. But, compared to what the OSes are doing on the main CPU, Minix is doing a toy job. So I'm not sure that it counts as evidence for much of anything.
surajrmal 2y ago
It's simply not true that microkernels have lower performance. At least it can strongly depend on what you measure. There are tradeoffs, but high-level workflows can accomplish comparable performance. The problem is that sometimes requires rearchitecting software against alternative interfaces. It's becoming increasingly more common for very high performance applications to all directly to hardware and bypass the kernel altogether. Async io such as io_uring also further reduces disadvantages of a microkernel, especially on systems with many cores.
ndriscoll 2y ago
Shouldn't the microkernel architecture be capable of having fewer barriers in the fast path? e.g. you could have hardware like a NIC or GPU be capable of partitioning address spaces to have per-program command/response queues. Then the kernel asks on an administrative queue for a program's IO queue to be created, configures the memory controller (assuming an IOMMU) for the program to have those queues in its address space, and context switches to the program. While the program runs, there are zero barriers and it can simply talk to the hardware directly by writing into pcie space. Hardware knows what queues commands are coming from and can enforce whatever resource partitioning within firmware. My understanding is that (perhaps except for virtualization/address space partitioning on the hardware device to be able to use it with many programs at once) this is approximately how/why user-mode networking is used for high performance today?
- immibis 2y ago
  Monolithic kernels also have these fast paths. It's how GPU drivers work under Linux - your process submits command buffers directly to the GPU.
- foota 2y ago
  That all requires a lot of hardware support that probably didn't exist at the time, and is probably still esoteric today.
- nostrademons 2y ago
  This is the basic idea behind the exokernel concept: have the kernel's only job be to enforce protection boundaries between processes, and then expose an interface to userspace that is identical to what the hardware exposes (eg. directly write to buffers and queues): https://en.wikipedia.org/wiki/Exokernel https://en.wikipedia.org/wiki/Exokernel No production exokernels were built, but the concept lives on as virtualization. Most hypervisors (eg. Xen) work on a substantially similar principal: the hypervisor is responsible for securely multiplexing the hardware between guest OSes, and then the guest OS talks to an interface that basically looks like the hardware. A step beyond this is to get rid of the guest OS and have application code talk directly to the hardware interface. This is a unikernel (or called a "libOS" in the original exokernel papers), and as you predict, unikernels running on a hypervisor are often faster than traditional monolithic kernel OSes. The main thing blocking their adoption is that they require significant rewrites to application-level code: most applications (and the libraries they depend on) generally assume POSIX syscalls are available and would need to be largely rewritten to work in terms of direct hardware access. You can rewrite POSIX as a library in userspace (this was the approach taken by the original exokernel research), but most of the performance gains come from bypassing inappropriate abstractions, so if you use the same abstractions you get basically the same performance. https://en.wikipedia.org/wiki/Unikernel https://en.wikipedia.org/wiki/Unikernel
nullindividual 2y ago
Whatever you want to call it, NT is neither a pure microkernel nor a pure monolithic kernel. Linus' opinion of the term 'hybrid' being a marketing term is great and all, but it's just an opinion. You could call NT a 'modified microkernel implementing monolithic kernel concepts', if you wish. The word 'hybrid' is much shorter, though.
0cf8612b2e1e 2y ago
in consumer OSes, the 5% (or lower) performance hit of a microkernel is a big deal This does not feel like it should have been true for a long time. Computers are just stupidly fast now. In the 90s, sure, performance was way more important to prioritize, but I would love to see the needle swing the other way to getting a more secure and robust OS. My system already squanders so many resources on Electron JavaScript nonsense. Why not let the OS take up a few more cycles to give me hard security guarantees.
- PaulHoule 2y ago
  Somebody tell that to my iPad which goes out to lunch for 30 seconds when I am browsing the web and always seems to be listening to the music in its head. Or the XBOX that takes 30 seconds to sorta kind boot but really 3 minutes for all the menus to load.
- mike_hearn 2y ago
  You can do that and people already do. The most microkernelly consumer OS is macOS/iOS and there you see a lot of code has been moved out of the kernel with time. Example: whilst APFS is in-kernel by default that isn't much security risk. More vulnerable components like FAT drivers have moved into user space. On iOS the networking stack also runs in user space for apps that use Apple's APIs! https://devstreaming-cdn.apple.com/videos/wwdc/2017/707h2gkb95cx1l/707/707_advances_in_networking_part_1.pdf https://devstreaming-cdn.apple.com/videos/wwdc/2017/707h2gkb... So the whole TCP/IP stack is just a regular userspace library on that platform. Only the wifi driver runs in-kernel. These platforms are made up of literally hundreds of Mach servers. Relatively little code runs in the kernel when you examine the structure of the whole OS. The kernel is really only running some hardware drivers where moving them to userspace is hard, and some basics like memory management, process scheduling, IPC and code signature verification. The kernel just isn't that big. It's important to realize though that moving even more stuff to processes does not automatically improve security or availability. Imagine there's an RCE vuln in your root filesystem code. Being in a userspace server doesn't help you: 1. If you can control the FS you can make a root owned file appear anywhere and then just run it, or alter system configuration in any other way. 2. If you can crash the FS the core kernel can't easily recover without special cases because where would it restart the FS process from? If you say "initramfs" then you're back to having an FS in the kernel. In practice there are lots of cases like that. Sometimes it's worth it - modern OS' can generally recover from a display server crash and run graphics mostly in user space. Sometimes it's not. Finally, bear in mind that Linux is very insecure due to its huge feature set and reliance on C, but kernel exploits are often not the biggest security problem on other platforms. The most serious bugs are all in userspace already on most platforms.
IshKebab 2y ago
There's no way a 5% performance difference is the reason. These OSes were designed in an era where you could get a 5% boost just by waiting a month. I think the real reason is they were also designed at a time when security was not as advanced as it is now, especially for Linux, and microkernels are more secure but also harder to write.

by wzdd 2y ago

area51org 2y ago

> Microkernels still have lower performance than their monolithic counterparts This is what I've understood, that the idea sounds good but in practice isn't as good as it seems like it would be.

pclmulqdq 2y ago
You necessarily cross from userland (end program) to OS to userland (driver) for most tasks that monokernels just do with the first userspace->OS transition.
- Findecanor 2y ago
  There have been architectures specifically designed to support some form of IPC or other directly between processes without going through the kernel, and thus without having more overhead. But those are a bit esoteric. The Mill is one, that has been in development for a while, but its company hasn't published much lately and its current status is a bit unclear. It would have had something very similar to Spring/Solaris "Doors" in hardware, where the caller could have passed temporary permissions to memory ranges. A "kernel" would have been a process like any other, just with more permissions.

deleted 2y ago

[deleted]

microtonal 2y ago

Not to mention Minix running on ~every recent Intel system And L4 on every Apple device with a secure enclave: https://support.apple.com/nl-nl/guide/security/sec59b0b31ff/web#:~:text=The%20Secure%20Enclave%20Processor%20runs,against%20clock%20and%20power%20attacks https://support.apple.com/nl-nl/guide/security/sec59b0b31ff/....

AnimalMuppet 2y ago

>Not to mention Minix running on ~every recent Intel system: https://en.wikipedia.org/wiki/Intel_Management_Engine https://en.wikipedia.org/wiki/Intel_Management_Engine But not as the actual system OS. Minix is doing a very little, running on a little side processor that also does very little. It's not running on the main CPU, nor having to interact with the insane number of possible peripherals that real CPUs and OSes have to interact with, nor with real system loads, nor with many of the other constraints that real OSes have to deal with. Yes, it's there. Yes, it's there in really massive numbers. But, compared to what the OSes are doing on the main CPU, Minix is doing a toy job. So I'm not sure that it counts as evidence for much of anything.

surajrmal 2y ago

It's simply not true that microkernels have lower performance. At least it can strongly depend on what you measure. There are tradeoffs, but high-level workflows can accomplish comparable performance. The problem is that sometimes requires rearchitecting software against alternative interfaces. It's becoming increasingly more common for very high performance applications to all directly to hardware and bypass the kernel altogether. Async io such as io_uring also further reduces disadvantages of a microkernel, especially on systems with many cores.

ndriscoll 2y ago

Shouldn't the microkernel architecture be capable of having fewer barriers in the fast path? e.g. you could have hardware like a NIC or GPU be capable of partitioning address spaces to have per-program command/response queues. Then the kernel asks on an administrative queue for a program's IO queue to be created, configures the memory controller (assuming an IOMMU) for the program to have those queues in its address space, and context switches to the program. While the program runs, there are zero barriers and it can simply talk to the hardware directly by writing into pcie space. Hardware knows what queues commands are coming from and can enforce whatever resource partitioning within firmware. My understanding is that (perhaps except for virtualization/address space partitioning on the hardware device to be able to use it with many programs at once) this is approximately how/why user-mode networking is used for high performance today?

immibis 2y ago
Monolithic kernels also have these fast paths. It's how GPU drivers work under Linux - your process submits command buffers directly to the GPU.
foota 2y ago
That all requires a lot of hardware support that probably didn't exist at the time, and is probably still esoteric today.
nostrademons 2y ago
This is the basic idea behind the exokernel concept: have the kernel's only job be to enforce protection boundaries between processes, and then expose an interface to userspace that is identical to what the hardware exposes (eg. directly write to buffers and queues): https://en.wikipedia.org/wiki/Exokernel https://en.wikipedia.org/wiki/Exokernel No production exokernels were built, but the concept lives on as virtualization. Most hypervisors (eg. Xen) work on a substantially similar principal: the hypervisor is responsible for securely multiplexing the hardware between guest OSes, and then the guest OS talks to an interface that basically looks like the hardware. A step beyond this is to get rid of the guest OS and have application code talk directly to the hardware interface. This is a unikernel (or called a "libOS" in the original exokernel papers), and as you predict, unikernels running on a hypervisor are often faster than traditional monolithic kernel OSes. The main thing blocking their adoption is that they require significant rewrites to application-level code: most applications (and the libraries they depend on) generally assume POSIX syscalls are available and would need to be largely rewritten to work in terms of direct hardware access. You can rewrite POSIX as a library in userspace (this was the approach taken by the original exokernel research), but most of the performance gains come from bypassing inappropriate abstractions, so if you use the same abstractions you get basically the same performance. https://en.wikipedia.org/wiki/Unikernel https://en.wikipedia.org/wiki/Unikernel

nullindividual 2y ago

Whatever you want to call it, NT is neither a pure microkernel nor a pure monolithic kernel. Linus' opinion of the term 'hybrid' being a marketing term is great and all, but it's just an opinion. You could call NT a 'modified microkernel implementing monolithic kernel concepts', if you wish. The word 'hybrid' is much shorter, though.

0cf8612b2e1e 2y ago

in consumer OSes, the 5% (or lower) performance hit of a microkernel is a big deal This does not feel like it should have been true for a long time. Computers are just stupidly fast now. In the 90s, sure, performance was way more important to prioritize, but I would love to see the needle swing the other way to getting a more secure and robust OS. My system already squanders so many resources on Electron JavaScript nonsense. Why not let the OS take up a few more cycles to give me hard security guarantees.

PaulHoule 2y ago
Somebody tell that to my iPad which goes out to lunch for 30 seconds when I am browsing the web and always seems to be listening to the music in its head. Or the XBOX that takes 30 seconds to sorta kind boot but really 3 minutes for all the menus to load.
mike_hearn 2y ago
You can do that and people already do. The most microkernelly consumer OS is macOS/iOS and there you see a lot of code has been moved out of the kernel with time. Example: whilst APFS is in-kernel by default that isn't much security risk. More vulnerable components like FAT drivers have moved into user space. On iOS the networking stack also runs in user space for apps that use Apple's APIs! https://devstreaming-cdn.apple.com/videos/wwdc/2017/707h2gkb95cx1l/707/707_advances_in_networking_part_1.pdf https://devstreaming-cdn.apple.com/videos/wwdc/2017/707h2gkb... So the whole TCP/IP stack is just a regular userspace library on that platform. Only the wifi driver runs in-kernel. These platforms are made up of literally hundreds of Mach servers. Relatively little code runs in the kernel when you examine the structure of the whole OS. The kernel is really only running some hardware drivers where moving them to userspace is hard, and some basics like memory management, process scheduling, IPC and code signature verification. The kernel just isn't that big. It's important to realize though that moving even more stuff to processes does not automatically improve security or availability. Imagine there's an RCE vuln in your root filesystem code. Being in a userspace server doesn't help you: 1. If you can control the FS you can make a root owned file appear anywhere and then just run it, or alter system configuration in any other way. 2. If you can crash the FS the core kernel can't easily recover without special cases because where would it restart the FS process from? If you say "initramfs" then you're back to having an FS in the kernel. In practice there are lots of cases like that. Sometimes it's worth it - modern OS' can generally recover from a display server crash and run graphics mostly in user space. Sometimes it's not. Finally, bear in mind that Linux is very insecure due to its huge feature set and reliance on C, but kernel exploits are often not the biggest security problem on other platforms. The most serious bugs are all in userspace already on most platforms.

IshKebab 2y ago

There's no way a 5% performance difference is the reason. These OSes were designed in an era where you could get a 5% boost just by waiting a month. I think the real reason is they were also designed at a time when security was not as advanced as it is now, especially for Linux, and microkernels are more secure but also harder to write.