4 ms·

While what you said may be theoretically true, does anyone run microkernels or exokernels on servers in production? I know microkernels like QNX are extremely p

While what you said may be theoretically true, does anyone run microkernels or exokernels on servers in production? I know microkernels like QNX are extremely popular and effective in the embedded market and as far as I can tell exokernels are more research oriented. But if I want to use a server with dozens of cores, hundreds of gigs of memory, dozens of drives, and several NICS what microkernel could I use? Currently I would use FreeBSD for such a role. What microkernel or exokernel provides all the features of FreeBSD on the server, like a COW file system, visibility(dtrace, vmstat, iostat, etc), os-level virtualization(jails), hypervisor(bhyve), performant network stack, linux binary emulation, and is open source? Unless people write a microkernel or exokernel that can replace Linux or FreeBSD on the server than people will continue to put monolithic kernels into production.

rwmj 10y ago
L4 runs on every phone in the world (to a first approximation). Xen (derived from the original exokernel work) runs AWS on a bazillion servers. Linux, FreeBSD, C are examples of how a heroic amount of work can make up for poor design, but it's clear that with a better architecture and the same number of developers they would be much more featureful. It's also significant, I think, that all the important distributed filesystems (eg. Gluster) are implemented as userspace servers, not as kernel drivers. Once you go beyond a certain level of complexity implementing a kernel filesystem isn't really feasible even with heroic effort.
- 2trill2spill 10y ago
  > L4 runs on every phone in the world (to a first approximation). I was talking about servers not phones. > Xen (derived from the original exokernel work) runs AWS on a bazillion servers. You forgot to mention Xen is running on Linux in AWS.
  - rwmj 10y ago
    Linux runs on Xen in AWS.
    - 2trill2spill 10y ago
      > Linux runs on Xen in AWS. Yea your right about that. But my point still stands almost all of AWS is being used to host monolithic kernels because the monolithic kernels(Linux, FreeBSD) have the features people wan't for running their business/organization.
      - kbenson 10y ago
        > But my point still stands almost all of AWS is being used to host monolithic kernels because the monolithic kernels(Linux, FreeBSD) have the features people wan't The implication there is that they have the features because they are monolithic kernels, which is presuming the very thing being discussed. The only thing you can say without a lot more proof is that people use Linux of FreeBSD because they have features people want. Whether being monolithic is a causation or correlation with those features is not established (in this discussion) yet.
        2trill2spill 10y ago
        > The implication there is that they have the features because they are monolithic kernels, which is presuming the very thing being discussed. The only thing you can say without a lot more proof is that people use Linux of FreeBSD because they have features people want. Whether being monolithic is a causation or correlation with those features is not established (in this discussion) yet. I'm not saying FreeBSD and Linux have those features because they are monolithic, i'm saying people use monolithic kernels because they are more feature complete at this point in time. More specifically if I want to run a big web app in production there are no suitable microkernels or exokernels to do so now.
        kbenson 10y ago
        > More specifically if I want to run a big web app in production there are no suitable microkernels or exokernels to do so now. Windows NT is a hybrid kernel[1], so there's somewhat conflicting evidence. Really, I think I could make an argument that the reason a microkernel hasn't gained popularity is because there's far too much focus on performance to the detriment of security and stability in much of the last few decades program development, much less OS development. People aren't good at correctly estimating risk and reward for future events that are more than a few years out, and worse when the risk is abstract in nature. Would everything be running slower if we used microkernels for the majority of systems? Probably (but who knows by how much? If there was a lot more work focused on that problem, we might have ways to alleviate much of it by now). Would our systems be more secure and less prone to software errors? I think so. Do I think it's possible we as a culture (species?) could choose stability and security over performance? Not without enforcement negative consequences when developers supply buggy and/or exploitable code, and consequences for those companies that choose to run that code even if they are carefully advised on the possible consequences. So, not likely, and that's a huge discussion in itself. 1: https://en.wikipedia.org/wiki/Hybrid_kernel#NT_kernel https://en.wikipedia.org/wiki/Hybrid_kernel#NT_kernel
  - jdc 10y ago
    Don't move the goal post.
- geofft 10y ago
  > Xen (derived from the original exokernel work) runs AWS on a bazillion servers. So there's an interesting point there: what's the practical difference between a microkernel architecture and a microservices architecture? Local privilege escalation on Linux has been entirely too easy. But privilege escalation on Xen (from one domain to another) has been relatively hard, and AWS works on that assumption. So you treat each Linux image as if it's within one privilege domain, and exploits like the one in this article don't affect you. You can't get from one VM to another, because each VM has its own filesystem server. The only practical difference, I think, is that microservices exist in the real world, and microkernels have been tried many times and have (with a few exceptions) failed. But the security benefits are about the same.
  - nostrademons 10y ago
    It's pretty similar from a security standpoint. The point of a microkernel is to get process-isolation guarantees at a much finer-grained level than traditional monolithic kernels allow: not only do you guarantee that processes can't stomp all over each other, but you also guarantee that they can't stomp on kernel data structures like filesystems or drivers. Microservices give this as well, since if there's only one process running on the box, there's not much for it to stomp on (except, well, all the other OS cruft like shells and debugging tools that typically goes on a VM but could be used to turn a pwned box into a spam farm). The big downside of microservices is the complexity they introduce, both in terms of the networking & IPC overhead (which isn't free, and in many cases can consume a majority of the CPU time of the service) and in terms of programming & devops time. Hence some of the interest in unikernels, which both eliminate the other OS cruft that's attackable and eliminate much of the CPU overhead of communicating between microservices. The tooling doesn't really exist for unikernels yet, though - there's no pragmatic, drop-in solution where you can just write a binary and be able to trace & debug performance problems on a production box.
wangchow 10y ago
Gnu Hurd! :) https://www.gnu.org/software/hurd/index.html https://www.gnu.org/software/hurd/index.html
- 2trill2spill 10y ago
  Hurd has nowhere near the features of FreeBSD or Linux, and their status page even says it's not ready for production. https://www.gnu.org/software/hurd/hurd/status.html https://www.gnu.org/software/hurd/hurd/status.html
monocasa 10y ago
I'd call the original Xbox and the Xbox 360's OS an exokernel. Threads, timers, encryption, and interrupt dispatching were handled in kernel, but USB, Ethernet, audio, and GPU drivers ran in the application image.

rwmj 10y ago

L4 runs on every phone in the world (to a first approximation). Xen (derived from the original exokernel work) runs AWS on a bazillion servers. Linux, FreeBSD, C are examples of how a heroic amount of work can make up for poor design, but it's clear that with a better architecture and the same number of developers they would be much more featureful. It's also significant, I think, that all the important distributed filesystems (eg. Gluster) are implemented as userspace servers, not as kernel drivers. Once you go beyond a certain level of complexity implementing a kernel filesystem isn't really feasible even with heroic effort.

2trill2spill 10y ago
> L4 runs on every phone in the world (to a first approximation). I was talking about servers not phones. > Xen (derived from the original exokernel work) runs AWS on a bazillion servers. You forgot to mention Xen is running on Linux in AWS.
- rwmj 10y ago
  Linux runs on Xen in AWS.
  - 2trill2spill 10y ago
    > Linux runs on Xen in AWS. Yea your right about that. But my point still stands almost all of AWS is being used to host monolithic kernels because the monolithic kernels(Linux, FreeBSD) have the features people wan't for running their business/organization.
    - kbenson 10y ago
      > But my point still stands almost all of AWS is being used to host monolithic kernels because the monolithic kernels(Linux, FreeBSD) have the features people wan't The implication there is that they have the features because they are monolithic kernels, which is presuming the very thing being discussed. The only thing you can say without a lot more proof is that people use Linux of FreeBSD because they have features people want. Whether being monolithic is a causation or correlation with those features is not established (in this discussion) yet.
      - 2trill2spill 10y ago
        > The implication there is that they have the features because they are monolithic kernels, which is presuming the very thing being discussed. The only thing you can say without a lot more proof is that people use Linux of FreeBSD because they have features people want. Whether being monolithic is a causation or correlation with those features is not established (in this discussion) yet. I'm not saying FreeBSD and Linux have those features because they are monolithic, i'm saying people use monolithic kernels because they are more feature complete at this point in time. More specifically if I want to run a big web app in production there are no suitable microkernels or exokernels to do so now.
        kbenson 10y ago
        > More specifically if I want to run a big web app in production there are no suitable microkernels or exokernels to do so now. Windows NT is a hybrid kernel[1], so there's somewhat conflicting evidence. Really, I think I could make an argument that the reason a microkernel hasn't gained popularity is because there's far too much focus on performance to the detriment of security and stability in much of the last few decades program development, much less OS development. People aren't good at correctly estimating risk and reward for future events that are more than a few years out, and worse when the risk is abstract in nature. Would everything be running slower if we used microkernels for the majority of systems? Probably (but who knows by how much? If there was a lot more work focused on that problem, we might have ways to alleviate much of it by now). Would our systems be more secure and less prone to software errors? I think so. Do I think it's possible we as a culture (species?) could choose stability and security over performance? Not without enforcement negative consequences when developers supply buggy and/or exploitable code, and consequences for those companies that choose to run that code even if they are carefully advised on the possible consequences. So, not likely, and that's a huge discussion in itself. 1: https://en.wikipedia.org/wiki/Hybrid_kernel#NT_kernel https://en.wikipedia.org/wiki/Hybrid_kernel#NT_kernel
- jdc 10y ago
  Don't move the goal post.
geofft 10y ago
> Xen (derived from the original exokernel work) runs AWS on a bazillion servers. So there's an interesting point there: what's the practical difference between a microkernel architecture and a microservices architecture? Local privilege escalation on Linux has been entirely too easy. But privilege escalation on Xen (from one domain to another) has been relatively hard, and AWS works on that assumption. So you treat each Linux image as if it's within one privilege domain, and exploits like the one in this article don't affect you. You can't get from one VM to another, because each VM has its own filesystem server. The only practical difference, I think, is that microservices exist in the real world, and microkernels have been tried many times and have (with a few exceptions) failed. But the security benefits are about the same.
- nostrademons 10y ago
  It's pretty similar from a security standpoint. The point of a microkernel is to get process-isolation guarantees at a much finer-grained level than traditional monolithic kernels allow: not only do you guarantee that processes can't stomp all over each other, but you also guarantee that they can't stomp on kernel data structures like filesystems or drivers. Microservices give this as well, since if there's only one process running on the box, there's not much for it to stomp on (except, well, all the other OS cruft like shells and debugging tools that typically goes on a VM but could be used to turn a pwned box into a spam farm). The big downside of microservices is the complexity they introduce, both in terms of the networking & IPC overhead (which isn't free, and in many cases can consume a majority of the CPU time of the service) and in terms of programming & devops time. Hence some of the interest in unikernels, which both eliminate the other OS cruft that's attackable and eliminate much of the CPU overhead of communicating between microservices. The tooling doesn't really exist for unikernels yet, though - there's no pragmatic, drop-in solution where you can just write a binary and be able to trace & debug performance problems on a production box.

wangchow 10y ago

Gnu Hurd! :) https://www.gnu.org/software/hurd/index.html https://www.gnu.org/software/hurd/index.html

2trill2spill 10y ago
Hurd has nowhere near the features of FreeBSD or Linux, and their status page even says it's not ready for production. https://www.gnu.org/software/hurd/hurd/status.html https://www.gnu.org/software/hurd/hurd/status.html

monocasa 10y ago

I'd call the original Xbox and the Xbox 360's OS an exokernel. Threads, timers, encryption, and interrupt dispatching were handled in kernel, but USB, Ethernet, audio, and GPU drivers ran in the application image.