8 ms·

My First Unikernel

267 points by the_eradicator 12y ago

j_s 12y ago
See also this discussion of MirageOS from 2.5 months ago: https://news.ycombinator.com/item?id=7726748 https://news.ycombinator.com/item?id=7726748
dkarapetyan 12y ago
I'm a little confused about the direction things are going in. I like high level languages and I like that the OS manages certain resources so I don't have to. This guy is writing directly to block devices from OCaml. Don't get me wrong it's all pretty cool but there is some kind of dissonance there I can't reconcile. Is Xen the new OS now?
- sp332 12y ago
  Yes, basically. You let the host manage memory, disk, network etc and your program is a guest OS that doesn't need all the extraneous processes that usually go with a whole OS. No background services or cron jobs. Edit: as enduser pointed out, in addition to saving resources, this means there is no context-switching from kernelspace to userspace in your guest VM!
  - wmf 12y ago
    You let the host manage memory, disk, network etc and your program is a [process] that doesn't need all the extraneous processes that usually go with a whole OS. No background services or cron jobs. So it's like Docker.
    - paulasmuth 12y ago
      No, this is fundamentally different from docker. Docker/cgroups/namespaces allow you to isolate multiple applications running in userspace on the same kernel to a very high degree. This gets rid of isolation and multiprocessing altogether and runs a single application without any kernel at all (i.e. with just a very limited amount of support code linked in). Put simply; the technology behind docker improves isolation between processes and this does the exact opposite.
      - andolanra 12y ago
        I don't see how this, "...does the exact opposite," of what Docker does—both are very different approaches to the similar goals of having individual processes run in isolated ways. Docker does it by isolating individual running Linux processes under a single kernel, whereas unikernel-based systems do it by building those processes as lightweight programs that get compiled to their own kernel images and then running those on a hypervisor to achieve isolation.
        paulasmuth 12y ago
        Yes, you are right, this is not a binary classification and it really depends on the perspective. What I was trying to say is that - on some levels - the two approaches are almost antithetical.
- hcarvalhoalves 12y ago
  > This guy is writing directly to block devices from OCaml. Don't get me wrong it's all pretty cool but there is some kind of dissonance there I can't reconcile. Is Xen the new OS now? For some applications removing this overhead might be interesting - e.g. databases tend to fight with the peculiarities in the host OS (scheduler, caching, disk access patterns). The major upside of this approach IMO is the added robustness you get, since deploying turns into a very deterministic procedure, there's no underlying OS updates and whatnot to break your app.
- derefr 12y ago
  Note, though, that he is indeed writing to a block device--which is much higher-level than writing to a disk. Remember back in the 80s, when the BIOS was actually an effective hardware abstraction layer--giving you a defined interrupt to ask the BIOS to, say, write to a disk--and the OS was just for module loading and scheduling and policy-based security? (Not that DOS did either of the latter.) Well, Xen isn't the new OS; instead, the domU is the new BIOS, and hypercalls are the new BIOS interrupts. I really hope to see Linux redone (or another *nix created) in this "unikernel" style, where everything hardware-like or HAL-like is taken out, and instead things like filesystem drivers are implemented directly in terms of hypercalls. I also hope to see UEFI reimplemented as a resident domU, such that a plain old desktop or notebook computer could treat its user OS as a container-image to be slung around, rather than having it "own" the hardware. UEFI actually already supports this mode of operation--allowing you to boot "UEFI applications" that keep UEFI around to provide BIOS-like functionality--but I don't know of a single OS that makes use of that, rather than overwriting the processor interrupt vectors and claiming all of physical memory for itself.
  - avsm 12y ago
    We have a feature issue tracking UEFI in Mirage, but noone's written the bootloader code yet: https://github.com/mirage/mirage/issues/187 https://github.com/mirage/mirage/issues/187 Interestingly, it looks like the easiest way to support Azure...
  - xxpor 12y ago
    >I also hope to see UEFI reimplemented as a resident domU, such that a plain old desktop or notebook computer could treat its user OS as a container-image to be slung around, rather than having it "own" the hardware. UEFI actually already supports this mode of operation--allowing you to boot "UEFI applications" that keep UEFI around to provide BIOS-like functionality--but I don't know of a single OS that makes use of that, rather than overwriting the processor interrupt vectors and claiming all of physical memory for itself. This is really the opposite direction things are going, especially in x86_64. PV is horrendously inefficient in 64 bit mode, because of the removal of CPU ring 1 and 2. http://wiki.xen.org/wiki/Virtualization_Spectrum#Problems_with_paravirtualization:_AMD_and_x86-64 http://wiki.xen.org/wiki/Virtualization_Spectrum#Problems_wi... HVM allows PCI passthrough if your CPU and chipset support it, which means domU now has direct access to the hardware, with no dom0/qemu layer to get in the way and slow things down.
enduser 12y ago
If it helps to add clarity to what this is about, Xen still requires a host (dom0) operating system, usually Linux but also NetBSD and OpenSolaris. Mirage OS (the basis for building your own unikernel) allows building applications as domU kernels. This is advantageous because it minimizes context-switching and allows all of the code in a Xen VM (domU) to be written in a safer language than C. So the dom0 OS (Linux) still determines which devices are exposed to the Unikernel. The Unikernel implementation is simple relative to a full "bare metal" OS because it only needs to support the Xen interfaces to block devices, the network, etc., and does not have to deal with disk drives, ethernet controllers, etc. If you are running your own hardware, you're probably better off using something like FreeBSD + Jails or Linux + LXC (Docker). The Unikernel approach is more appropriate for situations where you want to deploy applications on Xen-based cloud infrastructure (Amazon EC2, Rackspace Cloud Servers) and do not want to waste resources or increase security risk by running a full OS. The physical servers at Amazon and Rackspace are already running a full dom0 OS (probably Linux), so running another full OS on top of that just to run your app is inefficient.
- amirmc 12y ago
  Even if you have your own hardware there are scenarios where you might want stronger isolation, increased density and/or heterogeneous deployments. All of which is achievable with Unikernels running on Xen.
  - mwcampbell 12y ago
    How would unikernels running on Xen give you better density than containers (which are really just namespaced processes) running on a shared Unix kernel on bare metal? Wouldn't the Xen-based approach have more overhead?
    - avsm 12y ago
      This isn't supported yet, but unikernels on Xen don't currently switch address spaces (cr2 on x86). We're thinking about how to increase density by supporting process switching on Xen, which isn't hard, but needs to be done carefully. Xen also supports memory sharing among VMs if running in a hardware container, and Hwanju Kim added support for "PVH" mode to MiniOS recently which unlocks this functionality in Mirage. Early days on this work, but fun ones, since we aren't bound by the compatibility constraints of Linux containers, but have access to the same hardware resources.
paulasmuth 12y ago
This sounds like an awesome and fun project to hack on! However, optimizing around context switches and task preemptions is something you would usually do if your application is actually bound by IO/context switching, is extremely latency sensitive or when you are trying to squeeze the last bits of performance out of a machine. Why did you choose to build such a microoptimized system in a garbage collected language? Doesn't this defeat the purpose of the whole exercise? I feel the need for more powerful types and built-in/standardized exception handling too, but since performance seems to be one of your major goals, wouldn't something like C++ be a better fit here? You'd get proper error handling and a good type system (with some tradeoffs) without a significant performance penalty. On a sidenote, I agree that code written in a functional language with a strong type system tends to be easier to get right than bare C, but this doesn't imply that all low-level code is bug ridden and unsafe. In fact the linux kernel is one of the most stable and reliable pieces of software I've had the pleasure to work with so far. Suggesting there is a problem with the linux kernel because it contains "a large amount of C code in security-critical places" seems a bit dishonest.
- andolanra 12y ago
  Don't think of it as optimizing around context switches—think of it as just omitting what you don't need, and losing context-switches as a side benefit. A multi-user OS has a lot of stuff which might not be necessary for a single virtualized service (various security mechanisms, lots of file system niceties, running other services, &c), so writing a unikernel like this allows you to select exactly as much as you need for your particular service. OCaml isn't all that much slower than C++. To use the Programming Language Shootout as a rough esimation[^1], it can even come close to matching C++ in certain programs, and is rarely more than three times as slow. And of course—your type system will catch more errors and your resulting code will be much shorter (and in my opinion, at least, easier to understand.) Finally: the article didn't say "the Linux kernel"—it said "Ubuntu." The kernel itself might be secure and reliable, but a running Linux system is much, much more than just the kernel. And while C can be security-audited, many of the properties that are important to verify in a C program come entirely for free from something like OCaml—e.g., an arbitrary piece of C code might not segfault given certain input, but a given piece of OCaml code definitely won't. So maybe a running Linux system is "secure enough", but a unikernel like this will have a much smaller attack surface and stronger inherent security properties with basically no extra work. Disclaimer: I'm not the original author, I'm just speaking generally. [^1]: http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=ocaml&lang2=gpp&data=u64 http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...
616c 12y ago
So I have been following this stuff with a lot of interest, but I am not very familiar with Ocaml and have tried looking at Mirage docs with some limited downtime. If Ocaml can only handle one core and does not do SMP, how does it do in the cloud? Does this mean Mirage unikernels handle only one processor/core in Amazon and elsewhere?
- krat0sprakhar 12y ago
  > If Ocaml can only handle one core and does not do SMP, how does it do in the cloud? +1 to this question? Any one experienced enough in OCaml to answer this?
  - paulasmuth 12y ago
    One could argue that threading is not strictly necessary to utilize a multicore machine. A lot of applications (think web apps) can be implemented just fine as a single thread and then scaled up by increasing the number of processes (or the number of virtual machines in this case) rather than the number of threads. A lot of popular web frameworks follow this paradigm and I would guess that most application code running "in the cloud" is in fact single threaded.
    - hderms 12y ago
      You'd probably have to move more to a CSP type concurrency versus threads if you were to leverage multicore machines. Erlang doesn't allow individual processes to share memory with each other, but Erlang is undoubtedly a great platform to build apps in that make use of concurrency (depending on use case). Something tells me that the use cases for which one would use Mirage for would be those more likely to use Erlang-type concurrency semantics versus highly multithreaded code using shared memory.
  - 616c 12y ago
    Yeah I am familiar with the theory of alternative methods in a general way (CSP, message passing between full or light procs, green threads, etc.) but what is the story with OCaml and Mirage. I am curious because if it is the same as OCaml I am curious how much performance can be eeked out of it without SMP.
callumprentice 12y ago
As someone who knows nothing about lower level OS type stuff, I found this article very easy to understand, interesting and well written. Sounds like a lot of fun. Thank you.
callumprentice 12y ago
As someone who knows nothing about lower level OS type stuff, I found this article very easy to understand, interesting and well written. Sounds like a lot of fun. Thank you.
n0body 12y ago
interesting, although the old mantra of "hardware is cheap, developers are expensive" is still true. you could hire 10 perl/c/c++/javascript/php/etc dev with ease for your project, but struggle to find one ocaml dev. And even then your ocaml dev will need to know the mirage library, xen and have a good knowledge of way more stuff than your project scope that said, it's still really cool, but it's not something i'd use, and especially not in production
- rjsw 12y ago
  There is nothing stopping people from implementing something similar in other languages. The developer using this doesn't need to know anything about Xen, they just see a single address space system that runs their code.
  - n0body 12y ago
    i didn't say there was, but it's still complicating the task. i'm not saying it isn't cool, but it's a very round about way of doing something, and as such it becomes more expensive in development time and skill required
    - amirmc 12y ago
      More roundabout than the current paradigm of duplicate Linux stacks and supporting services for DevOps just to get stuff deployed? That doesn't make sense to me. Your argument is actually "It seems very new. Not enough people know/use it. Therefore, I won't use it." -- which is fine, but please don't mischaracterise it in terms of increased complexity or expense.
      - n0body 12y ago
        My argument is that you'll need better developers who cost more money. Not only that, but you're making the project more complicated. i.e if the project is to create x, then you have to also create y first and, it's harder to debug. and you've also got the added overhead of running xen. all in all it's a nice trick, but it's not very practical, because if it was practice we'd all be using dos for our vms, since you get the bare metal, and a little bit of a environment to bootstrap from.
guilloche 12y ago
Since unikernel still needs to run in hypervisor, so I believe the performance is worse than docker. Am I right?
shyknee 12y ago
Mirage was just featured on FLOSS Weekly: http://twit.tv/show/floss-weekly/302 http://twit.tv/show/floss-weekly/302

267 points by the_eradicator 12y ago

j_s 12y ago

See also this discussion of MirageOS from 2.5 months ago: https://news.ycombinator.com/item?id=7726748 https://news.ycombinator.com/item?id=7726748

dkarapetyan 12y ago

I'm a little confused about the direction things are going in. I like high level languages and I like that the OS manages certain resources so I don't have to. This guy is writing directly to block devices from OCaml. Don't get me wrong it's all pretty cool but there is some kind of dissonance there I can't reconcile. Is Xen the new OS now?

sp332 12y ago
Yes, basically. You let the host manage memory, disk, network etc and your program is a guest OS that doesn't need all the extraneous processes that usually go with a whole OS. No background services or cron jobs. Edit: as enduser pointed out, in addition to saving resources, this means there is no context-switching from kernelspace to userspace in your guest VM!
- wmf 12y ago
  You let the host manage memory, disk, network etc and your program is a [process] that doesn't need all the extraneous processes that usually go with a whole OS. No background services or cron jobs. So it's like Docker.
  - paulasmuth 12y ago
    No, this is fundamentally different from docker. Docker/cgroups/namespaces allow you to isolate multiple applications running in userspace on the same kernel to a very high degree. This gets rid of isolation and multiprocessing altogether and runs a single application without any kernel at all (i.e. with just a very limited amount of support code linked in). Put simply; the technology behind docker improves isolation between processes and this does the exact opposite.
    - andolanra 12y ago
      I don't see how this, "...does the exact opposite," of what Docker does—both are very different approaches to the similar goals of having individual processes run in isolated ways. Docker does it by isolating individual running Linux processes under a single kernel, whereas unikernel-based systems do it by building those processes as lightweight programs that get compiled to their own kernel images and then running those on a hypervisor to achieve isolation.
      - paulasmuth 12y ago
        Yes, you are right, this is not a binary classification and it really depends on the perspective. What I was trying to say is that - on some levels - the two approaches are almost antithetical.
hcarvalhoalves 12y ago
> This guy is writing directly to block devices from OCaml. Don't get me wrong it's all pretty cool but there is some kind of dissonance there I can't reconcile. Is Xen the new OS now? For some applications removing this overhead might be interesting - e.g. databases tend to fight with the peculiarities in the host OS (scheduler, caching, disk access patterns). The major upside of this approach IMO is the added robustness you get, since deploying turns into a very deterministic procedure, there's no underlying OS updates and whatnot to break your app.
derefr 12y ago
Note, though, that he is indeed writing to a block device--which is much higher-level than writing to a disk. Remember back in the 80s, when the BIOS was actually an effective hardware abstraction layer--giving you a defined interrupt to ask the BIOS to, say, write to a disk--and the OS was just for module loading and scheduling and policy-based security? (Not that DOS did either of the latter.) Well, Xen isn't the new OS; instead, the domU is the new BIOS, and hypercalls are the new BIOS interrupts. I really hope to see Linux redone (or another *nix created) in this "unikernel" style, where everything hardware-like or HAL-like is taken out, and instead things like filesystem drivers are implemented directly in terms of hypercalls. I also hope to see UEFI reimplemented as a resident domU, such that a plain old desktop or notebook computer could treat its user OS as a container-image to be slung around, rather than having it "own" the hardware. UEFI actually already supports this mode of operation--allowing you to boot "UEFI applications" that keep UEFI around to provide BIOS-like functionality--but I don't know of a single OS that makes use of that, rather than overwriting the processor interrupt vectors and claiming all of physical memory for itself.
- avsm 12y ago
  We have a feature issue tracking UEFI in Mirage, but noone's written the bootloader code yet: https://github.com/mirage/mirage/issues/187 https://github.com/mirage/mirage/issues/187 Interestingly, it looks like the easiest way to support Azure...
- xxpor 12y ago
  >I also hope to see UEFI reimplemented as a resident domU, such that a plain old desktop or notebook computer could treat its user OS as a container-image to be slung around, rather than having it "own" the hardware. UEFI actually already supports this mode of operation--allowing you to boot "UEFI applications" that keep UEFI around to provide BIOS-like functionality--but I don't know of a single OS that makes use of that, rather than overwriting the processor interrupt vectors and claiming all of physical memory for itself. This is really the opposite direction things are going, especially in x86_64. PV is horrendously inefficient in 64 bit mode, because of the removal of CPU ring 1 and 2. http://wiki.xen.org/wiki/Virtualization_Spectrum#Problems_with_paravirtualization:_AMD_and_x86-64 http://wiki.xen.org/wiki/Virtualization_Spectrum#Problems_wi... HVM allows PCI passthrough if your CPU and chipset support it, which means domU now has direct access to the hardware, with no dom0/qemu layer to get in the way and slow things down.

enduser 12y ago

If it helps to add clarity to what this is about, Xen still requires a host (dom0) operating system, usually Linux but also NetBSD and OpenSolaris. Mirage OS (the basis for building your own unikernel) allows building applications as domU kernels. This is advantageous because it minimizes context-switching and allows all of the code in a Xen VM (domU) to be written in a safer language than C. So the dom0 OS (Linux) still determines which devices are exposed to the Unikernel. The Unikernel implementation is simple relative to a full "bare metal" OS because it only needs to support the Xen interfaces to block devices, the network, etc., and does not have to deal with disk drives, ethernet controllers, etc. If you are running your own hardware, you're probably better off using something like FreeBSD + Jails or Linux + LXC (Docker). The Unikernel approach is more appropriate for situations where you want to deploy applications on Xen-based cloud infrastructure (Amazon EC2, Rackspace Cloud Servers) and do not want to waste resources or increase security risk by running a full OS. The physical servers at Amazon and Rackspace are already running a full dom0 OS (probably Linux), so running another full OS on top of that just to run your app is inefficient.

amirmc 12y ago
Even if you have your own hardware there are scenarios where you might want stronger isolation, increased density and/or heterogeneous deployments. All of which is achievable with Unikernels running on Xen.
- mwcampbell 12y ago
  How would unikernels running on Xen give you better density than containers (which are really just namespaced processes) running on a shared Unix kernel on bare metal? Wouldn't the Xen-based approach have more overhead?
  - avsm 12y ago
    This isn't supported yet, but unikernels on Xen don't currently switch address spaces (cr2 on x86). We're thinking about how to increase density by supporting process switching on Xen, which isn't hard, but needs to be done carefully. Xen also supports memory sharing among VMs if running in a hardware container, and Hwanju Kim added support for "PVH" mode to MiniOS recently which unlocks this functionality in Mirage. Early days on this work, but fun ones, since we aren't bound by the compatibility constraints of Linux containers, but have access to the same hardware resources.

paulasmuth 12y ago

This sounds like an awesome and fun project to hack on! However, optimizing around context switches and task preemptions is something you would usually do if your application is actually bound by IO/context switching, is extremely latency sensitive or when you are trying to squeeze the last bits of performance out of a machine. Why did you choose to build such a microoptimized system in a garbage collected language? Doesn't this defeat the purpose of the whole exercise? I feel the need for more powerful types and built-in/standardized exception handling too, but since performance seems to be one of your major goals, wouldn't something like C++ be a better fit here? You'd get proper error handling and a good type system (with some tradeoffs) without a significant performance penalty. On a sidenote, I agree that code written in a functional language with a strong type system tends to be easier to get right than bare C, but this doesn't imply that all low-level code is bug ridden and unsafe. In fact the linux kernel is one of the most stable and reliable pieces of software I've had the pleasure to work with so far. Suggesting there is a problem with the linux kernel because it contains "a large amount of C code in security-critical places" seems a bit dishonest.

andolanra 12y ago
Don't think of it as optimizing around context switches—think of it as just omitting what you don't need, and losing context-switches as a side benefit. A multi-user OS has a lot of stuff which might not be necessary for a single virtualized service (various security mechanisms, lots of file system niceties, running other services, &c), so writing a unikernel like this allows you to select exactly as much as you need for your particular service. OCaml isn't all that much slower than C++. To use the Programming Language Shootout as a rough esimation[^1], it can even come close to matching C++ in certain programs, and is rarely more than three times as slow. And of course—your type system will catch more errors and your resulting code will be much shorter (and in my opinion, at least, easier to understand.) Finally: the article didn't say "the Linux kernel"—it said "Ubuntu." The kernel itself might be secure and reliable, but a running Linux system is much, much more than just the kernel. And while C can be security-audited, many of the properties that are important to verify in a C program come entirely for free from something like OCaml—e.g., an arbitrary piece of C code might not segfault given certain input, but a given piece of OCaml code definitely won't. So maybe a running Linux system is "secure enough", but a unikernel like this will have a much smaller attack surface and stronger inherent security properties with basically no extra work. Disclaimer: I'm not the original author, I'm just speaking generally. [^1]: http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=ocaml&lang2=gpp&data=u64 http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...

616c 12y ago

So I have been following this stuff with a lot of interest, but I am not very familiar with Ocaml and have tried looking at Mirage docs with some limited downtime. If Ocaml can only handle one core and does not do SMP, how does it do in the cloud? Does this mean Mirage unikernels handle only one processor/core in Amazon and elsewhere?

krat0sprakhar 12y ago
> If Ocaml can only handle one core and does not do SMP, how does it do in the cloud? +1 to this question? Any one experienced enough in OCaml to answer this?
- paulasmuth 12y ago
  One could argue that threading is not strictly necessary to utilize a multicore machine. A lot of applications (think web apps) can be implemented just fine as a single thread and then scaled up by increasing the number of processes (or the number of virtual machines in this case) rather than the number of threads. A lot of popular web frameworks follow this paradigm and I would guess that most application code running "in the cloud" is in fact single threaded.
  - hderms 12y ago
    You'd probably have to move more to a CSP type concurrency versus threads if you were to leverage multicore machines. Erlang doesn't allow individual processes to share memory with each other, but Erlang is undoubtedly a great platform to build apps in that make use of concurrency (depending on use case). Something tells me that the use cases for which one would use Mirage for would be those more likely to use Erlang-type concurrency semantics versus highly multithreaded code using shared memory.
- 616c 12y ago
  Yeah I am familiar with the theory of alternative methods in a general way (CSP, message passing between full or light procs, green threads, etc.) but what is the story with OCaml and Mirage. I am curious because if it is the same as OCaml I am curious how much performance can be eeked out of it without SMP.

callumprentice 12y ago

As someone who knows nothing about lower level OS type stuff, I found this article very easy to understand, interesting and well written. Sounds like a lot of fun. Thank you.

callumprentice 12y ago

As someone who knows nothing about lower level OS type stuff, I found this article very easy to understand, interesting and well written. Sounds like a lot of fun. Thank you.

n0body 12y ago

interesting, although the old mantra of "hardware is cheap, developers are expensive" is still true. you could hire 10 perl/c/c++/javascript/php/etc dev with ease for your project, but struggle to find one ocaml dev. And even then your ocaml dev will need to know the mirage library, xen and have a good knowledge of way more stuff than your project scope that said, it's still really cool, but it's not something i'd use, and especially not in production

rjsw 12y ago
There is nothing stopping people from implementing something similar in other languages. The developer using this doesn't need to know anything about Xen, they just see a single address space system that runs their code.
- n0body 12y ago
  i didn't say there was, but it's still complicating the task. i'm not saying it isn't cool, but it's a very round about way of doing something, and as such it becomes more expensive in development time and skill required
  - amirmc 12y ago
    More roundabout than the current paradigm of duplicate Linux stacks and supporting services for DevOps just to get stuff deployed? That doesn't make sense to me. Your argument is actually "It seems very new. Not enough people know/use it. Therefore, I won't use it." -- which is fine, but please don't mischaracterise it in terms of increased complexity or expense.
    - n0body 12y ago
      My argument is that you'll need better developers who cost more money. Not only that, but you're making the project more complicated. i.e if the project is to create x, then you have to also create y first and, it's harder to debug. and you've also got the added overhead of running xen. all in all it's a nice trick, but it's not very practical, because if it was practice we'd all be using dos for our vms, since you get the bare metal, and a little bit of a environment to bootstrap from.

guilloche 12y ago

Since unikernel still needs to run in hypervisor, so I believe the performance is worse than docker. Am I right?

shyknee 12y ago

Mirage was just featured on FLOSS Weekly: http://twit.tv/show/floss-weekly/302 http://twit.tv/show/floss-weekly/302