8 ms·

Unikernels

129 points by seeker89 5y ago

kristianpaul 5y ago
You have Alpine Linux container image that is around 2 Mb in size. And its a shame AWS doesn't allow volumes with sizes smaller than 1 GiB that i understand you can get really small images with Unikernels
- eyberg 5y ago
  There are benefits to this if you are deploying something like a 5G microservice or some other single-purpose, small binary that takes few resources but at the end of the day if you are deploying a JVM application that is gigabytes in size it doesn't really matter how small the base image is. Same thing applies in unikernel-land. The 1 gig limitation on clouds is not such a huge deal though as we can upload your image, however small it is as that size and then tell the cloud to provision the disk the size you need which acts kind of like a sparse file (but technically not one).
seeker89 5y ago
It looks like around 2015 there was a lot of hype around the topic, but as I tried to document the state of the art, I noticed there hasn't been that much pick up yet. What's your take?
- gjvc 5y ago
  I suspect it was because containers sufficed and using and creating the tooling around them consumed the attention of those who might have otherwise looked at unikernels.
  - walterbell 5y ago
    Some marketing material on docker vs unikernels, https://nanovms.com/learn/docker-vs-unikernels https://nanovms.com/learn/docker-vs-unikernels
  - seeker89 5y ago
    >I suspect it was because containers sufficed and using and creating the tooling around them consumed the attention of those who might have otherwise looked at unikernels. I think you might be right. Right place at the right time for containers.
- gwd 5y ago
  The two main advantages of unikernels are performance (reduced CPU and/or memory requirements) and security (hypervisor rather than container boundary). It turns out that basically nobody cares about either of those. I know someone who for a while worked at a company that was trying to make money off unikernels. They ended up re-implementing a database in a unikernel, in a way that gave them a 2.5x performance improvement over the original project -- or to put it a different way, switching should allow companies to cut their hosting costs in half. Even with such a clear "win", it was still a difficult sell. Think about how much backend code is written in interpreted languages like Python, or PHP, or Javascript, rather than in compiled languages like Go or Rust. It's just simpler to start with the simple solution and then throw money at the problem as you scale. And while performance may be one of the reasons that people are choosing Go or Rust for backends, if it were the only advantage, it's unlikely that would be compelling enough.
bluepizza 5y ago
Because they are a silly idea. Why would you spin up a full kernel to run a single application on top of a hypervisor that is balancing resources? Operating systems already solve this problem relatively well, without the overhead, via processes and containers.
- igorkraw 5y ago
  What is the functional difference of os=>hypervisor=>unikernel vm vs. os=>capabilities and pledges or containers? I would get if we use a unikernel approach running on bare metal for high security, specialised applications but this doesn't seem to exist?
  - eyberg 5y ago
    The difference is that the vast majority of people are deploying to the cloud so they are already deploying to a hypervisor. Every single cloud is built on top of virtualization. AWS used to use Xen, now they use KVM. Google Cloud is entirely built on KVM. Azure uses Hyper-V. The cloud is just an API for virtualization. Instead of AWS (hypervisor) => linux => k8s => containers unikernels advocate for AWS (hypervisor) => unikernel and that makes them run much faster in general (we've clocked upwards of 300% req/sec for go/rust webservers on AWS for instance) and a lot safer.
- jasode 5y ago
  >Why would you spin up a full kernel to run a single application You're misunderstanding the levels of abstraction probably because the word "kernel" within "unikernel" is throwing you off. The idea is to use a partial kernel (only the minimum services one needs). The so-called "kernel" is a library of code where you compile the minimum bits into the single-process image. >Operating systems already solve this problem relatively well, without the overhead, via processes A full operating system like Linux is expending extra overhead to schedule/prioritize/monitor processes (plural) -- because Linux is designed to be more general purpose and open-ended than a specialized unikernel. In contrast, a unikernel with only 1 singular process (say a specialized db engine) doesn't need to expend extra cpu on processe(s) scheduling. All that said, it doesn't seem like unikernels have enough advantages to attract widespread adoption like containers.
amouat 5y ago
It's worth looking at https://kontain.app/ https://kontain.app/ - seems to address a lot of the mentioned pain points in the article.
- kalmi10 5y ago
  Took some clicking around to find a good description of what this actually is: https://kontainapp.github.io/guide/overview/#a-new-approach-with-no-change-in-code-or-devops-tooling https://kontainapp.github.io/guide/overview/#a-new-approach-...
kitd 5y ago
I think micro VMs solved a lot of the issues that people had with regular containers and that unikernels were going to fix. There is still probably a performance improvement to be had with unikernels, but not enough to throw away all the investment companies made into containers. That said, there are options for running unikernels as K8s workloads if you want, eg NanoVMs: https://docs.ops.city/ops/k8s https://docs.ops.city/ops/k8s
goodpoint 5y ago
Counterpoint: the absence of logging, monitoring, host-based IDS, and all the system engineering tools you have on Linux is a big negative for security.
- wyuenho 5y ago
  You can compile in these metric collection facilities into the unikernel if you need them. The whole point of unikernel is to allow you to mix and match only the things you need.
- dimitar 5y ago
  You can log and send metrics from your app over a network (which you should probably be doing instead of writing on disk), monitor a VM, IDS makes no sense when you don't have logins and such. And the tools are rarely installed on "cattle" anyway.
  - goodpoint 5y ago
    > You can log and send metrics from your app over a network That's far from enough. It's conceptually and practically wrong to rely on the application to monitor itself. > IDS makes no sense when you don't have logins and such On the contrary, there is plenty that IDS can do for webapps.
    - eyberg 5y ago
      I'm not sure anyone would advocate the application to monitor itself. Many companies have entire teams of people that have to deal with keeping machines up and they get paid big bucks to do so. As for the IDS question/statement - can you explain in more detail? Are you talking about file integrity checks or? Unikernels don't have the concept of users or shells or remote login or many of the things that an IDS would actually be looking at. If it was something such as an attacker overwriting a shared library and you want to monitor or ensure that can't happen both of those operations are feasible in unikernels.
      - 5y ago
marmarama 5y ago
https://github.com/google/gvisor https://github.com/google/gvisor gives you essentially the same benefits as a unikernel without having to compromise on compatibility or recompile your apps, and integrates nicely with Kubernetes already. It also doesn't require a hypervisor at all.
- bitcharmer 5y ago
  Since this looks like an intermediary layer between userspace and the host kernel (at least if I'm reading it correctly), does anyone know what its performance impact is?
  - marmarama 5y ago
    The gVisor documentation has performance comparisons vs. cgroup-style 'traditional' containers at https://gvisor.dev/docs/architecture_guide/performance/ https://gvisor.dev/docs/architecture_guide/performance/ There is definitely some performance overhead, but in most cases it is less than hypervisor-based approaches.
    - eyberg 5y ago
      In gVisor if you aren't using hardware acceleration (eg: virtualization) then you are using ptrace which is incredibly slow.
      - marmarama 5y ago
        The benchmarks in the gVisor docs above are using ptrace, and they don't look too shabby.
        eyberg 5y ago
        Using redis as an example it's basically half in every benchmark: https://gvisor.dev/docs/architecture_guide/performance/#system-calls https://gvisor.dev/docs/architecture_guide/performance/#syst... IO bound tasks can be up to 10x slower using ptrace. I think using hardware acceleration gives you acceptable performance but ptrace is just a non-starter for prod.
seeker89 5y ago
The lightweight-ness argument is enticing, but I'm wondering if the fact that now you have to give these VMs enough RAM to run the app won't mean that you end up with worse flatpacking in terms of RAM than if you used containers?
- eyberg 5y ago
  It depends. A t2.nano has 512mb of memory. If you are using Go or something like that you could go much lower but any runtime environment such as ruby/python/node are going to want a minimum of a few hundred meg. If you are using the JVM you most definitely are going to want an instance with much more memory. At the end of the day your application decides how much memory it wants and the sysadmin/SRE/devops person just ensures it has enough so it doesn't crash. If you are hosting your own workloads and those workloads only need tens of megs of ram than you can pack as much as your hypervisor can handle. Alfred talks about booting 110,000 vms on one host before memory exhaustion: https://ieeexplore.ieee.org/document/6753801 https://ieeexplore.ieee.org/document/6753801
- convolvatron 5y ago
  kinda. if the guest uses something like virtio balloon the sharing isn't any worse. handling exhaustion isn't any better - although I think transparent migration of vms is more of a standard function than for containers, so that's some up I guess.
seeker89 5y ago
Also, some interesting perspectives on Linux unikernels: https://www.bu.edu/rhcollab/files/2019/04/unikernel.pdf https://www.bu.edu/rhcollab/files/2019/04/unikernel.pdf
api 5y ago
I want a unikernel that runs as a process with no special privileges. Huge bonus points if its portable to many common operating systems. Since recompilation is necessary anyway for unikernels, syscalls could be replaced by function calls or some other user mode thing instead of trapped. It would allow entire containers to run as processes. Not that interesting for cloud, but very interesting for distribution to endpoints or self-hostable apps.
- convolvatron 5y ago
  like user mode linux?
  - api 5y ago
    Yes, but like portable UML. I've seen a few people try to port UML to other OSes but nothing is mature.
- mwcampbell 5y ago
  Honest question: How is this different from an ordinary program written in a portable language with minimal reliance on shared libraries?
fwsgonzo 5y ago
There seems to be a lot of guessing by the author. VMs that run on the CPU have the same performance as everyone else, except when you trap into the hypervisor, and especially on buggy CPUs. Now, there are a lot of buggy CPUs out there right now, so I won't say any more. But just imagine a world where we have a fully working io_uring (all system calls that make sense), and less buggy CPUs.
UltraViolence 5y ago
Microkernels are a much more viable way of solving security problems in an operating system. Windows and Linux could both be rewritten as microkernels within a couple of months or years.
- muricula 5y ago
  Windows and Darwin (MacOS) were originally designed to be hybrid kernels, but compromised by allowing more and more stuff into kernel space until they were the monolithic kernels we know today. Changing code built up over 20-30 years while maintaining compatibility, security, and performance guarantees is not something which could be accomplished in a couple of months.
  - eyberg 5y ago
    The Hurd is actually older than Linux now at a ripe age of 32, although I think it was Mendel Rosenblaum that said "hypervisors/machine monitors were microkernels done right". https://www.usenix.org/legacy/events/hotos05/final_papers/full_papers/hand/hand.pdf https://www.usenix.org/legacy/events/hotos05/final_papers/fu...
    - SubjectToChange 5y ago
      Modern hypervisors are more or less modeled on microkernels, not the other way around. Microkernels are just naturally a good fit for hypervisors, hence why nearly all commercial microkernels have an embedded hypervisor solution.
  - lazyier 5y ago
    Early versions of Windows NT WAS a 100% honest microkernel OS. Microsoft abandoned that approach when they realized they had zero chance of being competitive with Unix with a microkernel architecture. Darwin was never intended to be anything except what it is, which is a monolithic kernel. The XNU kernel was based on FreeBSD kernel and Mach kernels. Some versions of the Mach kernel were microkernel, but many were not. Both NT and XNU incorporate message passing features from microkernels, but they are monolithic in that they are essentially a single large process. "Hybrid kernel" is more of a marketing thing than an engineering term. Microkernels are a dead-end and never stopped being a dead end. It's a lovely idea that didn't work out. They had limited commercial success in embedded systems, but only because those embedded systems didn't actually do very much and what they did was largely not performance critical.

129 points by seeker89 5y ago

You have Alpine Linux container image that is around 2 Mb in size. And its a shame AWS doesn't allow volumes with sizes smaller than 1 GiB that i understand you can get really small images with Unikernels

eyberg 5y ago
There are benefits to this if you are deploying something like a 5G microservice or some other single-purpose, small binary that takes few resources but at the end of the day if you are deploying a JVM application that is gigabytes in size it doesn't really matter how small the base image is. Same thing applies in unikernel-land. The 1 gig limitation on clouds is not such a huge deal though as we can upload your image, however small it is as that size and then tell the cloud to provision the disk the size you need which acts kind of like a sparse file (but technically not one).

seeker89 5y ago

It looks like around 2015 there was a lot of hype around the topic, but as I tried to document the state of the art, I noticed there hasn't been that much pick up yet. What's your take?

gjvc 5y ago
I suspect it was because containers sufficed and using and creating the tooling around them consumed the attention of those who might have otherwise looked at unikernels.
- walterbell 5y ago
  Some marketing material on docker vs unikernels, https://nanovms.com/learn/docker-vs-unikernels https://nanovms.com/learn/docker-vs-unikernels
- seeker89 5y ago
  >I suspect it was because containers sufficed and using and creating the tooling around them consumed the attention of those who might have otherwise looked at unikernels. I think you might be right. Right place at the right time for containers.
gwd 5y ago
The two main advantages of unikernels are performance (reduced CPU and/or memory requirements) and security (hypervisor rather than container boundary). It turns out that basically nobody cares about either of those. I know someone who for a while worked at a company that was trying to make money off unikernels. They ended up re-implementing a database in a unikernel, in a way that gave them a 2.5x performance improvement over the original project -- or to put it a different way, switching should allow companies to cut their hosting costs in half. Even with such a clear "win", it was still a difficult sell. Think about how much backend code is written in interpreted languages like Python, or PHP, or Javascript, rather than in compiled languages like Go or Rust. It's just simpler to start with the simple solution and then throw money at the problem as you scale. And while performance may be one of the reasons that people are choosing Go or Rust for backends, if it were the only advantage, it's unlikely that would be compelling enough.

bluepizza 5y ago

Because they are a silly idea. Why would you spin up a full kernel to run a single application on top of a hypervisor that is balancing resources? Operating systems already solve this problem relatively well, without the overhead, via processes and containers.

igorkraw 5y ago
What is the functional difference of os=>hypervisor=>unikernel vm vs. os=>capabilities and pledges or containers? I would get if we use a unikernel approach running on bare metal for high security, specialised applications but this doesn't seem to exist?
- eyberg 5y ago
  The difference is that the vast majority of people are deploying to the cloud so they are already deploying to a hypervisor. Every single cloud is built on top of virtualization. AWS used to use Xen, now they use KVM. Google Cloud is entirely built on KVM. Azure uses Hyper-V. The cloud is just an API for virtualization. Instead of AWS (hypervisor) => linux => k8s => containers unikernels advocate for AWS (hypervisor) => unikernel and that makes them run much faster in general (we've clocked upwards of 300% req/sec for go/rust webservers on AWS for instance) and a lot safer.
jasode 5y ago
>Why would you spin up a full kernel to run a single application You're misunderstanding the levels of abstraction probably because the word "kernel" within "unikernel" is throwing you off. The idea is to use a partial kernel (only the minimum services one needs). The so-called "kernel" is a library of code where you compile the minimum bits into the single-process image. >Operating systems already solve this problem relatively well, without the overhead, via processes A full operating system like Linux is expending extra overhead to schedule/prioritize/monitor processes (plural) -- because Linux is designed to be more general purpose and open-ended than a specialized unikernel. In contrast, a unikernel with only 1 singular process (say a specialized db engine) doesn't need to expend extra cpu on processe(s) scheduling. All that said, it doesn't seem like unikernels have enough advantages to attract widespread adoption like containers.

amouat 5y ago

It's worth looking at https://kontain.app/ https://kontain.app/ - seems to address a lot of the mentioned pain points in the article.

kalmi10 5y ago
Took some clicking around to find a good description of what this actually is: https://kontainapp.github.io/guide/overview/#a-new-approach-with-no-change-in-code-or-devops-tooling https://kontainapp.github.io/guide/overview/#a-new-approach-...

kitd 5y ago

I think micro VMs solved a lot of the issues that people had with regular containers and that unikernels were going to fix. There is still probably a performance improvement to be had with unikernels, but not enough to throw away all the investment companies made into containers. That said, there are options for running unikernels as K8s workloads if you want, eg NanoVMs: https://docs.ops.city/ops/k8s https://docs.ops.city/ops/k8s

goodpoint 5y ago

Counterpoint: the absence of logging, monitoring, host-based IDS, and all the system engineering tools you have on Linux is a big negative for security.

wyuenho 5y ago
You can compile in these metric collection facilities into the unikernel if you need them. The whole point of unikernel is to allow you to mix and match only the things you need.
dimitar 5y ago
You can log and send metrics from your app over a network (which you should probably be doing instead of writing on disk), monitor a VM, IDS makes no sense when you don't have logins and such. And the tools are rarely installed on "cattle" anyway.
- goodpoint 5y ago
  > You can log and send metrics from your app over a network That's far from enough. It's conceptually and practically wrong to rely on the application to monitor itself. > IDS makes no sense when you don't have logins and such On the contrary, there is plenty that IDS can do for webapps.
  - eyberg 5y ago
    I'm not sure anyone would advocate the application to monitor itself. Many companies have entire teams of people that have to deal with keeping machines up and they get paid big bucks to do so. As for the IDS question/statement - can you explain in more detail? Are you talking about file integrity checks or? Unikernels don't have the concept of users or shells or remote login or many of the things that an IDS would actually be looking at. If it was something such as an attacker overwriting a shared library and you want to monitor or ensure that can't happen both of those operations are feasible in unikernels.
    - 5y ago

marmarama 5y ago

https://github.com/google/gvisor https://github.com/google/gvisor gives you essentially the same benefits as a unikernel without having to compromise on compatibility or recompile your apps, and integrates nicely with Kubernetes already. It also doesn't require a hypervisor at all.

bitcharmer 5y ago
Since this looks like an intermediary layer between userspace and the host kernel (at least if I'm reading it correctly), does anyone know what its performance impact is?
- marmarama 5y ago
  The gVisor documentation has performance comparisons vs. cgroup-style 'traditional' containers at https://gvisor.dev/docs/architecture_guide/performance/ https://gvisor.dev/docs/architecture_guide/performance/ There is definitely some performance overhead, but in most cases it is less than hypervisor-based approaches.
  - eyberg 5y ago
    In gVisor if you aren't using hardware acceleration (eg: virtualization) then you are using ptrace which is incredibly slow.
    - marmarama 5y ago
      The benchmarks in the gVisor docs above are using ptrace, and they don't look too shabby.
      - eyberg 5y ago
        Using redis as an example it's basically half in every benchmark: https://gvisor.dev/docs/architecture_guide/performance/#system-calls https://gvisor.dev/docs/architecture_guide/performance/#syst... IO bound tasks can be up to 10x slower using ptrace. I think using hardware acceleration gives you acceptable performance but ptrace is just a non-starter for prod.

seeker89 5y ago

The lightweight-ness argument is enticing, but I'm wondering if the fact that now you have to give these VMs enough RAM to run the app won't mean that you end up with worse flatpacking in terms of RAM than if you used containers?

eyberg 5y ago
It depends. A t2.nano has 512mb of memory. If you are using Go or something like that you could go much lower but any runtime environment such as ruby/python/node are going to want a minimum of a few hundred meg. If you are using the JVM you most definitely are going to want an instance with much more memory. At the end of the day your application decides how much memory it wants and the sysadmin/SRE/devops person just ensures it has enough so it doesn't crash. If you are hosting your own workloads and those workloads only need tens of megs of ram than you can pack as much as your hypervisor can handle. Alfred talks about booting 110,000 vms on one host before memory exhaustion: https://ieeexplore.ieee.org/document/6753801 https://ieeexplore.ieee.org/document/6753801
convolvatron 5y ago
kinda. if the guest uses something like virtio balloon the sharing isn't any worse. handling exhaustion isn't any better - although I think transparent migration of vms is more of a standard function than for containers, so that's some up I guess.

seeker89 5y ago

Also, some interesting perspectives on Linux unikernels: https://www.bu.edu/rhcollab/files/2019/04/unikernel.pdf https://www.bu.edu/rhcollab/files/2019/04/unikernel.pdf

api 5y ago

I want a unikernel that runs as a process with no special privileges. Huge bonus points if its portable to many common operating systems. Since recompilation is necessary anyway for unikernels, syscalls could be replaced by function calls or some other user mode thing instead of trapped. It would allow entire containers to run as processes. Not that interesting for cloud, but very interesting for distribution to endpoints or self-hostable apps.

convolvatron 5y ago
like user mode linux?
- api 5y ago
  Yes, but like portable UML. I've seen a few people try to port UML to other OSes but nothing is mature.
mwcampbell 5y ago
Honest question: How is this different from an ordinary program written in a portable language with minimal reliance on shared libraries?

fwsgonzo 5y ago

There seems to be a lot of guessing by the author. VMs that run on the CPU have the same performance as everyone else, except when you trap into the hypervisor, and especially on buggy CPUs. Now, there are a lot of buggy CPUs out there right now, so I won't say any more. But just imagine a world where we have a fully working io_uring (all system calls that make sense), and less buggy CPUs.

UltraViolence 5y ago

Microkernels are a much more viable way of solving security problems in an operating system. Windows and Linux could both be rewritten as microkernels within a couple of months or years.

muricula 5y ago
Windows and Darwin (MacOS) were originally designed to be hybrid kernels, but compromised by allowing more and more stuff into kernel space until they were the monolithic kernels we know today. Changing code built up over 20-30 years while maintaining compatibility, security, and performance guarantees is not something which could be accomplished in a couple of months.
- eyberg 5y ago
  The Hurd is actually older than Linux now at a ripe age of 32, although I think it was Mendel Rosenblaum that said "hypervisors/machine monitors were microkernels done right". https://www.usenix.org/legacy/events/hotos05/final_papers/full_papers/hand/hand.pdf https://www.usenix.org/legacy/events/hotos05/final_papers/fu...
  - SubjectToChange 5y ago
    Modern hypervisors are more or less modeled on microkernels, not the other way around. Microkernels are just naturally a good fit for hypervisors, hence why nearly all commercial microkernels have an embedded hypervisor solution.
- lazyier 5y ago
  Early versions of Windows NT WAS a 100% honest microkernel OS. Microsoft abandoned that approach when they realized they had zero chance of being competitive with Unix with a microkernel architecture. Darwin was never intended to be anything except what it is, which is a monolithic kernel. The XNU kernel was based on FreeBSD kernel and Mach kernels. Some versions of the Mach kernel were microkernel, but many were not. Both NT and XNU incorporate message passing features from microkernels, but they are monolithic in that they are essentially a single large process. "Hybrid kernel" is more of a marketing thing than an engineering term. Microkernels are a dead-end and never stopped being a dead end. It's a lovely idea that didn't work out. They had limited commercial success in embedded systems, but only because those embedded systems didn't actually do very much and what they did was largely not performance critical.