4 ms·

I wonder why unikernels haven't caught on. Years ago when they started to be hyped, I imagined the technology would surpass the adoption of containers. In princ

by imiric 3y ago

I wonder why unikernels haven't caught on. Years ago when they started to be hyped, I imagined the technology would surpass the adoption of containers. In principle, they have the security and isolation of VMs which containers lack, but also have even less overhead than containers, and can run on bare metal. It sounds like the best of both worlds. The main issues years ago were developer tooling, packaging, and difficulty with debugging, so the UX was far from what Docker did for containers. But I was hoping that these issues would be addressed (are they?), and that a defacto standard would emerge that would propel them into the mainstream. Instead, we have these disparate niche tools and services that seem to be competing, but there's no widespread industry adoption. I'm sure that the evolution of the container ecosystem made unikernels less desirable, and the increased popularity of container orchestration engines like Kubernetes cemented this trend. But I wish that the unikernel ecosystem were more mature so that it's at least a topic of conversation in tech circles, and k8s wasn't such an obvious choice. Though I might be in the wrong circles and have outdated information, so I'm curious to know what HN thinks.

vbezhenar 3y ago
I feel like you’re mixing layers. Alternative to VM is Linux namespaces. They provide isolation. Docker, Kubernetes, LXC, systemd-spawn and whatnot are just thin wrappers around Linux API. There’s nothing that would prevent you from running workloads in VMs and managing them with Kubernetes. Just write proper adapters. Actually there are projects in the wild which do exactly that. As to why VM is supposed to be lighter than container, I don’t really understand. I think it’s the other way around. Industry does not really care about security and losing performance for unclear benefits won’t fly for masses.
- imiric 3y ago
  > Alternative to VM is Linux namespaces. They're really not. Containers (and by that I mean their underlying technology) are not alternatives to VMs, in the same way that unikernels are not alternatives to containers (yet, anyway). They're technologies that can be used for similar purposes, but in practice, they all have their tradeoffs that should be considered before use. For one, containers share the same kernel of the host machine. The isolation provided by VMs is on a deeper level, which also makes them more secure. Unikernels are more like VMs in this sense. My point is that the only drawback of unikernels in theory is the immature tooling and support around it. So I'm wondering if there is another reason that they haven't been as widely adopted as containers. > As to why VM is supposed to be lighter than container, I don’t really understand. You misunderstood. I said that unikernels, not VMs, have less overhead than containers. Containers require a minimal, but fully functioning, OS to work, while unikernels only require a cut-down version of a kernel. Whether unikernels are more lightweight is debatable, since containers are merely another process running on the same host OS with improved isolation, while unikernels depend on a hypervisor to work, but if we define "lightweight" to include boot time and performance, unikernels have certainly surpassed containers in this sense as well[1]. So the main benefit containers had over VMs is also an attribute of unikernels. [1]: https://www.ietf.org/proceedings/95/slides/slides-95-nfvrg-2.pdf https://www.ietf.org/proceedings/95/slides/slides-95-nfvrg-2...
  - ComputerGuru 3y ago
    If you consider a firecracker app to be like a unikernel (just not integrated into one binary) then it’s certainly lighter weight than a Docker image.
supriyo-biswas 3y ago
Erlang’s cluster spawn + a unikernel model would be a match made in heaven.
- eyberg 3y ago
  FYI, erlang && elixir workloads can run on Nanos: https://github.com/nanovms/ops-examples/tree/master/elixir https://github.com/nanovms/ops-examples/tree/master/elixir . I do understand what you are asking for though. There used to be an older project called erlang-on-xen: https://github.com/cloudozer/ling https://github.com/cloudozer/ling . It'd be great to see this idea revisited although you'll need to create some sort of new handling/framework.
rwmj 3y ago
UX is definitely an issue. One interesting thing coming up is confidential computing (where basically the RAM and registers of the workload are encrypted so they hypervisor cannot see them). The hardware implementations are all based around VMs, so they require you to run a full kernel, even if what you're doing is basically a confidential container workload. We think there might be a niche for unikernels here.
- blueflow 3y ago
  > where basically the RAM and registers of the workload are encrypted so they hypervisor cannot see them I call BS. There is no way to hide code from the platform it runs on.
  - osigurdson 3y ago
    Other than homomorphic encryption of course. https://en.wikipedia.org/wiki/Homomorphic_encryption https://en.wikipedia.org/wiki/Homomorphic_encryption
  - rwmj 3y ago
    I didn't say "from the platform", I very specifically said they're encrypted so the hypervisor or other VMs cannot see them. Believe it or not, that's how Intel TDX, AMD SEV and other systems do work, and these are real world implementations you can buy right now.
    - immibis 3y ago
      Indeed, it's defense in depth. The hypervisor shouldn't be compromised, and other VMs shouldn't be able to read your VM's data, but if they are or can for some reason, it's encrypted.
cmrdporcupine 3y ago
I think you've touched on it in regards to ecosystem maturity. When ever I traipse through this topic I do find mostly research projects. I know there are startups in this space, but I think it'd be hard to show a $$ value proposition for them, so I can understand why most things look quite research focused instead. I think it's likely that instead of a generic unikernel ecosystem and companies built around that, we could instead see very specifically targeted unikernel applications as SaaS offerings. And as consumers of such a service we might not even know (or care) they are unikernel hosted at all, just that they're faster. E.g. my comment below about DB unikernels. If, say, e.g. Snowflake were to run bare on the hypervisor for extra oomph... it wouldn't impact their customers beyond some rather transparent performance improvements. And such a thing would be done only if there was profit in it that would exceed the R&D costs -- put another way, if you're a company considering this, you could either buy more and bigger hardware, or invest in unikernel... former is far less risk. Which is why I think where we'd see action in this space would be from very specific performance oriented spaces. E.g. super low latency transaction processing for e.g. telecomes, finance / high-frequency-trading, ad-tech RTB, games ... rather than say large scale analytic workloads.
bluepizza 3y ago
Because you can replace a group of unikernel VMs with an operating system and processes, and rely on those abstractions instead. Unikernels haven't caught on because the use cases where they are the most adequate solution are limited.
- crabbone 3y ago
  That's not true. Unikernels are a way to do the same thing you do with your "process abstraction", but better: using less resources, minimizing the number of failure points, improving speed of deployment, minimizing dependencies which also allows for faster iteration when developing software. The reason they didn't catch on is because overwhelming majority of programmers aren't inventors and aren't knowledgeable enough about the system aspect of their environment -- they need tools, frameworks, bundles of documentation and support forums to get anywhere. Now, all of the aids listed above can either be created by a dedicated group of enthusiasts or by large corporations investing into such a process. Large corporations have little interest in software qualities mentioned above because the "good enough" containers are capable of solving their business problems. So... we are left with enthusiasts, who are few and whose work isn't being integrated into "mainstream" development process. Will these enthusiasts eventually gain enough momentum to make it really attractive for the whole industry to switch, or with it be another IPv4 vs IPv6 -- time will show.
  - immibis 3y ago
    In what way does a hypervisor and unikernels use less resources, reduce the number of failure points, improve deployment speed or reduce dependencies compared to an operating system (no hypervisor) and processes?
    - eyberg 3y ago
      If you are comparing a bare metal system, indeed you might not see a lot of benefits. However, there is a basic assumption that these workloads are running in the cloud which means they are already virtualized and then when you compare against a ubuntu/debian/etc. vm they get their benefits.
pjmlp 3y ago
They have caught on, that is basically in a way what serveless do. I don't care if the language runtime runs bare metal, on a container, or inside some OS VM.
- edgyquant 3y ago
  That isn’t what serverless does. Serverless tends to run on Linux but can be any OS.
  - pjmlp 3y ago
    Depends on the cloud vendor, and on AWS, firecracker VM is close enough to the original idea.
crabbone 3y ago
Because it's hard to find programmers who can do it. Overwhelming majority of programming today is happening in userspace. And even there, it's typical to be far removed from system interfaces. Popular language runtimes s.a. Python or Java or JavaScript create a layer of obfuscation on top of system interfaces. Programmers who could realistically pull this out are either in embedded field or in OS / system field. OS / system is very tiny compared to everything else, but even with embedded being larger, the problem is also the lack of expertise necessary to tie the knowledge necessary to deal with unikernels with knowledge necessary to develop enterprise or user-facing applications. Another aspect is the lack of support from large cloud vendors. In principle, you could create RMIs (AWS EC2 VM images) from scratch, but it will be hard to integrate them with the rest of the cloud infrastructure. Popular tools that create RMIs aren't designed to deal with unikernel applications. They often assume that UNIX shell is available or that cloud-init is available etc. It's even worse with VHD. I don't know what the state of events is in GCP, but wouldn't expect that to be any different. Yet another aspect is the lack of instrumentation for debugging. A lot of common instrumentation assumes or relies on stuff like UNIX shell or presence of standard UNIX utilities. This is even more noticeable in automation / CI side. Containers are a lot easier because they rely on existing, however bloated and broken, infrastructure. They aren't better, and the way forward for containers promises no increase in quality... but it's the easy way "forward" (in temporal sense), and that's why it's being taken.
- eyberg 3y ago
  Definitely agree with the top part, however, I should note that, ops, the tool's, whole existence is to create disk images and upload them to any cloud, any hypervisor. In particular, both https://ops.city https://ops.city && https://nanos.org https://nanos.org are Go unikernels running on GCP and their deploys take just a few seconds to push out. AWS can be even faster cause we skip the s3 upload part. We also have lots of people using Azure which would be utilizing vhdx.
eyberg 3y ago
So my opinion, having worked in the space for years and the author of this post, is that the vast majority of people (engineers included) underestimate the sheer amount of work building a kernel that can run lots of applications on various clouds takes. Linux has been in development for ~30 years, Unix for 50. That is to say it isn't necessarily lack of use-cases - it is ensuring you have enough support built for said use-cases when some user wants to use it. I'll give an example. Years ago people would say something like "I see you have GCP support - that's awesome - what about Azure?" Back then I'd say something like, well, give us a month cause we have to write new disk/network drivers but yes we can make that happen - a month is too long for that person though and so you immediately lose that person as a user/potential customer and have to wait until they come back. (We have great azure support now fyi.) Now take that same example and add it to every language, every cloud, etc. It's a ton of work.
- imiric 3y ago
  Thanks for your perspective, and kudos on the article, impressive work! So it is a matter of lack of support and general tooling around it. Anyone interested in the technology needs to write support for each individual tech stack and infrastructure to run it. The amount of complex work and the scarcity of people knowledgeable enough to do it is a huge hurdle. Whereas Docker came around at a time when the low-level building blocks were already in place in Linux. Containers were already widely used in the industry, but each company had their own tooling around them. Docker made this technology much easier to use, which in turn helped define the standards that we use today. So I'm still hopeful that something similar will happen eventually with unikernels. There's been a lot of work in improving the tooling around it, as your company has done, adding support for various tech stacks, optimizing hypervisors to run them efficiently, etc., so hopefully in a few years a tool that greatly improves the UX will be built, and an industry standard will emerge. I think open source is a crucial part for this to happen. BTW, Nanos looks very cool!
toast0 3y ago
I've been messing around with a hobby OS that's adjacent to unikernels for a while now. Once I'm done enough, I plan to benchmark running my application+OS vs running my application as PID 1 in a mainstream OS. I imagine, if I'm lucky, performance will be similar, although chances are, the mainstream OS will be significantly better, and development time for application as PID 1 will be less than a month vs I've been tooling around for 3 years at this point (only really focused for a few months, though) Most applications lack the introspection to be really operable in a unikernel situation too, so then you're looking at unikernel with extras for administration vs minimal mainstream OS with a small set of administrative tools, and that's an easy win for the mainstream OS.
immibis 3y ago
Unikernels have caught on, in the form of processes. You have to run your unikernel in a hypervisor anyway, so why not give the hypervisor a bunch of features and then it's an operating system with a weird hardware-emulating interface. De-weird the interface, and then you have a normal operating system.
- eyberg 3y ago
  The big difference is the hypervisor in this case is the public cloud which is not just multi-tenant but massively multi-tenant. That is the host and vmware and aws and everyone else did a really excellent job of abstracting the host away to the extent that most developers don't even know what a server looks like anymore. Unikernels take the same approach but abstract the guest away. The abstraction is to treat each vm as an individual application (since we're treating the cloud as the operating system) versus trying to micro-manage hundreds or thousands of virtual operating systems inside the hypervisor.

by imiric 3y ago

vbezhenar 3y ago

I feel like you’re mixing layers. Alternative to VM is Linux namespaces. They provide isolation. Docker, Kubernetes, LXC, systemd-spawn and whatnot are just thin wrappers around Linux API. There’s nothing that would prevent you from running workloads in VMs and managing them with Kubernetes. Just write proper adapters. Actually there are projects in the wild which do exactly that. As to why VM is supposed to be lighter than container, I don’t really understand. I think it’s the other way around. Industry does not really care about security and losing performance for unclear benefits won’t fly for masses.

imiric 3y ago
> Alternative to VM is Linux namespaces. They're really not. Containers (and by that I mean their underlying technology) are not alternatives to VMs, in the same way that unikernels are not alternatives to containers (yet, anyway). They're technologies that can be used for similar purposes, but in practice, they all have their tradeoffs that should be considered before use. For one, containers share the same kernel of the host machine. The isolation provided by VMs is on a deeper level, which also makes them more secure. Unikernels are more like VMs in this sense. My point is that the only drawback of unikernels in theory is the immature tooling and support around it. So I'm wondering if there is another reason that they haven't been as widely adopted as containers. > As to why VM is supposed to be lighter than container, I don’t really understand. You misunderstood. I said that unikernels, not VMs, have less overhead than containers. Containers require a minimal, but fully functioning, OS to work, while unikernels only require a cut-down version of a kernel. Whether unikernels are more lightweight is debatable, since containers are merely another process running on the same host OS with improved isolation, while unikernels depend on a hypervisor to work, but if we define "lightweight" to include boot time and performance, unikernels have certainly surpassed containers in this sense as well[1]. So the main benefit containers had over VMs is also an attribute of unikernels. [1]: https://www.ietf.org/proceedings/95/slides/slides-95-nfvrg-2.pdf https://www.ietf.org/proceedings/95/slides/slides-95-nfvrg-2...
- ComputerGuru 3y ago
  If you consider a firecracker app to be like a unikernel (just not integrated into one binary) then it’s certainly lighter weight than a Docker image.

supriyo-biswas 3y ago

Erlang’s cluster spawn + a unikernel model would be a match made in heaven.

eyberg 3y ago
FYI, erlang && elixir workloads can run on Nanos: https://github.com/nanovms/ops-examples/tree/master/elixir https://github.com/nanovms/ops-examples/tree/master/elixir . I do understand what you are asking for though. There used to be an older project called erlang-on-xen: https://github.com/cloudozer/ling https://github.com/cloudozer/ling . It'd be great to see this idea revisited although you'll need to create some sort of new handling/framework.

rwmj 3y ago

UX is definitely an issue. One interesting thing coming up is confidential computing (where basically the RAM and registers of the workload are encrypted so they hypervisor cannot see them). The hardware implementations are all based around VMs, so they require you to run a full kernel, even if what you're doing is basically a confidential container workload. We think there might be a niche for unikernels here.

blueflow 3y ago
> where basically the RAM and registers of the workload are encrypted so they hypervisor cannot see them I call BS. There is no way to hide code from the platform it runs on.
- osigurdson 3y ago
  Other than homomorphic encryption of course. https://en.wikipedia.org/wiki/Homomorphic_encryption https://en.wikipedia.org/wiki/Homomorphic_encryption
- rwmj 3y ago
  I didn't say "from the platform", I very specifically said they're encrypted so the hypervisor or other VMs cannot see them. Believe it or not, that's how Intel TDX, AMD SEV and other systems do work, and these are real world implementations you can buy right now.
  - immibis 3y ago
    Indeed, it's defense in depth. The hypervisor shouldn't be compromised, and other VMs shouldn't be able to read your VM's data, but if they are or can for some reason, it's encrypted.

cmrdporcupine 3y ago

I think you've touched on it in regards to ecosystem maturity. When ever I traipse through this topic I do find mostly research projects. I know there are startups in this space, but I think it'd be hard to show a $$ value proposition for them, so I can understand why most things look quite research focused instead. I think it's likely that instead of a generic unikernel ecosystem and companies built around that, we could instead see very specifically targeted unikernel applications as SaaS offerings. And as consumers of such a service we might not even know (or care) they are unikernel hosted at all, just that they're faster. E.g. my comment below about DB unikernels. If, say, e.g. Snowflake were to run bare on the hypervisor for extra oomph... it wouldn't impact their customers beyond some rather transparent performance improvements. And such a thing would be done only if there was profit in it that would exceed the R&D costs -- put another way, if you're a company considering this, you could either buy more and bigger hardware, or invest in unikernel... former is far less risk. Which is why I think where we'd see action in this space would be from very specific performance oriented spaces. E.g. super low latency transaction processing for e.g. telecomes, finance / high-frequency-trading, ad-tech RTB, games ... rather than say large scale analytic workloads.

bluepizza 3y ago

Because you can replace a group of unikernel VMs with an operating system and processes, and rely on those abstractions instead. Unikernels haven't caught on because the use cases where they are the most adequate solution are limited.

crabbone 3y ago
That's not true. Unikernels are a way to do the same thing you do with your "process abstraction", but better: using less resources, minimizing the number of failure points, improving speed of deployment, minimizing dependencies which also allows for faster iteration when developing software. The reason they didn't catch on is because overwhelming majority of programmers aren't inventors and aren't knowledgeable enough about the system aspect of their environment -- they need tools, frameworks, bundles of documentation and support forums to get anywhere. Now, all of the aids listed above can either be created by a dedicated group of enthusiasts or by large corporations investing into such a process. Large corporations have little interest in software qualities mentioned above because the "good enough" containers are capable of solving their business problems. So... we are left with enthusiasts, who are few and whose work isn't being integrated into "mainstream" development process. Will these enthusiasts eventually gain enough momentum to make it really attractive for the whole industry to switch, or with it be another IPv4 vs IPv6 -- time will show.
- immibis 3y ago
  In what way does a hypervisor and unikernels use less resources, reduce the number of failure points, improve deployment speed or reduce dependencies compared to an operating system (no hypervisor) and processes?
  - eyberg 3y ago
    If you are comparing a bare metal system, indeed you might not see a lot of benefits. However, there is a basic assumption that these workloads are running in the cloud which means they are already virtualized and then when you compare against a ubuntu/debian/etc. vm they get their benefits.

pjmlp 3y ago

They have caught on, that is basically in a way what serveless do. I don't care if the language runtime runs bare metal, on a container, or inside some OS VM.

edgyquant 3y ago
That isn’t what serverless does. Serverless tends to run on Linux but can be any OS.
- pjmlp 3y ago
  Depends on the cloud vendor, and on AWS, firecracker VM is close enough to the original idea.

crabbone 3y ago

Because it's hard to find programmers who can do it. Overwhelming majority of programming today is happening in userspace. And even there, it's typical to be far removed from system interfaces. Popular language runtimes s.a. Python or Java or JavaScript create a layer of obfuscation on top of system interfaces. Programmers who could realistically pull this out are either in embedded field or in OS / system field. OS / system is very tiny compared to everything else, but even with embedded being larger, the problem is also the lack of expertise necessary to tie the knowledge necessary to deal with unikernels with knowledge necessary to develop enterprise or user-facing applications. Another aspect is the lack of support from large cloud vendors. In principle, you could create RMIs (AWS EC2 VM images) from scratch, but it will be hard to integrate them with the rest of the cloud infrastructure. Popular tools that create RMIs aren't designed to deal with unikernel applications. They often assume that UNIX shell is available or that cloud-init is available etc. It's even worse with VHD. I don't know what the state of events is in GCP, but wouldn't expect that to be any different. Yet another aspect is the lack of instrumentation for debugging. A lot of common instrumentation assumes or relies on stuff like UNIX shell or presence of standard UNIX utilities. This is even more noticeable in automation / CI side. Containers are a lot easier because they rely on existing, however bloated and broken, infrastructure. They aren't better, and the way forward for containers promises no increase in quality... but it's the easy way "forward" (in temporal sense), and that's why it's being taken.

eyberg 3y ago
Definitely agree with the top part, however, I should note that, ops, the tool's, whole existence is to create disk images and upload them to any cloud, any hypervisor. In particular, both https://ops.city https://ops.city && https://nanos.org https://nanos.org are Go unikernels running on GCP and their deploys take just a few seconds to push out. AWS can be even faster cause we skip the s3 upload part. We also have lots of people using Azure which would be utilizing vhdx.

eyberg 3y ago

So my opinion, having worked in the space for years and the author of this post, is that the vast majority of people (engineers included) underestimate the sheer amount of work building a kernel that can run lots of applications on various clouds takes. Linux has been in development for ~30 years, Unix for 50. That is to say it isn't necessarily lack of use-cases - it is ensuring you have enough support built for said use-cases when some user wants to use it. I'll give an example. Years ago people would say something like "I see you have GCP support - that's awesome - what about Azure?" Back then I'd say something like, well, give us a month cause we have to write new disk/network drivers but yes we can make that happen - a month is too long for that person though and so you immediately lose that person as a user/potential customer and have to wait until they come back. (We have great azure support now fyi.) Now take that same example and add it to every language, every cloud, etc. It's a ton of work.

imiric 3y ago
Thanks for your perspective, and kudos on the article, impressive work! So it is a matter of lack of support and general tooling around it. Anyone interested in the technology needs to write support for each individual tech stack and infrastructure to run it. The amount of complex work and the scarcity of people knowledgeable enough to do it is a huge hurdle. Whereas Docker came around at a time when the low-level building blocks were already in place in Linux. Containers were already widely used in the industry, but each company had their own tooling around them. Docker made this technology much easier to use, which in turn helped define the standards that we use today. So I'm still hopeful that something similar will happen eventually with unikernels. There's been a lot of work in improving the tooling around it, as your company has done, adding support for various tech stacks, optimizing hypervisors to run them efficiently, etc., so hopefully in a few years a tool that greatly improves the UX will be built, and an industry standard will emerge. I think open source is a crucial part for this to happen. BTW, Nanos looks very cool!

toast0 3y ago

I've been messing around with a hobby OS that's adjacent to unikernels for a while now. Once I'm done enough, I plan to benchmark running my application+OS vs running my application as PID 1 in a mainstream OS. I imagine, if I'm lucky, performance will be similar, although chances are, the mainstream OS will be significantly better, and development time for application as PID 1 will be less than a month vs I've been tooling around for 3 years at this point (only really focused for a few months, though) Most applications lack the introspection to be really operable in a unikernel situation too, so then you're looking at unikernel with extras for administration vs minimal mainstream OS with a small set of administrative tools, and that's an easy win for the mainstream OS.

immibis 3y ago

Unikernels have caught on, in the form of processes. You have to run your unikernel in a hypervisor anyway, so why not give the hypervisor a bunch of features and then it's an operating system with a weird hardware-emulating interface. De-weird the interface, and then you have a normal operating system.

eyberg 3y ago
The big difference is the hypervisor in this case is the public cloud which is not just multi-tenant but massively multi-tenant. That is the host and vmware and aws and everyone else did a really excellent job of abstracting the host away to the extent that most developers don't even know what a server looks like anymore. Unikernels take the same approach but abstract the guest away. The abstraction is to treat each vm as an individual application (since we're treating the cloud as the operating system) versus trying to micro-manage hundreds or thousands of virtual operating systems inside the hypervisor.