4 ms·

I used to be in the unikernel camp of "this is the next step in virtualization tech", but having played around with both containers and unikernels, and now deve

by hacknat 11y ago

I used to be in the unikernel camp of "this is the next step in virtualization tech", but having played around with both containers and unikernels, and now developing with containers, I think unikernels are going to occupy only a very niche space. There are two touted benefits of unikernels, performance and security. Performance turns out to be a red herring, as the overhead of an OS vs a Hypervisor turns out to be roughly equivalent (with the OS actually winning in some use cases). Security is definitely an issue, but it's so abstract. My company is a compliance (a very specific industry's compliance) cloud provider and we have gone with Docker as we get to use the OS as our Hypervisor, which means it is much more extensible and, in our use case, secure as we are able to auto-encrypt all network traffic coming out of the hosts with a tap/tun virtual device. Two things need to happen to make unikernels attractive. A new Hypervisor needs to get made, one that is just as extensible as an OS around the isolated primitives. It should also have something extra too (like the ability to fine tune resource management better than an OS can). Secondly a user friendly mechanism like Docker needs to happen.

derefr 11y ago
> Performance turns out to be a red herring, as the overhead of an OS vs a Hypervisor turns out to be roughly equivalent That presumes you get to choose. When you have a hypervisor either way (as on IaaS), the only "choice" is between a "bare" hypervisor, or a hypervisor plus an OS doing containerization.
- hacknat 11y ago
  On AWS Linux is the dom0 OS on Xen (I don't know of any other CPs who are different, unless you get a dedicated host), so the same is true of unikernels right now as well, not of docker though, as they have a containers product now. So...
  - geofft 11y ago
    I'd hope that AWS' container product runs at least each customer's containers in a separate VM per customer, if not one VM per app. It is too easy for a malicious customer to break out of a Docker container (far easier than breaking out of Xen, and they take those security vulnerabilities seriously). The advantage here would be that Amazon could change their container product to allow only one layer of isolation (user code in a unikernel, inside Amazon-run Xen) instead of two (user code in Docker, inside Amazon-run Linux per customer, inside Amazon-run Xen).
    - kylequest 11y ago
      Yep, they do run each customer's containers in separate VMs. It's actually bring own VM kind of design :-) You configure your own EC2 instances, install ECS agents...
    - hacknat 11y ago
      I think this is a bit of an overstatement, breaking out of a docker container is not "easy", especially if you add other security products on top. Then there is the extensibility of the OS to be used to make things even more secure. If you look at security through the narrow lens of "being able to break out of your environment" then yes, Hypervisors are more secure, but you have to look at more than just that. The OS allows you to make more than just the host secure, it allows you to make the network secure, ensure that all customers get encrypted disks, etc... Full disclosure, I work for a company doing this right now (Catalyze Inc.)
      - geofft 11y ago
        Yeah, I don't mean to say breaking out of Docker confinement is easy by any objective measure, just that it is much less hard than breaking out of Xen confinement, and even that seems to be a fair bit of concern for AWS already.
  - monocasa 11y ago
    But on Xen, it's not like to dom0 is required to be called in the dataplane. It's mainly there for config and emulation of devices that don't have SR-IOV.
  - SEJeff 11y ago
    From https://cloud.google.com/compute/docs/faq#whatis https://cloud.google.com/compute/docs/faq#whatis ... """ Google Compute Engine includes Linux and Windows based virtual machines running on KVM, local and durable storage options, and a simple REST based API for configuration and control. """ No Xen there
  - cthalupa 11y ago
    The dom0 is not the hypervisor. Guests are not running on top of a dom0 in Xen - the dom0 is a management domain that is a VM running on top of the hypervisor just like any other, just with some additional privileges and management functionality.
amirmc 11y ago
There are many other benefits to unikernels especially depending on which implementation you choose to go with. For example, MirageOS and Rumprun are good examples of clean-slate vs current systems. I'd recommend reading some of the articles at http://unikernel.org/resources http://unikernel.org/resources to get better view of this. As for 'a user friendly mechanism like Docker' ... well, I hope today's news convinces you that it's in progress.
- hacknat 11y ago
  I came down harder than I wanted to. An acquisition like this definitely moves the ball forward. I'm excited to see what comes of it!
hughw 11y ago
> "a user friendly mechanism like Docker needs to happen" Had you looked at OSv and Capstan[1]? It is just like Docker in that respect. [1] https://github.com/cloudius-systems/capstan/blob/master/README.md#usage https://github.com/cloudius-systems/capstan/blob/master/READ...
- hacknat 11y ago
  Very interesting. I will give this a look.
Pxtl 11y ago
Wouldn't the unikernel security benefit be the same for a lightweight OS where the libraries/services had been carefully pared down to the bare minimum for the application? It sounds like the main benefit to the Unikernel security is the implicit audit of all the services you want to roll into the kernel down to the bare minimum. I imagine that could be done with a more conventional architecture, it's just that nobody ever does.
- axelfontaine 11y ago
  We do. We generate ultra-minimal Linux-based images tailor-made for your JVM app and provide rapid local testing on VirtualBox and full zero-downtime blue/green deployment orchestration on AWS https://boxfuse.com https://boxfuse.com
- talex5 11y ago
  It depends on which lightweight OS and which unikernel. But e.g. a stripped down Linux will still have a huge amount of C. If you're going to write your kernel in something safer, then you might as well make a unikernel, rather than creating a kernel/userspace split. It's hard to see how you'd get a traditional OS stripped down anywhere close to e.g. the mirage-firewall unikernel (http://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/ http://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewal...)
  - EvanPlaice 11y ago
    Not necessarily. NodeOS cut out everything but the absolutely essential parts of the linux kernel. No c libraries, no c compiler, no POSIX utilities, no user space, etc. Instead, everything runs on V8 (ie which also takes care of sandboxing) and minimal tools were rewritten in pure javascript, incl a git clone tool. It's amazing how much you can cut out and still have a decent platform to build servers.
    - lmm 11y ago
      You're still running the linux scheduler in C, and you're still context switching for system calls. And can you run your whole stack as a user-mode program by changing one line in the build file the way you can with Mirage?
Symmetry 11y ago
There are certainly ways to drastically improve some sorts of performance but certainly not with a lot more work than just running a standard unikernel on Xen. https://arrakis.cs.washington.edu/ https://arrakis.cs.washington.edu/
av8or 11y ago
Security and performance of hypervisors will never be like physical, PDI can that is Storage agnostic can deliver both better and with full compliance. See Http://jentu-networks.com
av8or 11y ago
Security and performance of hypervisors will never be like physical, PDI can that is Storage agnostic can deliver both better and with full compliance. See Http://jentu-networks.com

by hacknat 11y ago

derefr 11y ago

> Performance turns out to be a red herring, as the overhead of an OS vs a Hypervisor turns out to be roughly equivalent That presumes you get to choose. When you have a hypervisor either way (as on IaaS), the only "choice" is between a "bare" hypervisor, or a hypervisor plus an OS doing containerization.

hacknat 11y ago
On AWS Linux is the dom0 OS on Xen (I don't know of any other CPs who are different, unless you get a dedicated host), so the same is true of unikernels right now as well, not of docker though, as they have a containers product now. So...
- geofft 11y ago
  I'd hope that AWS' container product runs at least each customer's containers in a separate VM per customer, if not one VM per app. It is too easy for a malicious customer to break out of a Docker container (far easier than breaking out of Xen, and they take those security vulnerabilities seriously). The advantage here would be that Amazon could change their container product to allow only one layer of isolation (user code in a unikernel, inside Amazon-run Xen) instead of two (user code in Docker, inside Amazon-run Linux per customer, inside Amazon-run Xen).
  - kylequest 11y ago
    Yep, they do run each customer's containers in separate VMs. It's actually bring own VM kind of design :-) You configure your own EC2 instances, install ECS agents...
  - hacknat 11y ago
    I think this is a bit of an overstatement, breaking out of a docker container is not "easy", especially if you add other security products on top. Then there is the extensibility of the OS to be used to make things even more secure. If you look at security through the narrow lens of "being able to break out of your environment" then yes, Hypervisors are more secure, but you have to look at more than just that. The OS allows you to make more than just the host secure, it allows you to make the network secure, ensure that all customers get encrypted disks, etc... Full disclosure, I work for a company doing this right now (Catalyze Inc.)
    - geofft 11y ago
      Yeah, I don't mean to say breaking out of Docker confinement is easy by any objective measure, just that it is much less hard than breaking out of Xen confinement, and even that seems to be a fair bit of concern for AWS already.
- monocasa 11y ago
  But on Xen, it's not like to dom0 is required to be called in the dataplane. It's mainly there for config and emulation of devices that don't have SR-IOV.
- SEJeff 11y ago
  From https://cloud.google.com/compute/docs/faq#whatis https://cloud.google.com/compute/docs/faq#whatis ... """ Google Compute Engine includes Linux and Windows based virtual machines running on KVM, local and durable storage options, and a simple REST based API for configuration and control. """ No Xen there
- cthalupa 11y ago
  The dom0 is not the hypervisor. Guests are not running on top of a dom0 in Xen - the dom0 is a management domain that is a VM running on top of the hypervisor just like any other, just with some additional privileges and management functionality.

amirmc 11y ago

There are many other benefits to unikernels especially depending on which implementation you choose to go with. For example, MirageOS and Rumprun are good examples of clean-slate vs current systems. I'd recommend reading some of the articles at http://unikernel.org/resources http://unikernel.org/resources to get better view of this. As for 'a user friendly mechanism like Docker' ... well, I hope today's news convinces you that it's in progress.

hacknat 11y ago
I came down harder than I wanted to. An acquisition like this definitely moves the ball forward. I'm excited to see what comes of it!

hughw 11y ago

> "a user friendly mechanism like Docker needs to happen" Had you looked at OSv and Capstan[1]? It is just like Docker in that respect. [1] https://github.com/cloudius-systems/capstan/blob/master/README.md#usage https://github.com/cloudius-systems/capstan/blob/master/READ...

hacknat 11y ago
Very interesting. I will give this a look.

Pxtl 11y ago

Wouldn't the unikernel security benefit be the same for a lightweight OS where the libraries/services had been carefully pared down to the bare minimum for the application? It sounds like the main benefit to the Unikernel security is the implicit audit of all the services you want to roll into the kernel down to the bare minimum. I imagine that could be done with a more conventional architecture, it's just that nobody ever does.

axelfontaine 11y ago
We do. We generate ultra-minimal Linux-based images tailor-made for your JVM app and provide rapid local testing on VirtualBox and full zero-downtime blue/green deployment orchestration on AWS https://boxfuse.com https://boxfuse.com
talex5 11y ago
It depends on which lightweight OS and which unikernel. But e.g. a stripped down Linux will still have a huge amount of C. If you're going to write your kernel in something safer, then you might as well make a unikernel, rather than creating a kernel/userspace split. It's hard to see how you'd get a traditional OS stripped down anywhere close to e.g. the mirage-firewall unikernel (http://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/ http://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewal...)
- EvanPlaice 11y ago
  Not necessarily. NodeOS cut out everything but the absolutely essential parts of the linux kernel. No c libraries, no c compiler, no POSIX utilities, no user space, etc. Instead, everything runs on V8 (ie which also takes care of sandboxing) and minimal tools were rewritten in pure javascript, incl a git clone tool. It's amazing how much you can cut out and still have a decent platform to build servers.
  - lmm 11y ago
    You're still running the linux scheduler in C, and you're still context switching for system calls. And can you run your whole stack as a user-mode program by changing one line in the build file the way you can with Mirage?

Symmetry 11y ago

There are certainly ways to drastically improve some sorts of performance but certainly not with a lot more work than just running a standard unikernel on Xen. https://arrakis.cs.washington.edu/ https://arrakis.cs.washington.edu/

av8or 11y ago

Security and performance of hypervisors will never be like physical, PDI can that is Storage agnostic can deliver both better and with full compliance. See Http://jentu-networks.com

av8or 11y ago

Security and performance of hypervisors will never be like physical, PDI can that is Storage agnostic can deliver both better and with full compliance. See Http://jentu-networks.com