5 ms·

I have never really got unikernels, maybe someone could educate me? - There is the argument of reduced attack surface/typechecked safety. I get that, but I do

I have never really got unikernels, maybe someone could educate me? - There is the argument of reduced attack surface/typechecked safety. I get that, but I don't see how this is different from compiling custom Linux/NetBSD kernels per VM? - There is the argument for increased performance. I don't get that at all -- I've done experiments on unikernel system for MirageOS vs IncludeOS vs Kernel on a web and memcache server and the performance results were worse (15%), 0.4% better respectively for MirageOS respectively but for a tonne more effort and really painful debugability. - There is the argument for isolation but I don't see how it's different because the biggest cause of churn is the hypervisor? My results with Xen 4.x didn't show any difference in isolation performance with both Include and Mirage. - There is the argument that OSes don't map to existing hardware as well, e.g. in the InfoQ talk in the comments the speaker talks about PCIe VFs for example. All of these VFs map to physical resources, and you end up doing a much worse job when you can't centrally manage these -- that's one of the main reasons SR-IOV hasn't been adopted (including the fact that migration is non-existent). Similarly with CPU hardware threads or NVMe device drivers -- they are abstractions provided for convenience and reducing the burden of porting existing drivers to work in hypervisor environments but surely the same issues that appear with VM contexts (hardware scheduling and isolation) still exist in unikernel environments? I feel I'm missing something in particular about this point as I see it made often and it doesn't make sense to me. - How do unikernels work in a world of containers? This has always puzzled me as I thought people moved away from VMs due to the heavyweight nature but of course the flip side is the underlying isolation abstractions are more lightweight -- so in essence are unikernels in containers user-level processes or are they replacing the kernel bits too?

fwsgonzo 8y ago
I am a kernel developer working on IncludeOS. Take everything I say with a grain of salt as this is just my thoughts at 9 in the morning in written form. 1. I can build a bootable binary without the full TCP and UDP, shaving off 60% of the whole IP stack. LTO removes a large amount of code as well. It makes all the difference. 2. Performance comes from building a fixed binary (all memory addresses are constant), as well as using LTO and C++. IncludeOS is not at its most performant because we haven't been working on performance. Project is still young. We should be faster overall compared to just about anything that isn't compiling to a fixed binary layout. 3. Isolation can come in many forms. On traditional hypervisors you have the same old same old. While on a newer type of hypervisor you can do just about anything from avoiding VMEXITs altogether to removing the ability to even talk to the hypervisor after configuration stage. If the hypervisor is just a thin configuration program... 4. Yes doubly so if your OS uses threads as well, and the schedulers are almost always beating differently on the hypervisor and the guest. This causes even more trashing than normal. In any case, IncludeOS is not using threads at all. It is fully async and matches the hardware really well. The bottleneck is the multiplexing ability of the cloud platform itself. We can live with that. 5. I don't know much about containers other than that I avoid them. Containers share kernel. Sorry, I can't answer this.
- bizzleDawg 8y ago
  I was reading this thinking that security would be mentioned at some point in your response! Presumably a big benefit of smaller codebases is a more realistic possibility of auditing the security?
  - fwsgonzo 8y ago
    Less code means less attack surface and easier auditability. VMs are traditionally very isolated, and guests running in special hypervisors like uKVM (and others) are even more so. The simpler hypercall API simplifies things. No shared kernel, like with containers. Immutability. I could write about this in much more detail, but then I would have to have help writing it. So I won't. I should also add that unikernels traditionally run with kernel privileges, but that is not a requirement. They can run just fine with some kind of modern hypervisor that sets everything up beforehand. In that case, the unikernel runs like a normal userspace program, just inside a VM. All communication would be with MMIO and MSRs that don't exit.
  - johannes1234321 8y ago
    While the code base of a non-unikernel can be larger a mainstream project like Linux also attracts more reviewers, who spend more time reviewing the code (especially in the critical areas, some hardly used device driver might see less attention)
- chrisper 8y ago
  Any reasons you are avoiding containers?
  - fwsgonzo 8y ago
    I am simply unfamiliar with them. Created one container once, and that's it. If I knew them better I would probably use them more.
- progval 8y ago
  > Performance comes from building a fixed binary (all memory addresses are constant), So, no (K)ASLR?

fwsgonzo 8y ago

I am a kernel developer working on IncludeOS. Take everything I say with a grain of salt as this is just my thoughts at 9 in the morning in written form. 1. I can build a bootable binary without the full TCP and UDP, shaving off 60% of the whole IP stack. LTO removes a large amount of code as well. It makes all the difference. 2. Performance comes from building a fixed binary (all memory addresses are constant), as well as using LTO and C++. IncludeOS is not at its most performant because we haven't been working on performance. Project is still young. We should be faster overall compared to just about anything that isn't compiling to a fixed binary layout. 3. Isolation can come in many forms. On traditional hypervisors you have the same old same old. While on a newer type of hypervisor you can do just about anything from avoiding VMEXITs altogether to removing the ability to even talk to the hypervisor after configuration stage. If the hypervisor is just a thin configuration program... 4. Yes doubly so if your OS uses threads as well, and the schedulers are almost always beating differently on the hypervisor and the guest. This causes even more trashing than normal. In any case, IncludeOS is not using threads at all. It is fully async and matches the hardware really well. The bottleneck is the multiplexing ability of the cloud platform itself. We can live with that. 5. I don't know much about containers other than that I avoid them. Containers share kernel. Sorry, I can't answer this.

bizzleDawg 8y ago
I was reading this thinking that security would be mentioned at some point in your response! Presumably a big benefit of smaller codebases is a more realistic possibility of auditing the security?
- fwsgonzo 8y ago
  Less code means less attack surface and easier auditability. VMs are traditionally very isolated, and guests running in special hypervisors like uKVM (and others) are even more so. The simpler hypercall API simplifies things. No shared kernel, like with containers. Immutability. I could write about this in much more detail, but then I would have to have help writing it. So I won't. I should also add that unikernels traditionally run with kernel privileges, but that is not a requirement. They can run just fine with some kind of modern hypervisor that sets everything up beforehand. In that case, the unikernel runs like a normal userspace program, just inside a VM. All communication would be with MMIO and MSRs that don't exit.
- johannes1234321 8y ago
  While the code base of a non-unikernel can be larger a mainstream project like Linux also attracts more reviewers, who spend more time reviewing the code (especially in the critical areas, some hardly used device driver might see less attention)
chrisper 8y ago
Any reasons you are avoiding containers?
- fwsgonzo 8y ago
  I am simply unfamiliar with them. Created one container once, and that's it. If I knew them better I would probably use them more.
progval 8y ago
> Performance comes from building a fixed binary (all memory addresses are constant), So, no (K)ASLR?