5 ms·

I am a kernel developer working on IncludeOS. Take everything I say with a grain of salt as this is just my thoughts at 9 in the morning in written form. 1. I

I am a kernel developer working on IncludeOS. Take everything I say with a grain of salt as this is just my thoughts at 9 in the morning in written form. 1. I can build a bootable binary without the full TCP and UDP, shaving off 60% of the whole IP stack. LTO removes a large amount of code as well. It makes all the difference. 2. Performance comes from building a fixed binary (all memory addresses are constant), as well as using LTO and C++. IncludeOS is not at its most performant because we haven't been working on performance. Project is still young. We should be faster overall compared to just about anything that isn't compiling to a fixed binary layout. 3. Isolation can come in many forms. On traditional hypervisors you have the same old same old. While on a newer type of hypervisor you can do just about anything from avoiding VMEXITs altogether to removing the ability to even talk to the hypervisor after configuration stage. If the hypervisor is just a thin configuration program... 4. Yes doubly so if your OS uses threads as well, and the schedulers are almost always beating differently on the hypervisor and the guest. This causes even more trashing than normal. In any case, IncludeOS is not using threads at all. It is fully async and matches the hardware really well. The bottleneck is the multiplexing ability of the cloud platform itself. We can live with that. 5. I don't know much about containers other than that I avoid them. Containers share kernel. Sorry, I can't answer this.

bizzleDawg 8y ago
I was reading this thinking that security would be mentioned at some point in your response! Presumably a big benefit of smaller codebases is a more realistic possibility of auditing the security?
- fwsgonzo 8y ago
  Less code means less attack surface and easier auditability. VMs are traditionally very isolated, and guests running in special hypervisors like uKVM (and others) are even more so. The simpler hypercall API simplifies things. No shared kernel, like with containers. Immutability. I could write about this in much more detail, but then I would have to have help writing it. So I won't. I should also add that unikernels traditionally run with kernel privileges, but that is not a requirement. They can run just fine with some kind of modern hypervisor that sets everything up beforehand. In that case, the unikernel runs like a normal userspace program, just inside a VM. All communication would be with MMIO and MSRs that don't exit.
- johannes1234321 8y ago
  While the code base of a non-unikernel can be larger a mainstream project like Linux also attracts more reviewers, who spend more time reviewing the code (especially in the critical areas, some hardly used device driver might see less attention)
chrisper 8y ago
Any reasons you are avoiding containers?
- fwsgonzo 8y ago
  I am simply unfamiliar with them. Created one container once, and that's it. If I knew them better I would probably use them more.
progval 8y ago
> Performance comes from building a fixed binary (all memory addresses are constant), So, no (K)ASLR?
- fwsgonzo 8y ago
  Yes and no, if you really want it there is no reason you can't have it. I know that some OSes now do some link-stage procedures at boot, but we don't. Unikernels seldom need to reboot, and if they do, you might as well update it while preserving all the important bits (including TCP connections) by doing a live update. It's not exactly a simple solution as now you suddenly need a management solution. We also have the management part, but it's not a part of the open-source OS.
bullen 8y ago
So how can you build a parallel application on top of this that can share all cores of the CPU?
- fwsgonzo 8y ago
  I wrote an SMP API once, and while its fully implemented still in the OS, it won't work right because we changed C API to MUSL and it enables/disables locks internally by counting up/down number of threads. So, that means you cant use it at the moment. It looks like this: https://github.com/hioa-cs/IncludeOS/blob/dev/api/smp https://github.com/hioa-cs/IncludeOS/blob/dev/api/smp With that you can schedule work to CPUs directly.
- weberc2 8y ago
  The idea is that things you would normally get from an OS are now libraries that are compiled into your application. If you want a scheduled, you compile it in. If you want a TCP stack, you compile it in.
tinktank 8y ago
Thank you very much for your response. Some follow-up questions, in the same order as your responses: 1. So, in production, we do this all the time with the kernel. We have specific .config's that we use on a per-app basis so I'm not sure why this point is made against the kernel? Is it because in general people run what they're provided by the distro? I don't really understand the LTO argument as well, is it that the kernel is smaller? When I compared IncludeOS against the production kernels we run for memcache they were 1.5x bigger. In general, how much of an issue is kernel code size? a 50M kernel, while big pales in comparison to the amount of memory required for runtime operation. Have you done any measurements on runtime costs? 2. Do you have any benchmarks/evidence you could point me to? My benchmarks showed the exact opposite at a huge development cost (debugging unikernels is very painful - at least for me) 3. What do you mean by "same old same old"? The primitives for isolation are provided by the hardware with VT-x and VT-d and your point about "removing the ability to even talk to the hypervisor after configuration stage" is referring to VT-x from what I gather. All modern hypervisors use these primitives and in combination with VT-x and PCIe VFs provide the illusion of complete access to hardware so I think I'm missing your point. 4. I'm sorry but I find the argument that "threads automatically mean worse" hard to understand. The cost of a context switch is the cost of a context switch. The cost of a context switch between threads/processes on the same CPU is still much cheaper than the cost of a context switch between VMs on the same CPU. If a program requires threading then, with the Unikernel either I need an abstraction for threading (e.g. co-operative scheduling) that bypasses the cost of the switch _or_ I need to write the program differently. In your answer, you make the point you don't support threads which means that either I move to an event-driven model (which I can do in Linux too) until I run out of CPU or I switch between VM contexts, which in my understanding is much more costly. Do you mind elaborating further? 5. Do you have an official line on this? I would assume the people you're selling against will be deploying containers so it would be good to get a feel for unikernels on containers. Thanks again for your response. I found it most helpful.
- fwsgonzo 8y ago
  Hey, IncludeOS is mostly a work in progress, and so the answers will reflect that. 1 and 2: IncludeOS is a service where more stuff is compiled and inlined as a whole, compared to a normal kernel. The whole thing will be baked into one, in many cases inline. This gives the compiler a chance to simplify more things and optimize better, and while the size doesn't have to be smaller, we do have less code overall. Especially after the linker removes unused objects. IncludeOS is not being built with LTO at all presently (although it does have that ability on the Linux userspace platform), nor does it have any major performance work done other than making sure there are no big performance bugs around. I am working on this, but it's not a trivial thing. In a perfect world I would just enable LTO and it would just work, like I already do on the userspace Linux platform of IncludeOS. I have written a beginners-kernel project where I show that LTO reduces image size to 2/3 from 32kb to 11kb: https://github.com/fwsGonzo/barebones https://github.com/fwsGonzo/barebones 2. There is a paper showing this linked to in the repo itself. You can at least build the unikernel on linux userspace, so it shouldn't be too hard to debug it while you are creating it. Other than that, debugging is not easy in production just like you say. 3. Traditional hypervisors manage access to hardware all the time, while a new type of unikernel hypervisor wouldn't. It would just sit there after the configuration/startup phase and never be exited into. The number of VMEXITs something has to do for each packet is a good estimator for how performant something is. If the number is zero, then it must be performant. See bareflank and others. I can't elaborate further because it's an area of research, and I'm not an expert. Afaik there is nothing like this running in production yet, even if we can prove that it's possible to run exitless and with no kernel privileges! Maybe next year we will know more. 4. We absolutely support threads, its just not working right now. We also have an SMP API that is quite good, and the virtual CPUs are scheduled on the host side. We also have fibres. While we do have tests, I don't know the state of things. We have so many other things going trying to make the OS stable in a production environment. I couldn't find a single thing proving my assertion when I made a quick search, so I will just retract until I can find it again. As I'm sure you know unikernels tend to have a single address space, which removes the need to do a full context switch. 5. I don't think our customers will be deploying containers. Rather, I think they will be running hundreds of instances per physical host in their private clouds. Instances will act as Internet gateways, firewalls etc. Some network function. That is the stage we are at now. I really don't see containers being used after the binaries are built, but I could be wrong.

bizzleDawg 8y ago

I was reading this thinking that security would be mentioned at some point in your response! Presumably a big benefit of smaller codebases is a more realistic possibility of auditing the security?

fwsgonzo 8y ago
Less code means less attack surface and easier auditability. VMs are traditionally very isolated, and guests running in special hypervisors like uKVM (and others) are even more so. The simpler hypercall API simplifies things. No shared kernel, like with containers. Immutability. I could write about this in much more detail, but then I would have to have help writing it. So I won't. I should also add that unikernels traditionally run with kernel privileges, but that is not a requirement. They can run just fine with some kind of modern hypervisor that sets everything up beforehand. In that case, the unikernel runs like a normal userspace program, just inside a VM. All communication would be with MMIO and MSRs that don't exit.
johannes1234321 8y ago
While the code base of a non-unikernel can be larger a mainstream project like Linux also attracts more reviewers, who spend more time reviewing the code (especially in the critical areas, some hardly used device driver might see less attention)

chrisper 8y ago

Any reasons you are avoiding containers?

fwsgonzo 8y ago
I am simply unfamiliar with them. Created one container once, and that's it. If I knew them better I would probably use them more.

progval 8y ago

> Performance comes from building a fixed binary (all memory addresses are constant), So, no (K)ASLR?

fwsgonzo 8y ago
Yes and no, if you really want it there is no reason you can't have it. I know that some OSes now do some link-stage procedures at boot, but we don't. Unikernels seldom need to reboot, and if they do, you might as well update it while preserving all the important bits (including TCP connections) by doing a live update. It's not exactly a simple solution as now you suddenly need a management solution. We also have the management part, but it's not a part of the open-source OS.

bullen 8y ago

So how can you build a parallel application on top of this that can share all cores of the CPU?

fwsgonzo 8y ago
I wrote an SMP API once, and while its fully implemented still in the OS, it won't work right because we changed C API to MUSL and it enables/disables locks internally by counting up/down number of threads. So, that means you cant use it at the moment. It looks like this: https://github.com/hioa-cs/IncludeOS/blob/dev/api/smp https://github.com/hioa-cs/IncludeOS/blob/dev/api/smp With that you can schedule work to CPUs directly.
weberc2 8y ago
The idea is that things you would normally get from an OS are now libraries that are compiled into your application. If you want a scheduled, you compile it in. If you want a TCP stack, you compile it in.

tinktank 8y ago

Thank you very much for your response. Some follow-up questions, in the same order as your responses: 1. So, in production, we do this all the time with the kernel. We have specific .config's that we use on a per-app basis so I'm not sure why this point is made against the kernel? Is it because in general people run what they're provided by the distro? I don't really understand the LTO argument as well, is it that the kernel is smaller? When I compared IncludeOS against the production kernels we run for memcache they were 1.5x bigger. In general, how much of an issue is kernel code size? a 50M kernel, while big pales in comparison to the amount of memory required for runtime operation. Have you done any measurements on runtime costs? 2. Do you have any benchmarks/evidence you could point me to? My benchmarks showed the exact opposite at a huge development cost (debugging unikernels is very painful - at least for me) 3. What do you mean by "same old same old"? The primitives for isolation are provided by the hardware with VT-x and VT-d and your point about "removing the ability to even talk to the hypervisor after configuration stage" is referring to VT-x from what I gather. All modern hypervisors use these primitives and in combination with VT-x and PCIe VFs provide the illusion of complete access to hardware so I think I'm missing your point. 4. I'm sorry but I find the argument that "threads automatically mean worse" hard to understand. The cost of a context switch is the cost of a context switch. The cost of a context switch between threads/processes on the same CPU is still much cheaper than the cost of a context switch between VMs on the same CPU. If a program requires threading then, with the Unikernel either I need an abstraction for threading (e.g. co-operative scheduling) that bypasses the cost of the switch _or_ I need to write the program differently. In your answer, you make the point you don't support threads which means that either I move to an event-driven model (which I can do in Linux too) until I run out of CPU or I switch between VM contexts, which in my understanding is much more costly. Do you mind elaborating further? 5. Do you have an official line on this? I would assume the people you're selling against will be deploying containers so it would be good to get a feel for unikernels on containers. Thanks again for your response. I found it most helpful.

fwsgonzo 8y ago
Hey, IncludeOS is mostly a work in progress, and so the answers will reflect that. 1 and 2: IncludeOS is a service where more stuff is compiled and inlined as a whole, compared to a normal kernel. The whole thing will be baked into one, in many cases inline. This gives the compiler a chance to simplify more things and optimize better, and while the size doesn't have to be smaller, we do have less code overall. Especially after the linker removes unused objects. IncludeOS is not being built with LTO at all presently (although it does have that ability on the Linux userspace platform), nor does it have any major performance work done other than making sure there are no big performance bugs around. I am working on this, but it's not a trivial thing. In a perfect world I would just enable LTO and it would just work, like I already do on the userspace Linux platform of IncludeOS. I have written a beginners-kernel project where I show that LTO reduces image size to 2/3 from 32kb to 11kb: https://github.com/fwsGonzo/barebones https://github.com/fwsGonzo/barebones 2. There is a paper showing this linked to in the repo itself. You can at least build the unikernel on linux userspace, so it shouldn't be too hard to debug it while you are creating it. Other than that, debugging is not easy in production just like you say. 3. Traditional hypervisors manage access to hardware all the time, while a new type of unikernel hypervisor wouldn't. It would just sit there after the configuration/startup phase and never be exited into. The number of VMEXITs something has to do for each packet is a good estimator for how performant something is. If the number is zero, then it must be performant. See bareflank and others. I can't elaborate further because it's an area of research, and I'm not an expert. Afaik there is nothing like this running in production yet, even if we can prove that it's possible to run exitless and with no kernel privileges! Maybe next year we will know more. 4. We absolutely support threads, its just not working right now. We also have an SMP API that is quite good, and the virtual CPUs are scheduled on the host side. We also have fibres. While we do have tests, I don't know the state of things. We have so many other things going trying to make the OS stable in a production environment. I couldn't find a single thing proving my assertion when I made a quick search, so I will just retract until I can find it again. As I'm sure you know unikernels tend to have a single address space, which removes the need to do a full context switch. 5. I don't think our customers will be deploying containers. Rather, I think they will be running hundreds of instances per physical host in their private clouds. Instances will act as Internet gateways, firewalls etc. Some network function. That is the stage we are at now. I really don't see containers being used after the binaries are built, but I could be wrong.