7 ms·

I'm not sure I get what the point of unikernels is given his explanation. GCP runs your VM as a Linux process with KVM. If you run a unikernel on that... that'

I'm not sure I get what the point of unikernels is given his explanation. GCP runs your VM as a Linux process with KVM. If you run a unikernel on that... that's roughly comparable to running your workload directly on the host - i.e. something more like App Engine. You still have Linux on top taking care of everything. You've just replaced the system call interface with a paravirtualized hardware interface. Which... is quite likely to be a bit slower if anything. You also have the overhead of two-stage paging (even for a unikernel, apps will want to use things like mmap(), which I assume means they can't run with guest paging off; and there's no paravirt API to mess with the VM memory layout to emulate this at the host level AFAIK) Of course you could in principle run a unikernel on bare metal, but I don't see that ever taking off. You need more software to manage a real machine, plus drivers. It's a lot easier to build your own OS when you're targeting fairly simple VM platforms with paravirtualized hardware. Bare metal isn't the same, and who wants to dedicate a whole machine to one process? Basically, this sounds to me like a less than ideal workaround to reduce VM overhead in existing cloud environments; not something that makes a whole lot of sense when you look at the big picture. The ASLR story is also silly. You can get the same effect with traditional fork/exec request handling (think CGI) on a normal OS, and of course that's going to be faster than booting the unikernel and the app for each request.

johncolanduoni 5y ago
I don’t think App Engine shares a Linux kernel between users. That would be a surprising level of confidence in lack of Linux system call vulnerabilities that no other cloud vendor seems to have (see AWS’s Firecracker VM or Google’s own gVisor). So at the end of the day, you’re still going to be dealing with a hypervisor. It’s also worth noting that at least AWS appears to hardware accelerate their paravirtualized interfaces so it’s not necessarily the case that IO operations are causing VM exits.
- marcan_42 5y ago
  App Engine absolutely shares a Linux between users. They have a heavily audited sandbox with multiple layers of defence. Of course customers don't get to call arbitrary system calls, but it's also not a VM.
  - naikrovek 5y ago
    I don't know if they use Firejail or not but that sounds a lot like Firejail.
  - johncolanduoni 5y ago
    GCP indicates that App Engine standard environment at least uses gVisor[1] (i.e. KVM with emulated Linux syscall interface). The docs also indicate you can run custom binaries on most of the standard runtime languages[2] (not to mention being able to ship Go binaries), so I don’t see how they could get around allowing system calls. App Engine must have used something other than gVisor originally since Go didn’t even exist when it launched, but I’m having trouble finding any concrete indication as to whether they shared kernels between multiple customers. [1]: https://cloud.google.com/blog/products/containers-kubernetes/how-gvisor-protects-google-cloud-services-from-cve-2020-14386 https://cloud.google.com/blog/products/containers-kubernetes... [2]: https://cloud.google.com/appengine/docs/the-appengine-environments https://cloud.google.com/appengine/docs/the-appengine-enviro...
    - monocasa 5y ago
      gVisor's kvm backend is still experimental AFAIK, in favor of the ptrace backend.
      - amscanne 5y ago
        I think the experimental warning was removed on the KVM platform a while ago?
    - marcan_42 5y ago
      They definitely shared kernels between customers (and with other internal stuff). I don't know if they're using the KVM backend now, but if they are, that's definitely more recent. There was no virtualization involved when it all started. Source: I'm a xoogler.
awild 5y ago
> Bare metal isn't the same, and who wants to dedicate a whole machine to one process? 1) I actually do that at work. A few bare metal machines each have to run just one large-ish process. If I had the time I'd really invest in a unikernel type setup just for the simplicity of it. 2) if you're using any cloud hosted vm to run your apps as the post describes you could benefit a lot from a unikernel setup. Haven't used aws et al much the last few years but at my last gig we hosted everything in their own vm,which would've been great to slim down hard.
WastingMyTime89 5y ago
The alternative is not between bare metal and paravirtualization. Hardware assisted virtualisation is a thing. The hypervisor can actually be very thin. I think what you are describing is actually an argument in favor of unikernel. The idea is to get the best of both world. You get the isolation guaranty provided by virtualisation without the cost of a full OS running on a full OS and with the added benefits of compiler optimisation running on the whole stake at once. Intellectually it makes a lot of sense. The question is: can it actually work without creating a lot of frictions during development and deliver these theoretical gains.
- marcan_42 5y ago
  The host interface is paravirtualized. Hardware assisted virtualization only covers the CPU in most cases. All the hardware and I/O is going to be using paravirtualized devices (virtio or similar). There is no inherent reason why that interface is going to be more secure than the syscall interface. If the goal is security, there are easier ways of hardening that without reaching for virtualization. Virtualization itself doesn't really buy you anything there; we only consider VMs more secure due to how existing implementations work, not due to anything intrinsic.
  - WastingMyTime89 5y ago
    > Hardware assisted virtualization only covers the CPU in most cases. All the hardware and I/O is going to be using paravirtualized devices (virtio or similar). Well, no. IOMMU is a thing. You can do passthrough of I/O devices is you need to. > There is no inherent reason why that interface is going to be more secure than the syscall interface. If the goal is security, there are easier ways of hardening that without reaching for virtualization. There is no question that running a software in a VM is safer than running it straight on the kernel. The isolation enforced is far stronger. You could legitimately ask if it’s safer than running it inside a container. That’s a good question especially now that the orchestration surrounding containers is better. A unikernel keeps the advantage of discarding plenty of attack vectors just by virtue of every unnecessary part of the kernel not being there however. > we only consider VMs more secure due to how existing implementations work, not due to anything intrinsic. It’s weird argument. We do indeed consider VMs more secure than running things directly on the kernel because the way they work intrinsically means are. You can certainly harden systems in different way and enforce isolation though different means but VMs have the advantage of being a fairly old and well known technology with good toolings around them, especially in the entreprise world.
Tyr42 5y ago
When I studied in Uni, it seemed like there were two different paths to reducing the number of copies and system calls in handling a network request. 1. Move the network stack to userspace. Then the code running in userland can get at the request without a copy. 2. Move the running code into the kernel. Then the code running in the kernel can get at the request without a copy. Unikernals are a way to do 2. People just really don't want to pay for that syscall boundary.
- tremon 5y ago
  I think the parent's point is that running a unikernel on paravirtualized hardware is actually neither: the entire VM runs in userspace as seen from the host, but you still have the data copy from the host's physical network interface to the paravirtual interface on the VM.
- marcan_42 5y ago
  But you're paying for the virtualization boundary, which is usually more expensive than a syscall boundary. You're still a VM guest running under the host kernel, which is a comparable situation to a userspace process.

johncolanduoni 5y ago

I don’t think App Engine shares a Linux kernel between users. That would be a surprising level of confidence in lack of Linux system call vulnerabilities that no other cloud vendor seems to have (see AWS’s Firecracker VM or Google’s own gVisor). So at the end of the day, you’re still going to be dealing with a hypervisor. It’s also worth noting that at least AWS appears to hardware accelerate their paravirtualized interfaces so it’s not necessarily the case that IO operations are causing VM exits.

marcan_42 5y ago
App Engine absolutely shares a Linux between users. They have a heavily audited sandbox with multiple layers of defence. Of course customers don't get to call arbitrary system calls, but it's also not a VM.
- naikrovek 5y ago
  I don't know if they use Firejail or not but that sounds a lot like Firejail.
- johncolanduoni 5y ago
  GCP indicates that App Engine standard environment at least uses gVisor[1] (i.e. KVM with emulated Linux syscall interface). The docs also indicate you can run custom binaries on most of the standard runtime languages[2] (not to mention being able to ship Go binaries), so I don’t see how they could get around allowing system calls. App Engine must have used something other than gVisor originally since Go didn’t even exist when it launched, but I’m having trouble finding any concrete indication as to whether they shared kernels between multiple customers. [1]: https://cloud.google.com/blog/products/containers-kubernetes/how-gvisor-protects-google-cloud-services-from-cve-2020-14386 https://cloud.google.com/blog/products/containers-kubernetes... [2]: https://cloud.google.com/appengine/docs/the-appengine-environments https://cloud.google.com/appengine/docs/the-appengine-enviro...
  - monocasa 5y ago
    gVisor's kvm backend is still experimental AFAIK, in favor of the ptrace backend.
    - amscanne 5y ago
      I think the experimental warning was removed on the KVM platform a while ago?
  - marcan_42 5y ago
    They definitely shared kernels between customers (and with other internal stuff). I don't know if they're using the KVM backend now, but if they are, that's definitely more recent. There was no virtualization involved when it all started. Source: I'm a xoogler.

awild 5y ago

> Bare metal isn't the same, and who wants to dedicate a whole machine to one process? 1) I actually do that at work. A few bare metal machines each have to run just one large-ish process. If I had the time I'd really invest in a unikernel type setup just for the simplicity of it. 2) if you're using any cloud hosted vm to run your apps as the post describes you could benefit a lot from a unikernel setup. Haven't used aws et al much the last few years but at my last gig we hosted everything in their own vm,which would've been great to slim down hard.

WastingMyTime89 5y ago

The alternative is not between bare metal and paravirtualization. Hardware assisted virtualisation is a thing. The hypervisor can actually be very thin. I think what you are describing is actually an argument in favor of unikernel. The idea is to get the best of both world. You get the isolation guaranty provided by virtualisation without the cost of a full OS running on a full OS and with the added benefits of compiler optimisation running on the whole stake at once. Intellectually it makes a lot of sense. The question is: can it actually work without creating a lot of frictions during development and deliver these theoretical gains.

marcan_42 5y ago
The host interface is paravirtualized. Hardware assisted virtualization only covers the CPU in most cases. All the hardware and I/O is going to be using paravirtualized devices (virtio or similar). There is no inherent reason why that interface is going to be more secure than the syscall interface. If the goal is security, there are easier ways of hardening that without reaching for virtualization. Virtualization itself doesn't really buy you anything there; we only consider VMs more secure due to how existing implementations work, not due to anything intrinsic.
- WastingMyTime89 5y ago
  > Hardware assisted virtualization only covers the CPU in most cases. All the hardware and I/O is going to be using paravirtualized devices (virtio or similar). Well, no. IOMMU is a thing. You can do passthrough of I/O devices is you need to. > There is no inherent reason why that interface is going to be more secure than the syscall interface. If the goal is security, there are easier ways of hardening that without reaching for virtualization. There is no question that running a software in a VM is safer than running it straight on the kernel. The isolation enforced is far stronger. You could legitimately ask if it’s safer than running it inside a container. That’s a good question especially now that the orchestration surrounding containers is better. A unikernel keeps the advantage of discarding plenty of attack vectors just by virtue of every unnecessary part of the kernel not being there however. > we only consider VMs more secure due to how existing implementations work, not due to anything intrinsic. It’s weird argument. We do indeed consider VMs more secure than running things directly on the kernel because the way they work intrinsically means are. You can certainly harden systems in different way and enforce isolation though different means but VMs have the advantage of being a fairly old and well known technology with good toolings around them, especially in the entreprise world.

Tyr42 5y ago

When I studied in Uni, it seemed like there were two different paths to reducing the number of copies and system calls in handling a network request. 1. Move the network stack to userspace. Then the code running in userland can get at the request without a copy. 2. Move the running code into the kernel. Then the code running in the kernel can get at the request without a copy. Unikernals are a way to do 2. People just really don't want to pay for that syscall boundary.

tremon 5y ago
I think the parent's point is that running a unikernel on paravirtualized hardware is actually neither: the entire VM runs in userspace as seen from the host, but you still have the data copy from the host's physical network interface to the paravirtual interface on the VM.
marcan_42 5y ago
But you're paying for the virtualization boundary, which is usually more expensive than a syscall boundary. You're still a VM guest running under the host kernel, which is a comparable situation to a userspace process.