5 ms·

I'm grown increasingly dubious of unikernels; cutting out a layer of indirection by running the target process in kernel space seemed like a good idea -- as lon

I'm grown increasingly dubious of unikernels; cutting out a layer of indirection by running the target process in kernel space seemed like a good idea -- as long as you didn't need, you know, multiple processes -- but it looks like the performance gains just aren't there. Unikernels can boot incredibly fast. Compared to a normal virtual machine that can take seconds to boot unikernels can boot in 50ms (quote from a white paper on the nanovms website) Considering that unikernels, by definition, don't have any userland, you really need to compare the unikernel boot time to the "time to reach init" boot time on a traditional OS -- which is under 100 ms with a Linux kernel, and down to 20 ms with FreeBSD now. A unikernel boot time of 50 ms really isn't anything special.

KRAKRISMOTT 3y ago
They are great for scale to zero use cases (cloud hosting serverless firecrackers). With KVM rapid booting you want to remove as much overhead as possible, it keeps things cheap. The ultimate endgame would be to have a cold-start time almost equivalent to a prewarmed in-memory image using current technology. Of course, this is just one contender among many similar tech. If WebAssembly truly takes off, then unikernels would become irrelevant.
- eyberg 3y ago
  Wasm and unikernels are more complimentary than competitive. You could actually run wasm in something like wasmer or wasmtime wrapped inside Nanos wrapped inside Firecracker if you wanted to. I think as you point out where wasm can shine are the function-as-a-service platforms, although the memory safety issues would probably need to be addressed at some point (writing to 0x0, no ro mem, etc.).
  - KRAKRISMOTT 3y ago
    The thing is Wasm already provides sandboxing, so you don't really need any additional KVM sandboxing, unless your hostile code threat model goes beyond even what typical cloud providers face. The Linux host for most wasm environments is only a single instance, often already running on bare metal, so there is no significant benefit to stripping it down to a unikernel.
eyberg 3y ago
There are definitely other projects that are <10ms but I'm not convinced that a fast boot time is something worth spending a ton of time on. Many languages and frameworks take seconds to boot regardless (jvm, python ml frameworks, rails, etc.) I'll also say that if you are setting up your own bridge and using something like dhcp nanos actually boots faster than the response can come in sometimes so there' s a flag you have to set. I think the 'fast boot time' usecases are a relatively small set. Having said that there are many many other performance considerations besides fast boot time such as throughput, or request/second or random io read/writes/etc. Many web languages are single-process/single thread (eg: all interpreted ones) and since that is the vast majority of our users it isn't a problem. Threads are great if you have access to them and if not you scale horizontally (just like the interpreters do today). Multiple processes as an architectural design outside of scaling (and numerous issues when it actually does come to scaling) is something the industry really needs to reckon with.
sacnoradhq 3y ago
Unikernels are the emperor's new clothes: they lack the stack of services and capabilities delivered to normal system images necessary for real production use: firewalls, security, monitoring, performance measurement, backup, auditing, and fs ACLs. To skip them is to put blinders on and walk around without clothes on.
- eyberg 3y ago
  * firewalls - nanos.org is a go unikernel running on GCP and we punch holes for it to talk on https * security - nanos supports much of the same security you'll see on linux but provides more: aslr, no stack/heap exec, rodata no exec, text no write, no null mapping; virtio-rng (for some clouds where it is supported - not everywhere it is), pledge, unveil, honestly - there's a lot here * monitoring - plenty of apm vendors work out of the box but also things like cloudwatch, and our own custom service as well * performance measurement - we have things like ftrace and many other tools * backup - pretty easy to clone vms * auditing - glad you pointed this out as this becomes much much easier; we actually analyzed a bunch of STIGs and measured the reduction for each STIG as compared to nanos - scroll down to the page here: https://nanovms.com/security https://nanovms.com/security - essentially if you are in a regulated industry like finance, health or defense this is a major benefit * fs ACLs - we have unveil support and many other nanos specific things
rwmj 3y ago
"Not all Unikernels". UKL is Linux and has a userland if you want to use it. https://github.com/unikernelLinux/ukl https://github.com/unikernelLinux/ukl And it supports vsock already. I don't need to check because it's just Linux so it supports everything Linux supports.
lathiat 3y ago
Related: https://twitter.com/cperciva/status/1659059445685420032/photo/1 https://twitter.com/cperciva/status/1659059445685420032/phot... Progress on speeding up the FreeBSD boot over the years: FreeBSD 11.1-RELEASE on EC2: 28208 ms. FreeBSD 13.1-RELEASE on EC2: 9698 ms. FreeBSD 14.0-CURRENT on Firecracker: 620 ms.

KRAKRISMOTT 3y ago

They are great for scale to zero use cases (cloud hosting serverless firecrackers). With KVM rapid booting you want to remove as much overhead as possible, it keeps things cheap. The ultimate endgame would be to have a cold-start time almost equivalent to a prewarmed in-memory image using current technology. Of course, this is just one contender among many similar tech. If WebAssembly truly takes off, then unikernels would become irrelevant.

eyberg 3y ago
Wasm and unikernels are more complimentary than competitive. You could actually run wasm in something like wasmer or wasmtime wrapped inside Nanos wrapped inside Firecracker if you wanted to. I think as you point out where wasm can shine are the function-as-a-service platforms, although the memory safety issues would probably need to be addressed at some point (writing to 0x0, no ro mem, etc.).
- KRAKRISMOTT 3y ago
  The thing is Wasm already provides sandboxing, so you don't really need any additional KVM sandboxing, unless your hostile code threat model goes beyond even what typical cloud providers face. The Linux host for most wasm environments is only a single instance, often already running on bare metal, so there is no significant benefit to stripping it down to a unikernel.

eyberg 3y ago

There are definitely other projects that are <10ms but I'm not convinced that a fast boot time is something worth spending a ton of time on. Many languages and frameworks take seconds to boot regardless (jvm, python ml frameworks, rails, etc.) I'll also say that if you are setting up your own bridge and using something like dhcp nanos actually boots faster than the response can come in sometimes so there' s a flag you have to set. I think the 'fast boot time' usecases are a relatively small set. Having said that there are many many other performance considerations besides fast boot time such as throughput, or request/second or random io read/writes/etc. Many web languages are single-process/single thread (eg: all interpreted ones) and since that is the vast majority of our users it isn't a problem. Threads are great if you have access to them and if not you scale horizontally (just like the interpreters do today). Multiple processes as an architectural design outside of scaling (and numerous issues when it actually does come to scaling) is something the industry really needs to reckon with.

sacnoradhq 3y ago

Unikernels are the emperor's new clothes: they lack the stack of services and capabilities delivered to normal system images necessary for real production use: firewalls, security, monitoring, performance measurement, backup, auditing, and fs ACLs. To skip them is to put blinders on and walk around without clothes on.

eyberg 3y ago
* firewalls - nanos.org is a go unikernel running on GCP and we punch holes for it to talk on https * security - nanos supports much of the same security you'll see on linux but provides more: aslr, no stack/heap exec, rodata no exec, text no write, no null mapping; virtio-rng (for some clouds where it is supported - not everywhere it is), pledge, unveil, honestly - there's a lot here * monitoring - plenty of apm vendors work out of the box but also things like cloudwatch, and our own custom service as well * performance measurement - we have things like ftrace and many other tools * backup - pretty easy to clone vms * auditing - glad you pointed this out as this becomes much much easier; we actually analyzed a bunch of STIGs and measured the reduction for each STIG as compared to nanos - scroll down to the page here: https://nanovms.com/security https://nanovms.com/security - essentially if you are in a regulated industry like finance, health or defense this is a major benefit * fs ACLs - we have unveil support and many other nanos specific things

rwmj 3y ago

"Not all Unikernels". UKL is Linux and has a userland if you want to use it. https://github.com/unikernelLinux/ukl https://github.com/unikernelLinux/ukl And it supports vsock already. I don't need to check because it's just Linux so it supports everything Linux supports.

lathiat 3y ago

Related: https://twitter.com/cperciva/status/1659059445685420032/photo/1 https://twitter.com/cperciva/status/1659059445685420032/phot... Progress on speeding up the FreeBSD boot over the years: FreeBSD 11.1-RELEASE on EC2: 28208 ms. FreeBSD 13.1-RELEASE on EC2: 9698 ms. FreeBSD 14.0-CURRENT on Firecracker: 620 ms.