4 ms·

I've recently become thoroughly convinced of the merits of consolidating onto as few machines (physical or virtual) as possible. One reason is that I recently c

I've recently become thoroughly convinced of the merits of consolidating onto as few machines (physical or virtual) as possible. One reason is that I recently consolidated some of my company's infrastructure onto a single bare-metal server to reduce costs. And then in the middle of that, this post came out: https://rachelbythebay.com/w/2022/01/27/scale/ https://rachelbythebay.com/w/2022/01/27/scale/ It seems to me that running lots of small VMs with unikernels is inherently wasteful compared to running many processes on a single machine with a shared kernel that can make optimal use of the machine's resources. Sure, the unikernel-based VMs can be smaller than equivalent Linux VMs, but one still has to allocate a fixed amount of RAM, storage, and (for public cloud platforms) CPU to each VM. We inevitably add some padding to those allocations to ensure that we have headroom, and the total probably adds up to more than we would need to allocate to a single machine (physical or virtual) running all of those processes on a single kernel. And on public cloud platforms, we have to pay for those padded resource allocations. I've certainly done deployments with lots of small Linux VMs in the past; in my recent migration process, I was replacing such a setup with one big box. Creating lots of small VMs is certainly a convenient and robust way to independently deploy and update several components. But it's obviously not the only way. The home page of the forthcoming Unikraft Cloud service says, "The cloud is essential to your business but you know you are overpaying." But I think a better answer is to consolidate onto a few big VMs, using container orchestration to keep deployment manageable.

nderjung 5y ago
It really depends on your usecase here. If the many application processes rely on the same OS libraries, versioned language runtimes (e.g. same python version), kernel version, etc. AND you trust the OS then it may make sense. However, unikernels offer the lightweightness of a container process with the security of a VM. In addition to this, memory ballooning[0][1] and other resource-elastic features are available to VMs too: allowng you to under-provision them and then later increase resources when load demands it. Unikraft unikernels can also be managed using the same orchestration tools as containers, check out the talk at CNCF[2]. [0]: https://github.com/unikraft/unikraft/pull/219 https://github.com/unikraft/unikraft/pull/219 [1]: https://pmhahn.github.io/virtio-balloon/ https://pmhahn.github.io/virtio-balloon/ [2]: https://www.youtube.com/watch?v=cV-xawN9_cg https://www.youtube.com/watch?v=cV-xawN9_cg
- mwcampbell 5y ago
  > memory ballooning[0][1] and other resource-elastic features are available to VMs too If I'm using a public cloud platform's hypervisor, those features may benefit the cloud provider, but not me. Or are you targeting users running a hypervisor on bare metal?
  - nderjung 5y ago
    I think it can benefit both parties -- underprovisioning for a smaller bill and then increasing when demands need it so as to prevent degregation in QoS. That said, if load increases exponentially high it can induce high costs. Some cloud providers do not offer memory (or just any resource) ballooning, like AWS, so there you will experience the problem you have discussed about over-provisioning. However, we aim to alleviate some of these problems with the uniqueness of Unikernels with our soon-to-be-released Cloud Platform at https://unikraft.io https://unikraft.io. Features like memory ballooning (with hard upper limits) and other features like deep in-kernel monitoring to understand application performance.
fire 5y ago
Sort of sounds like this is a good tool to pair with things like firecracker? The goal with most of these ( aside security ) is a better ability to (bin?) pack these small "blocks" into a larger "box". Tradeoffs between actual usage costs and devops time cost, imo
- felipehuici 5y ago
  Yes, in fact we have some early support for Firecracker, where we can at least boot some basic Unikraft images with it (e.g., see page 10 of this paper[0], FIgure 10, where we get the shortest boot times with Firecracker). We're still missing networking supporto on FC, which we're working on. [0] https://dl.acm.org/doi/pdf/10.1145/3447786.3456248 https://dl.acm.org/doi/pdf/10.1145/3447786.3456248
felipehuici 5y ago
Definitely agree, in fact we tend to use the term "massive consolidation", where we run thousands of VMs on a single server, thus saving costs. Unikraft unikernels are a perfect fit for this since they consume little memory and boot relatively fast. In early work [0] we were booting as many as 8000 hello world VMs/unikernels on the same server on the Xen hypervisor. More recently we have been booting 1K NGINX Unikraft images on a single server. [0] https://dl.acm.org/doi/10.1145/3132747.3132763 https://dl.acm.org/doi/10.1145/3132747.3132763

nderjung 5y ago

It really depends on your usecase here. If the many application processes rely on the same OS libraries, versioned language runtimes (e.g. same python version), kernel version, etc. AND you trust the OS then it may make sense. However, unikernels offer the lightweightness of a container process with the security of a VM. In addition to this, memory ballooning[0][1] and other resource-elastic features are available to VMs too: allowng you to under-provision them and then later increase resources when load demands it. Unikraft unikernels can also be managed using the same orchestration tools as containers, check out the talk at CNCF[2]. [0]: https://github.com/unikraft/unikraft/pull/219 https://github.com/unikraft/unikraft/pull/219 [1]: https://pmhahn.github.io/virtio-balloon/ https://pmhahn.github.io/virtio-balloon/ [2]: https://www.youtube.com/watch?v=cV-xawN9_cg https://www.youtube.com/watch?v=cV-xawN9_cg

mwcampbell 5y ago
> memory ballooning[0][1] and other resource-elastic features are available to VMs too If I'm using a public cloud platform's hypervisor, those features may benefit the cloud provider, but not me. Or are you targeting users running a hypervisor on bare metal?
- nderjung 5y ago
  I think it can benefit both parties -- underprovisioning for a smaller bill and then increasing when demands need it so as to prevent degregation in QoS. That said, if load increases exponentially high it can induce high costs. Some cloud providers do not offer memory (or just any resource) ballooning, like AWS, so there you will experience the problem you have discussed about over-provisioning. However, we aim to alleviate some of these problems with the uniqueness of Unikernels with our soon-to-be-released Cloud Platform at https://unikraft.io https://unikraft.io. Features like memory ballooning (with hard upper limits) and other features like deep in-kernel monitoring to understand application performance.

fire 5y ago

Sort of sounds like this is a good tool to pair with things like firecracker? The goal with most of these ( aside security ) is a better ability to (bin?) pack these small "blocks" into a larger "box". Tradeoffs between actual usage costs and devops time cost, imo

felipehuici 5y ago
Yes, in fact we have some early support for Firecracker, where we can at least boot some basic Unikraft images with it (e.g., see page 10 of this paper[0], FIgure 10, where we get the shortest boot times with Firecracker). We're still missing networking supporto on FC, which we're working on. [0] https://dl.acm.org/doi/pdf/10.1145/3447786.3456248 https://dl.acm.org/doi/pdf/10.1145/3447786.3456248

felipehuici 5y ago

Definitely agree, in fact we tend to use the term "massive consolidation", where we run thousands of VMs on a single server, thus saving costs. Unikraft unikernels are a perfect fit for this since they consume little memory and boot relatively fast. In early work [0] we were booting as many as 8000 hello world VMs/unikernels on the same server on the Xen hypervisor. More recently we have been booting 1K NGINX Unikraft images on a single server. [0] https://dl.acm.org/doi/10.1145/3132747.3132763 https://dl.acm.org/doi/10.1145/3132747.3132763