4 ms·

From a first-principles perspective, the stack is as follows: VMs on warpbuild: baremetal > hypervisor for VM > runner workload Container: baremetal > (cloud V

by suryao 3y ago

From a first-principles perspective, the stack is as follows: VMs on warpbuild: baremetal > hypervisor for VM > runner workload Container: baremetal > (cloud VM [1]) > k8s worker node OS > containerd > container OS > runner workload This assumes containers are running on k8s, which is an okay assumption in this case. The perf penalty of using a VM is much lower. Note: if you are referring to the VM spin up time, then it is a whole other story and we have taken some pains to mitigate that to achieve comparable spinup time. [1] if using a non-bare metal ec2/gce instance, say

Cu3PO42 3y ago
I like your product and I think VMs are a fine choice, but this representation seems somewhat inaccurate. For one: don't the VMs you launch have their own OS? And also while containerd and potentially some Linux distro in a container are involved, they aren't really adding runtime overhead. containerd (via runc) instructs the Linux kernel to isolate the workload processes, but doesn't really sit in between them and the kernel. Further, the OS in the container doesn't have its own kernel and most of the time not even its own init. It's really just a set of libraries and binaries. I believe that you run workloads faster than competitors relying on containers, but it doesn't seem to me that containers are the problem. If you installed Linux on the same baremetal host, I'm convinced you can get that same performance in a container that you can get in a VM on that host.
- suryao 3y ago
  You're right - I missed that VM OS in my illustration up there. Benchmarks are hard to come by but iirc, each VMM adds a 3-5% overhead. The gatekeeping for permissions and sandboxing that is done at each level involves active compute cycles and that adds a little overhead. Something of the order of 10% may not be large but there's a difference.[1] Disk IO is another major factor btw, since virtual filesystems can be ... flaky. [1] https://www.vmware.com/pdf/hypervisor_performance.pdf https://www.vmware.com/pdf/hypervisor_performance.pdf

by suryao 3y ago

Cu3PO42 3y ago

I like your product and I think VMs are a fine choice, but this representation seems somewhat inaccurate. For one: don't the VMs you launch have their own OS? And also while containerd and potentially some Linux distro in a container are involved, they aren't really adding runtime overhead. containerd (via runc) instructs the Linux kernel to isolate the workload processes, but doesn't really sit in between them and the kernel. Further, the OS in the container doesn't have its own kernel and most of the time not even its own init. It's really just a set of libraries and binaries. I believe that you run workloads faster than competitors relying on containers, but it doesn't seem to me that containers are the problem. If you installed Linux on the same baremetal host, I'm convinced you can get that same performance in a container that you can get in a VM on that host.

suryao 3y ago
You're right - I missed that VM OS in my illustration up there. Benchmarks are hard to come by but iirc, each VMM adds a 3-5% overhead. The gatekeeping for permissions and sandboxing that is done at each level involves active compute cycles and that adds a little overhead. Something of the order of 10% may not be large but there's a difference.[1] Disk IO is another major factor btw, since virtual filesystems can be ... flaky. [1] https://www.vmware.com/pdf/hypervisor_performance.pdf https://www.vmware.com/pdf/hypervisor_performance.pdf