4 ms·

Why would you want to migrate a firecracker vm? They are supposed to be for small, short lived processes. Or in this case, Docker containers (which are likely t

Why would you want to migrate a firecracker vm? They are supposed to be for small, short lived processes. Or in this case, Docker containers (which are likely to be controlled by some kind of orchestration). If you want to be able to migrate a running VM, do you still really need it to start super fast? It’s going to be a long lived service (otherwise you wouldn’t migrate it), so a longer spin up time should be acceptable. If that’s the case, then you can use a different VM tech that supports migration (KVM/Xen/etc).

0xbadcafebee 4y ago
I think that is GP's point: Firecracker shouldn't be considered a general-purpose solution for more secure containerized workloads.
- tptacek 4y ago
  Can you try to make that a little more specific? What does "general-purpose" mean here? "Secure containerized workloads" tends to imply serverside. It's hard to think of things --- other than GPU access --- that Firecracker takes away here.
  - topspin 4y ago
    > It's hard to think of things No, it isn't. Investigate the K8S world and observe the backflips and somersaults being performed there to deal with networking and storage. Much of that is a consequence of people trying make containers do things that should be run in VMs. A full featured Linux kernel and certain key user space tools can obviate much of that nonsense with ease, deliver better performance, use less resources and be vastly less complex.
    - tptacek 4y ago
      If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one? Firecracker runs a full-featured Linux kernel. It's a hypervisor. It's not a kind of kernel.
      - topspin 4y ago
        > If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one? I already did that with live migration. But ok. Encrypted storage: https://github.com/firecracker-microvm/firecracker/issues/653 https://github.com/firecracker-microvm/firecracker/issues/65...: WONTFIX The answer given is appropriate for firecracker use cases but insufficient otherwise. I'm not anti-firecracker; it's the right choice for many things. Just not all things. The sort of VM I want orchestrated has encrypted (by contract) multi-pathed network block devices to encrypted storage volumes. 3-10 per tenant. This is trivial for a full-featured kernel; multi-path just works, encryption just works. VLAN: https://github.com/weaveworks/ignite/issues/810 https://github.com/weaveworks/ignite/issues/810: Open. Maybe one day. Again, trivial for a full-featured Linux kernel. Has been for ages. I think you're missing the point. It's not about what hypothetical thing firecracker can or can't do. It's about elevating VM orchestration to some degree of parity with what has been created for container orchestration. These VMs and their complex storage and networking requirements should be modeled as we model containers now; through an orchestration system that makes management easy and as foolproof as possible. The fact that firecracker isn't sufficient to be the Micro-VM of choice for this isn't relevant.
        tptacek 4y ago
        You can encrypt a drive from within Firecracker trivially. It's just Linux, and they're just block devices. You can do all the standard Linux tap interface networking with Firecracker; it just presents as virtio ethernet to the guest. This is the second time you've compared Firecracker to a "full-featured Linux kernel". Again: Firecracker is a hypervisor. It's not a kernel. It runs Linux kernels. "Full-featured" Linux kernels. Whatever kernel you compile.
        topspin 4y ago
        Well now you're arguing with the developers of both firecracker and ignite. Maybe go to the issues I linked and correct them.
topspin 4y ago
> Why would you want to migrate a firecracker vm? I wouldn't. I pointed out the lack of live migration as an of example why firecracker can't cut it as the basis of general purpose VM orchestration platform. > then you can use a different VM tech that supports migration (KVM/Xen/etc) "KVM/Zen/etc" are in the dark ages compared to the state of the art of Kubernetes container orchestration. The reason for that isn't technical; KVM is incredible and full-featured VMs conceivably could be orchestrated with the same fidelity as containers. The reason is business: the cloud behemoths invest only in their serverless business models; thus k8s, firecracker, etc. I contend there is an enormous market for orchestration of more complex workloads that VMs suit well.
regularfry 4y ago
> If you want to be able to migrate a running VM, do you still really need it to start super fast? Yes. "You" is two different people in this sentence. The person doing the migration is often not the person who owns whatever service is running in the VM. They have uncorrelated needs. As a platform operator, I want to migrate VMs to be able to balance load and evacuate dying hardware. As a service operator on that platform I want high uptime - or, more accurately to this case, low downtime. One of the reasons someone goes for a long-lived service on a VM platform is because the service is in some sense critical: it's Bad News when it drops a connection or doesn't respond. So the operator of that service wants something with a fast restart so that if it ever does need to be bounced, it's back quickly and they return to whatever redundancy level they originally designed to as fast as possible. "Long uptime means a slow boot is fine" has never been true, in my experience.
- nik736 4y ago
  Just spawn a new VM on a hypervisor and delete the old VM on the hypervisor that is dying. You can also balance load this way.
  - regularfry 4y ago
    Assume any suggestion starting with "just" is equally obvious to the operators as it is to you. In this case, it would be helpful to start from the observation that live migration is non-negotiable, which immediately rules out any form of "turn it on then off again."
    - mbreese 4y ago
      In this case, I think the “just” is justified. (Normally I’d agree with you) If you’re starting from the premise that live migration is non-negotiable, then I’d argue that you likely don’t need a quick start. Long lived services can afford a slower start using a different VM technology. We’re not talking hours to provision a server, but the difference between 100ms and a few minutes. My question to you is — what kind of service would need to start fast and be able to migrate? I just don’t see that Venn diagram having a big overlap. But maybe I’m wrong…

0xbadcafebee 4y ago

I think that is GP's point: Firecracker shouldn't be considered a general-purpose solution for more secure containerized workloads.

tptacek 4y ago
Can you try to make that a little more specific? What does "general-purpose" mean here? "Secure containerized workloads" tends to imply serverside. It's hard to think of things --- other than GPU access --- that Firecracker takes away here.
- topspin 4y ago
  > It's hard to think of things No, it isn't. Investigate the K8S world and observe the backflips and somersaults being performed there to deal with networking and storage. Much of that is a consequence of people trying make containers do things that should be run in VMs. A full featured Linux kernel and certain key user space tools can obviate much of that nonsense with ease, deliver better performance, use less resources and be vastly less complex.
  - tptacek 4y ago
    If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one? Firecracker runs a full-featured Linux kernel. It's a hypervisor. It's not a kind of kernel.
    - topspin 4y ago
      > If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one? I already did that with live migration. But ok. Encrypted storage: https://github.com/firecracker-microvm/firecracker/issues/653 https://github.com/firecracker-microvm/firecracker/issues/65...: WONTFIX The answer given is appropriate for firecracker use cases but insufficient otherwise. I'm not anti-firecracker; it's the right choice for many things. Just not all things. The sort of VM I want orchestrated has encrypted (by contract) multi-pathed network block devices to encrypted storage volumes. 3-10 per tenant. This is trivial for a full-featured kernel; multi-path just works, encryption just works. VLAN: https://github.com/weaveworks/ignite/issues/810 https://github.com/weaveworks/ignite/issues/810: Open. Maybe one day. Again, trivial for a full-featured Linux kernel. Has been for ages. I think you're missing the point. It's not about what hypothetical thing firecracker can or can't do. It's about elevating VM orchestration to some degree of parity with what has been created for container orchestration. These VMs and their complex storage and networking requirements should be modeled as we model containers now; through an orchestration system that makes management easy and as foolproof as possible. The fact that firecracker isn't sufficient to be the Micro-VM of choice for this isn't relevant.
      - tptacek 4y ago
        You can encrypt a drive from within Firecracker trivially. It's just Linux, and they're just block devices. You can do all the standard Linux tap interface networking with Firecracker; it just presents as virtio ethernet to the guest. This is the second time you've compared Firecracker to a "full-featured Linux kernel". Again: Firecracker is a hypervisor. It's not a kernel. It runs Linux kernels. "Full-featured" Linux kernels. Whatever kernel you compile.
        topspin 4y ago
        Well now you're arguing with the developers of both firecracker and ignite. Maybe go to the issues I linked and correct them.

topspin 4y ago

> Why would you want to migrate a firecracker vm? I wouldn't. I pointed out the lack of live migration as an of example why firecracker can't cut it as the basis of general purpose VM orchestration platform. > then you can use a different VM tech that supports migration (KVM/Xen/etc) "KVM/Zen/etc" are in the dark ages compared to the state of the art of Kubernetes container orchestration. The reason for that isn't technical; KVM is incredible and full-featured VMs conceivably could be orchestrated with the same fidelity as containers. The reason is business: the cloud behemoths invest only in their serverless business models; thus k8s, firecracker, etc. I contend there is an enormous market for orchestration of more complex workloads that VMs suit well.

regularfry 4y ago

> If you want to be able to migrate a running VM, do you still really need it to start super fast? Yes. "You" is two different people in this sentence. The person doing the migration is often not the person who owns whatever service is running in the VM. They have uncorrelated needs. As a platform operator, I want to migrate VMs to be able to balance load and evacuate dying hardware. As a service operator on that platform I want high uptime - or, more accurately to this case, low downtime. One of the reasons someone goes for a long-lived service on a VM platform is because the service is in some sense critical: it's Bad News when it drops a connection or doesn't respond. So the operator of that service wants something with a fast restart so that if it ever does need to be bounced, it's back quickly and they return to whatever redundancy level they originally designed to as fast as possible. "Long uptime means a slow boot is fine" has never been true, in my experience.

nik736 4y ago
Just spawn a new VM on a hypervisor and delete the old VM on the hypervisor that is dying. You can also balance load this way.
- regularfry 4y ago
  Assume any suggestion starting with "just" is equally obvious to the operators as it is to you. In this case, it would be helpful to start from the observation that live migration is non-negotiable, which immediately rules out any form of "turn it on then off again."
  - mbreese 4y ago
    In this case, I think the “just” is justified. (Normally I’d agree with you) If you’re starting from the premise that live migration is non-negotiable, then I’d argue that you likely don’t need a quick start. Long lived services can afford a slower start using a different VM technology. We’re not talking hours to provision a server, but the difference between 100ms and a few minutes. My question to you is — what kind of service would need to start fast and be able to migrate? I just don’t see that Venn diagram having a big overlap. But maybe I’m wrong…