3 ms·

I had a similar question, but my impression from the tutorial (and my pondering) is that you couldn't really do a live migration with containers. I'm not sure i

by mbreese 12y ago

I had a similar question, but my impression from the tutorial (and my pondering) is that you couldn't really do a live migration with containers. I'm not sure it will ever be supported. I don't think that the host would have enough information about the processes in order to capture their state and migrate running processes to a different host. I'm not sure this would even be possible outside of a virtual machine. VM hosts have significantly more information and control about the VM than the container host has about the container. For example, a VM host knows exactly what memory is hot and what is cold to enable sub one-second switchover times. It might be possible to add such support to the Linux kernel itself to support process migration between hosts, but that's the level of work required. The alternative that I was thinking about using your standard high-availability tools to create a new worker on the new host, then gradually remove existing workers on the old node. I think that might be the only way to really make "live" migrations work from a practical standpoint. For web-like services, this would work. For others, it may not be practical. EDIT: Looks like the checkpointing running processes might work after all! (see: http://en.wikipedia.org/wiki/CRIU http://en.wikipedia.org/wiki/CRIU) I must admit that I'm still a little skeptical, but would love to see this working!

lewq 12y ago
I believe it is possible to do container live migration -- and for certain stateful services in particular, may make more sense than "add some nodes, remove some others" as you describe. In a previous product, we built seamless container migration of FreeBSD jails based on a network proxy. It did shut down and start up the process, but as long as the process can be gracefully restarted (which most can!) from the user's perspective it just increases latency slightly.
- mbreese 12y ago
  Oh yeah, if you can survive stopping/starting the process, it should be easily do-able. You just have to manage migrating any stateful data (which you already do!). But, I wouldn't call that a "live migration" in the VM live migration sense. However, I wouldn't worry too much about it, since you're absolutely right that most processes can restart gracefully.
jsmthrowaway 12y ago
CRIU has worked fine for a long time. I've used CRIU under LXC in a lab, and CRIU alone in production. This problem arises because Docker positions their intrinsics as novel containers, even though there was an entire field of prior art long before Docker showed up. Hence your confusion and worry checkpointing will never be supported; it already was, but you didn't know about it, because to you Docker == containers. One of my many problems with Docker, because they feed that. Look how easy it is: http://criu.org/LXC http://criu.org/LXC
- mbreese 12y ago
  From that example, it's pretty clear you can checkpoint and restart a container with CRIU, but can you migrate that state/checkpoint file to a different host? I can see restarting it on the same host, but a different host sounds "difficult". That's what I have the most questions about. I'm not sure how that would work between systems, particularly open files handles, sockets, etc... If you have a "share-nothing" architecture, I think that would be easier, but it you have to maintain some kind of state within the container, things get real hairy fast. I have to agree though that it seems strange to me that Docker has gotten a lot of support so quickly when things like LXC, FreeBSD Jails, or Solaris Zones have been established for so long. I've only played with Jails a bit, having done more work with full virtual machines in Linux. The other containers, I at least had a familiarity with before Docker (nothing in production). However, I had not heard about CRIU until today. But it looks really interesting!
- wernerb 12y ago
  There is an active pull request here: https://github.com/docker/libcontainer/pull/479#issuecomment-94060567 https://github.com/docker/libcontainer/pull/479#issuecomment...

by mbreese 12y ago

lewq 12y ago

I believe it is possible to do container live migration -- and for certain stateful services in particular, may make more sense than "add some nodes, remove some others" as you describe. In a previous product, we built seamless container migration of FreeBSD jails based on a network proxy. It did shut down and start up the process, but as long as the process can be gracefully restarted (which most can!) from the user's perspective it just increases latency slightly.

mbreese 12y ago
Oh yeah, if you can survive stopping/starting the process, it should be easily do-able. You just have to manage migrating any stateful data (which you already do!). But, I wouldn't call that a "live migration" in the VM live migration sense. However, I wouldn't worry too much about it, since you're absolutely right that most processes can restart gracefully.

jsmthrowaway 12y ago

CRIU has worked fine for a long time. I've used CRIU under LXC in a lab, and CRIU alone in production. This problem arises because Docker positions their intrinsics as novel containers, even though there was an entire field of prior art long before Docker showed up. Hence your confusion and worry checkpointing will never be supported; it already was, but you didn't know about it, because to you Docker == containers. One of my many problems with Docker, because they feed that. Look how easy it is: http://criu.org/LXC http://criu.org/LXC

mbreese 12y ago
From that example, it's pretty clear you can checkpoint and restart a container with CRIU, but can you migrate that state/checkpoint file to a different host? I can see restarting it on the same host, but a different host sounds "difficult". That's what I have the most questions about. I'm not sure how that would work between systems, particularly open files handles, sockets, etc... If you have a "share-nothing" architecture, I think that would be easier, but it you have to maintain some kind of state within the container, things get real hairy fast. I have to agree though that it seems strange to me that Docker has gotten a lot of support so quickly when things like LXC, FreeBSD Jails, or Solaris Zones have been established for so long. I've only played with Jails a bit, having done more work with full virtual machines in Linux. The other containers, I at least had a familiarity with before Docker (nothing in production). However, I had not heard about CRIU until today. But it looks really interesting!
wernerb 12y ago
There is an active pull request here: https://github.com/docker/libcontainer/pull/479#issuecomment-94060567 https://github.com/docker/libcontainer/pull/479#issuecomment...