4 ms·

Great to see development in storage with docker! I like how this migration works. What about live migrations? Maybe I misunderstood, but I think in the tutoria

by wernerb 11y ago

Great to see development in storage with docker! I like how this migration works. What about live migrations? Maybe I misunderstood, but I think in the tutorial the service is offline for some time. What about maybe applying backpressure (tcp level) to timeout the request while migration is underway? Edit: If we are doing tarball migrations, then maybe i'd rather have an rsync backend :) Oh and if we are doing large amounts of data migration (gigabytes), maybe even use something akin to something like what bittorrent sync is doing? The scenario would be that you use bittorrent sync (this is a theoretical example..) to one-way migrate continuously (to maybe multiple hosts..??), then pause/hold traffic when satisfied, complete final sync and you are migrated.

mbreese 11y ago
I had a similar question, but my impression from the tutorial (and my pondering) is that you couldn't really do a live migration with containers. I'm not sure it will ever be supported. I don't think that the host would have enough information about the processes in order to capture their state and migrate running processes to a different host. I'm not sure this would even be possible outside of a virtual machine. VM hosts have significantly more information and control about the VM than the container host has about the container. For example, a VM host knows exactly what memory is hot and what is cold to enable sub one-second switchover times. It might be possible to add such support to the Linux kernel itself to support process migration between hosts, but that's the level of work required. The alternative that I was thinking about using your standard high-availability tools to create a new worker on the new host, then gradually remove existing workers on the old node. I think that might be the only way to really make "live" migrations work from a practical standpoint. For web-like services, this would work. For others, it may not be practical. EDIT: Looks like the checkpointing running processes might work after all! (see: http://en.wikipedia.org/wiki/CRIU http://en.wikipedia.org/wiki/CRIU) I must admit that I'm still a little skeptical, but would love to see this working!
- lewq 11y ago
  I believe it is possible to do container live migration -- and for certain stateful services in particular, may make more sense than "add some nodes, remove some others" as you describe. In a previous product, we built seamless container migration of FreeBSD jails based on a network proxy. It did shut down and start up the process, but as long as the process can be gracefully restarted (which most can!) from the user's perspective it just increases latency slightly.
  - mbreese 11y ago
    Oh yeah, if you can survive stopping/starting the process, it should be easily do-able. You just have to manage migrating any stateful data (which you already do!). But, I wouldn't call that a "live migration" in the VM live migration sense. However, I wouldn't worry too much about it, since you're absolutely right that most processes can restart gracefully.
- jsmthrowaway 11y ago
  CRIU has worked fine for a long time. I've used CRIU under LXC in a lab, and CRIU alone in production. This problem arises because Docker positions their intrinsics as novel containers, even though there was an entire field of prior art long before Docker showed up. Hence your confusion and worry checkpointing will never be supported; it already was, but you didn't know about it, because to you Docker == containers. One of my many problems with Docker, because they feed that. Look how easy it is: http://criu.org/LXC http://criu.org/LXC
  - mbreese 11y ago
    From that example, it's pretty clear you can checkpoint and restart a container with CRIU, but can you migrate that state/checkpoint file to a different host? I can see restarting it on the same host, but a different host sounds "difficult". That's what I have the most questions about. I'm not sure how that would work between systems, particularly open files handles, sockets, etc... If you have a "share-nothing" architecture, I think that would be easier, but it you have to maintain some kind of state within the container, things get real hairy fast. I have to agree though that it seems strange to me that Docker has gotten a lot of support so quickly when things like LXC, FreeBSD Jails, or Solaris Zones have been established for so long. I've only played with Jails a bit, having done more work with full virtual machines in Linux. The other containers, I at least had a familiarity with before Docker (nothing in production). However, I had not heard about CRIU until today. But it looks really interesting!
  - wernerb 11y ago
    There is an active pull request here: https://github.com/docker/libcontainer/pull/479#issuecomment-94060567 https://github.com/docker/libcontainer/pull/479#issuecomment...
lewq 11y ago
Hey, yes! We're super interested in varying grades of seamlessness wrt container migration. What would your use case be? Is it OK to shut down the container and pause network connections before starting the container up on the other host and then unleashing them? Or would you be looking for full process freeze/thaw like with CRIU?
- wernerb 11y ago
  Yes, full freeze of process with CRIU could be an option though not limited to that. Tcp network pausing/throttling as well. Regarding backpressure I am talking about reactive streams, of which I am a fan of late, this would be better suited perhaps at the application level [0]. Regarding infra/pod level migrations, I am wondering if this would be something to implement at the kubernetes level , perhaps a new type of controller/service ? You could then perhaps use labels to identify throttling/freezing/thawing of pods during migration. [0]: http://www.reactive-streams.org/ http://www.reactive-streams.org/

by wernerb 11y ago

mbreese 11y ago

I had a similar question, but my impression from the tutorial (and my pondering) is that you couldn't really do a live migration with containers. I'm not sure it will ever be supported. I don't think that the host would have enough information about the processes in order to capture their state and migrate running processes to a different host. I'm not sure this would even be possible outside of a virtual machine. VM hosts have significantly more information and control about the VM than the container host has about the container. For example, a VM host knows exactly what memory is hot and what is cold to enable sub one-second switchover times. It might be possible to add such support to the Linux kernel itself to support process migration between hosts, but that's the level of work required. The alternative that I was thinking about using your standard high-availability tools to create a new worker on the new host, then gradually remove existing workers on the old node. I think that might be the only way to really make "live" migrations work from a practical standpoint. For web-like services, this would work. For others, it may not be practical. EDIT: Looks like the checkpointing running processes might work after all! (see: http://en.wikipedia.org/wiki/CRIU http://en.wikipedia.org/wiki/CRIU) I must admit that I'm still a little skeptical, but would love to see this working!

lewq 11y ago
I believe it is possible to do container live migration -- and for certain stateful services in particular, may make more sense than "add some nodes, remove some others" as you describe. In a previous product, we built seamless container migration of FreeBSD jails based on a network proxy. It did shut down and start up the process, but as long as the process can be gracefully restarted (which most can!) from the user's perspective it just increases latency slightly.
- mbreese 11y ago
  Oh yeah, if you can survive stopping/starting the process, it should be easily do-able. You just have to manage migrating any stateful data (which you already do!). But, I wouldn't call that a "live migration" in the VM live migration sense. However, I wouldn't worry too much about it, since you're absolutely right that most processes can restart gracefully.
jsmthrowaway 11y ago
CRIU has worked fine for a long time. I've used CRIU under LXC in a lab, and CRIU alone in production. This problem arises because Docker positions their intrinsics as novel containers, even though there was an entire field of prior art long before Docker showed up. Hence your confusion and worry checkpointing will never be supported; it already was, but you didn't know about it, because to you Docker == containers. One of my many problems with Docker, because they feed that. Look how easy it is: http://criu.org/LXC http://criu.org/LXC
- mbreese 11y ago
  From that example, it's pretty clear you can checkpoint and restart a container with CRIU, but can you migrate that state/checkpoint file to a different host? I can see restarting it on the same host, but a different host sounds "difficult". That's what I have the most questions about. I'm not sure how that would work between systems, particularly open files handles, sockets, etc... If you have a "share-nothing" architecture, I think that would be easier, but it you have to maintain some kind of state within the container, things get real hairy fast. I have to agree though that it seems strange to me that Docker has gotten a lot of support so quickly when things like LXC, FreeBSD Jails, or Solaris Zones have been established for so long. I've only played with Jails a bit, having done more work with full virtual machines in Linux. The other containers, I at least had a familiarity with before Docker (nothing in production). However, I had not heard about CRIU until today. But it looks really interesting!
- wernerb 11y ago
  There is an active pull request here: https://github.com/docker/libcontainer/pull/479#issuecomment-94060567 https://github.com/docker/libcontainer/pull/479#issuecomment...

lewq 11y ago

Hey, yes! We're super interested in varying grades of seamlessness wrt container migration. What would your use case be? Is it OK to shut down the container and pause network connections before starting the container up on the other host and then unleashing them? Or would you be looking for full process freeze/thaw like with CRIU?

wernerb 11y ago
Yes, full freeze of process with CRIU could be an option though not limited to that. Tcp network pausing/throttling as well. Regarding backpressure I am talking about reactive streams, of which I am a fan of late, this would be better suited perhaps at the application level [0]. Regarding infra/pod level migrations, I am wondering if this would be something to implement at the kubernetes level , perhaps a new type of controller/service ? You could then perhaps use labels to identify throttling/freezing/thawing of pods during migration. [0]: http://www.reactive-streams.org/ http://www.reactive-streams.org/