4 ms·

Are there any plans to migrate volumes that aren't ZFS backed? (or does it do this already?) I'd think that the workflow would be similar to migrating ZFS snap

by mbreese 12y ago

Are there any plans to migrate volumes that aren't ZFS backed? (or does it do this already?) I'd think that the workflow would be similar to migrating ZFS snapshots, just extracting volumes from docker directly, and sending tarballs to the new node, as opposed to the built-in ZFS send/receive. It probably wouldn't be as efficient, but it would make Flocker an easier sell to more people. Another question - about what size do you think portable volumes need to be before they stop making sense and you need to move to some other kind of either shared storage (NFS) or remote object storage?

lewq 12y ago
We are working on block device backends which integrate Flocker with OpenStack, EBS and more, as well as ZFS on local storage. Doing a tarball based export-import backend would be an interesting Flocker backend ;) With regards large volumes, adding more backends to Flocker will definitely open up the scope for volumes then a ZFS pool (zpool) you'd want to have on a single node. Generally though, when you need to scale data up to larger scales, it's better to use a distributed data store which can shard your data across a larger number of smaller stateful containers, which can also be managed by Flocker, but of course that's application-dependent.
- wernerb 12y ago
  Great to see development in storage with docker! I like how this migration works. What about live migrations? Maybe I misunderstood, but I think in the tutorial the service is offline for some time. What about maybe applying backpressure (tcp level) to timeout the request while migration is underway? Edit: If we are doing tarball migrations, then maybe i'd rather have an rsync backend :) Oh and if we are doing large amounts of data migration (gigabytes), maybe even use something akin to something like what bittorrent sync is doing? The scenario would be that you use bittorrent sync (this is a theoretical example..) to one-way migrate continuously (to maybe multiple hosts..??), then pause/hold traffic when satisfied, complete final sync and you are migrated.
  - mbreese 12y ago
    I had a similar question, but my impression from the tutorial (and my pondering) is that you couldn't really do a live migration with containers. I'm not sure it will ever be supported. I don't think that the host would have enough information about the processes in order to capture their state and migrate running processes to a different host. I'm not sure this would even be possible outside of a virtual machine. VM hosts have significantly more information and control about the VM than the container host has about the container. For example, a VM host knows exactly what memory is hot and what is cold to enable sub one-second switchover times. It might be possible to add such support to the Linux kernel itself to support process migration between hosts, but that's the level of work required. The alternative that I was thinking about using your standard high-availability tools to create a new worker on the new host, then gradually remove existing workers on the old node. I think that might be the only way to really make "live" migrations work from a practical standpoint. For web-like services, this would work. For others, it may not be practical. EDIT: Looks like the checkpointing running processes might work after all! (see: http://en.wikipedia.org/wiki/CRIU http://en.wikipedia.org/wiki/CRIU) I must admit that I'm still a little skeptical, but would love to see this working!
    - lewq 12y ago
      I believe it is possible to do container live migration -- and for certain stateful services in particular, may make more sense than "add some nodes, remove some others" as you describe. In a previous product, we built seamless container migration of FreeBSD jails based on a network proxy. It did shut down and start up the process, but as long as the process can be gracefully restarted (which most can!) from the user's perspective it just increases latency slightly.
      - mbreese 12y ago
        Oh yeah, if you can survive stopping/starting the process, it should be easily do-able. You just have to manage migrating any stateful data (which you already do!). But, I wouldn't call that a "live migration" in the VM live migration sense. However, I wouldn't worry too much about it, since you're absolutely right that most processes can restart gracefully.

by mbreese 12y ago

lewq 12y ago

We are working on block device backends which integrate Flocker with OpenStack, EBS and more, as well as ZFS on local storage. Doing a tarball based export-import backend would be an interesting Flocker backend ;) With regards large volumes, adding more backends to Flocker will definitely open up the scope for volumes then a ZFS pool (zpool) you'd want to have on a single node. Generally though, when you need to scale data up to larger scales, it's better to use a distributed data store which can shard your data across a larger number of smaller stateful containers, which can also be managed by Flocker, but of course that's application-dependent.

wernerb 12y ago
Great to see development in storage with docker! I like how this migration works. What about live migrations? Maybe I misunderstood, but I think in the tutorial the service is offline for some time. What about maybe applying backpressure (tcp level) to timeout the request while migration is underway? Edit: If we are doing tarball migrations, then maybe i'd rather have an rsync backend :) Oh and if we are doing large amounts of data migration (gigabytes), maybe even use something akin to something like what bittorrent sync is doing? The scenario would be that you use bittorrent sync (this is a theoretical example..) to one-way migrate continuously (to maybe multiple hosts..??), then pause/hold traffic when satisfied, complete final sync and you are migrated.
- mbreese 12y ago
  I had a similar question, but my impression from the tutorial (and my pondering) is that you couldn't really do a live migration with containers. I'm not sure it will ever be supported. I don't think that the host would have enough information about the processes in order to capture their state and migrate running processes to a different host. I'm not sure this would even be possible outside of a virtual machine. VM hosts have significantly more information and control about the VM than the container host has about the container. For example, a VM host knows exactly what memory is hot and what is cold to enable sub one-second switchover times. It might be possible to add such support to the Linux kernel itself to support process migration between hosts, but that's the level of work required. The alternative that I was thinking about using your standard high-availability tools to create a new worker on the new host, then gradually remove existing workers on the old node. I think that might be the only way to really make "live" migrations work from a practical standpoint. For web-like services, this would work. For others, it may not be practical. EDIT: Looks like the checkpointing running processes might work after all! (see: http://en.wikipedia.org/wiki/CRIU http://en.wikipedia.org/wiki/CRIU) I must admit that I'm still a little skeptical, but would love to see this working!
  - lewq 12y ago
    I believe it is possible to do container live migration -- and for certain stateful services in particular, may make more sense than "add some nodes, remove some others" as you describe. In a previous product, we built seamless container migration of FreeBSD jails based on a network proxy. It did shut down and start up the process, but as long as the process can be gracefully restarted (which most can!) from the user's perspective it just increases latency slightly.
    - mbreese 12y ago
      Oh yeah, if you can survive stopping/starting the process, it should be easily do-able. You just have to manage migrating any stateful data (which you already do!). But, I wouldn't call that a "live migration" in the VM live migration sense. However, I wouldn't worry too much about it, since you're absolutely right that most processes can restart gracefully.