5 ms·

Data migration with Kubernetes and Flocker

49 points by lewq 11y ago

errordeveloper 11y ago
Well done, Kai! Cannot wait to try it out ;)
lewq 11y ago
We're super excited about bringing portable volumes to lots of different orchestration frameworks, including Kubernetes. Any questions, we'll be hanging out here, so ask away ;)
- mbreese 11y ago
  Are there any plans to migrate volumes that aren't ZFS backed? (or does it do this already?) I'd think that the workflow would be similar to migrating ZFS snapshots, just extracting volumes from docker directly, and sending tarballs to the new node, as opposed to the built-in ZFS send/receive. It probably wouldn't be as efficient, but it would make Flocker an easier sell to more people. Another question - about what size do you think portable volumes need to be before they stop making sense and you need to move to some other kind of either shared storage (NFS) or remote object storage?
  - lewq 11y ago
    We are working on block device backends which integrate Flocker with OpenStack, EBS and more, as well as ZFS on local storage. Doing a tarball based export-import backend would be an interesting Flocker backend ;) With regards large volumes, adding more backends to Flocker will definitely open up the scope for volumes then a ZFS pool (zpool) you'd want to have on a single node. Generally though, when you need to scale data up to larger scales, it's better to use a distributed data store which can shard your data across a larger number of smaller stateful containers, which can also be managed by Flocker, but of course that's application-dependent.
    - wernerb 11y ago
      Great to see development in storage with docker! I like how this migration works. What about live migrations? Maybe I misunderstood, but I think in the tutorial the service is offline for some time. What about maybe applying backpressure (tcp level) to timeout the request while migration is underway? Edit: If we are doing tarball migrations, then maybe i'd rather have an rsync backend :) Oh and if we are doing large amounts of data migration (gigabytes), maybe even use something akin to something like what bittorrent sync is doing? The scenario would be that you use bittorrent sync (this is a theoretical example..) to one-way migrate continuously (to maybe multiple hosts..??), then pause/hold traffic when satisfied, complete final sync and you are migrated.
      - mbreese 11y ago
        I had a similar question, but my impression from the tutorial (and my pondering) is that you couldn't really do a live migration with containers. I'm not sure it will ever be supported. I don't think that the host would have enough information about the processes in order to capture their state and migrate running processes to a different host. I'm not sure this would even be possible outside of a virtual machine. VM hosts have significantly more information and control about the VM than the container host has about the container. For example, a VM host knows exactly what memory is hot and what is cold to enable sub one-second switchover times. It might be possible to add such support to the Linux kernel itself to support process migration between hosts, but that's the level of work required. The alternative that I was thinking about using your standard high-availability tools to create a new worker on the new host, then gradually remove existing workers on the old node. I think that might be the only way to really make "live" migrations work from a practical standpoint. For web-like services, this would work. For others, it may not be practical. EDIT: Looks like the checkpointing running processes might work after all! (see: http://en.wikipedia.org/wiki/CRIU http://en.wikipedia.org/wiki/CRIU) I must admit that I'm still a little skeptical, but would love to see this working!
        lewq 11y ago
        I believe it is possible to do container live migration -- and for certain stateful services in particular, may make more sense than "add some nodes, remove some others" as you describe. In a previous product, we built seamless container migration of FreeBSD jails based on a network proxy. It did shut down and start up the process, but as long as the process can be gracefully restarted (which most can!) from the user's perspective it just increases latency slightly.
- SEJeff 11y ago
  Any thoughts of flocker using hdfs?
  - lewq 11y ago
    This is definitely interesting. What is the use-case you have in mind for that?
    - SEJeff 11y ago
      zero downtime migrations.
- DannoHung 11y ago
  Have you considered making any integrations between Flocker and SmartDataCenter? It seems like it'd be a pretty nice pairing, especially given that SmartOS natively supports ZFS.
- thinkersilver 11y ago
  Any word on how far off Docker extensions/plugins are? Haven't seen much recent talk on the github issues raised around Dec. The pace of Docker interoperability is frustratingly slow
  - weavenetwork 11y ago
    Hoping to see something soon! We at Weaveworks are involved and mentioned progress on our blog this week.

49 points by lewq 11y ago

Well done, Kai! Cannot wait to try it out ;)

lewq 11y ago

We're super excited about bringing portable volumes to lots of different orchestration frameworks, including Kubernetes. Any questions, we'll be hanging out here, so ask away ;)

mbreese 11y ago
Are there any plans to migrate volumes that aren't ZFS backed? (or does it do this already?) I'd think that the workflow would be similar to migrating ZFS snapshots, just extracting volumes from docker directly, and sending tarballs to the new node, as opposed to the built-in ZFS send/receive. It probably wouldn't be as efficient, but it would make Flocker an easier sell to more people. Another question - about what size do you think portable volumes need to be before they stop making sense and you need to move to some other kind of either shared storage (NFS) or remote object storage?
- lewq 11y ago
  We are working on block device backends which integrate Flocker with OpenStack, EBS and more, as well as ZFS on local storage. Doing a tarball based export-import backend would be an interesting Flocker backend ;) With regards large volumes, adding more backends to Flocker will definitely open up the scope for volumes then a ZFS pool (zpool) you'd want to have on a single node. Generally though, when you need to scale data up to larger scales, it's better to use a distributed data store which can shard your data across a larger number of smaller stateful containers, which can also be managed by Flocker, but of course that's application-dependent.
  - wernerb 11y ago
    Great to see development in storage with docker! I like how this migration works. What about live migrations? Maybe I misunderstood, but I think in the tutorial the service is offline for some time. What about maybe applying backpressure (tcp level) to timeout the request while migration is underway? Edit: If we are doing tarball migrations, then maybe i'd rather have an rsync backend :) Oh and if we are doing large amounts of data migration (gigabytes), maybe even use something akin to something like what bittorrent sync is doing? The scenario would be that you use bittorrent sync (this is a theoretical example..) to one-way migrate continuously (to maybe multiple hosts..??), then pause/hold traffic when satisfied, complete final sync and you are migrated.
    - mbreese 11y ago
      I had a similar question, but my impression from the tutorial (and my pondering) is that you couldn't really do a live migration with containers. I'm not sure it will ever be supported. I don't think that the host would have enough information about the processes in order to capture their state and migrate running processes to a different host. I'm not sure this would even be possible outside of a virtual machine. VM hosts have significantly more information and control about the VM than the container host has about the container. For example, a VM host knows exactly what memory is hot and what is cold to enable sub one-second switchover times. It might be possible to add such support to the Linux kernel itself to support process migration between hosts, but that's the level of work required. The alternative that I was thinking about using your standard high-availability tools to create a new worker on the new host, then gradually remove existing workers on the old node. I think that might be the only way to really make "live" migrations work from a practical standpoint. For web-like services, this would work. For others, it may not be practical. EDIT: Looks like the checkpointing running processes might work after all! (see: http://en.wikipedia.org/wiki/CRIU http://en.wikipedia.org/wiki/CRIU) I must admit that I'm still a little skeptical, but would love to see this working!
      - lewq 11y ago
        I believe it is possible to do container live migration -- and for certain stateful services in particular, may make more sense than "add some nodes, remove some others" as you describe. In a previous product, we built seamless container migration of FreeBSD jails based on a network proxy. It did shut down and start up the process, but as long as the process can be gracefully restarted (which most can!) from the user's perspective it just increases latency slightly.
SEJeff 11y ago
Any thoughts of flocker using hdfs?
- lewq 11y ago
  This is definitely interesting. What is the use-case you have in mind for that?
  - SEJeff 11y ago
    zero downtime migrations.
DannoHung 11y ago
Have you considered making any integrations between Flocker and SmartDataCenter? It seems like it'd be a pretty nice pairing, especially given that SmartOS natively supports ZFS.
thinkersilver 11y ago
Any word on how far off Docker extensions/plugins are? Haven't seen much recent talk on the github issues raised around Dec. The pace of Docker interoperability is frustratingly slow
- weavenetwork 11y ago
  Hoping to see something soon! We at Weaveworks are involved and mentioned progress on our blog this week.