3 ms·

The 2 examples in Docker Swarm were Redis and MySQL. From the announcement: "Docker Swarm provides high-availability and failover. Docker Swarm continuously he

by scanr 12y ago

The 2 examples in Docker Swarm were Redis and MySQL. From the announcement: "Docker Swarm provides high-availability and failover. Docker Swarm continuously health-checks the Docker daemon’s hosts and, should one suffer an outage, automatically rebalances by moving and re-starting the Docker containers from the failed host to a new one.". Does anyone know how they'll handle the data? Both Redis and MySQL have various ways to deal with high availability e.g. Redis Sentinel, MySQL master / slave or MySQL multi master with Galera.

cpuguy83 12y ago
I believe to start off with, anything with volumes won't be moved... at least this was what was said during the POC demo given for hack day last month.
HorizonXP 12y ago
I'd like to see a reasonable answer to this. Because up until now, I've been using data-only containers to mount directories into these app containers (i.e. Redis and PostgreSQL). The fail over handling has been horrendous for me, because there's no easy way to migrate the data across machines, unless you setup multiple hot slaves or something. And this happens often with CoreOS updates on the alpha channel. Ultimately, I gave up and created a separate Ubuntu VM to run as an NFS server. Every CoreOS instance mounts it, and then my data-only containers now map back to the NFS mount. This way, when CoreOS moves the Redis or PostgreSQL containers, it has the data available to it. It's not my favourite setup, but it's worked well-enough this past week that I haven't had to manually correct things while on vacation. I'm hopeful that someone smarter/more experienced can share a better solution.
- lclarkmichalek 12y ago
  So kube doesn't have this atm, but is looking at introducing some kind of support with a forgiveness based system, along with hooks for services that require data migrations: https://github.com/GoogleCloudPlatform/kubernetes/issues/598 https://github.com/GoogleCloudPlatform/kubernetes/issues/598
- rcoder 12y ago
  Mounting your database storage volume via NFS seems like a surefire way to cause yourself pain down the road. You might want to review the following (old but still relevant) article to understand some of the pitfalls: http://www.time-travellers.org/shane/papers/NFS_considered_harmful.html http://www.time-travellers.org/shane/papers/NFS_considered_h... The tl;dr basically boils down to the fact that PostgreSQL and MySQL (or really any good database engine running on *NIX systems) make very strong assumptions about the POSIX-ness of their underlying filesystem: flock() is fast, sync() calls actually mean data has hit disk, etc. Docker/CoreOS/etc. aren't a replacement for a good SAN or other reliable storage. If you value your data I'd suggest keeping your core database(s) on dedicated machines/VMs (ideally SSD-equipped and UPS-backed). If managing those is too much work, consider a managed cloud database; DynamoDB and RDS can stand in for Redis and Postgres, respectively.
  - rckclmbr 12y ago
    So much this. RDS is a great solution to this problem which hasn't been solved yet. Basically zero maintenance.
    - HorizonXP 12y ago
      My immediate problem is that my software is running on a dedicated server hosted on-site; I have Internet access, but everything is hosted and run on a single massive VMWare ESXi server. I don't have the benefit of cloud-based services like RDS. I could modify my architecture to utilize that instead, and that's something I've thought about doing. As it stands, the VM server is UPS-backed, but does not run on SSDs. There is no SAN. If I were to fix the existing implementation, I would: a) Add a secondary VM server as a redundant backup. b) Add a SAN However, I don't think I can justify the capital expenditures for that. So what I'll likely do is replace the NFS server with a dedicated PostgreSQL server (VM), and perhaps start thinking about moving the majority of the infrastructure out of the building and into the cloud to take advantage of things like RDS. The latter is even more important for scalability as we add more customers.
  - tw04 12y ago
    *Mounting your database that doesn't properly support NFS Oracle runs just fine on NFS. In fact, some of the largest implementations in the world utilize the protocol.
- cpuguy83 12y ago
  I've created a tool to help with migrating volume data: github.com/cpuguy83/docker-volumes and also this PR to help bring forward volume management within Docker: https://github.com/docker/docker/pull/8484 https://github.com/docker/docker/pull/8484
bjt 12y ago
No idea how Docker Swarm handles it, but that the data locality problem for ephemeral containers seems to be the central focus of https://clusterhq.com/ https://clusterhq.com/ (makers of Flocker).
wmf 12y ago
Deis is using Ceph to store persistent data for containers. I don't know if it will be suitable for databases, but I like the idea. http://deis.io/deis-0-13-0-ha-data-store/ http://deis.io/deis-0-13-0-ha-data-store/
lewq 12y ago
We are working on solving this specific problem with Flocker at https://github.com/clusterhq/flocker/ https://github.com/clusterhq/flocker/ - see http://docs.clusterhq.com/en/latest/introduction.html http://docs.clusterhq.com/en/latest/introduction.html for a description of how data migration works. We will be working on high availability after we've made some more progress on getting to a production-ready framework for volume management. We are super interested in building this as a plugin for Swarm, and are trying to work with Docker to make sure the plugins system is suitable for us to be able to contribute it for Swarm users in general.

by scanr 12y ago

cpuguy83 12y ago

I believe to start off with, anything with volumes won't be moved... at least this was what was said during the POC demo given for hack day last month.

HorizonXP 12y ago

I'd like to see a reasonable answer to this. Because up until now, I've been using data-only containers to mount directories into these app containers (i.e. Redis and PostgreSQL). The fail over handling has been horrendous for me, because there's no easy way to migrate the data across machines, unless you setup multiple hot slaves or something. And this happens often with CoreOS updates on the alpha channel. Ultimately, I gave up and created a separate Ubuntu VM to run as an NFS server. Every CoreOS instance mounts it, and then my data-only containers now map back to the NFS mount. This way, when CoreOS moves the Redis or PostgreSQL containers, it has the data available to it. It's not my favourite setup, but it's worked well-enough this past week that I haven't had to manually correct things while on vacation. I'm hopeful that someone smarter/more experienced can share a better solution.

lclarkmichalek 12y ago
So kube doesn't have this atm, but is looking at introducing some kind of support with a forgiveness based system, along with hooks for services that require data migrations: https://github.com/GoogleCloudPlatform/kubernetes/issues/598 https://github.com/GoogleCloudPlatform/kubernetes/issues/598
rcoder 12y ago
Mounting your database storage volume via NFS seems like a surefire way to cause yourself pain down the road. You might want to review the following (old but still relevant) article to understand some of the pitfalls: http://www.time-travellers.org/shane/papers/NFS_considered_harmful.html http://www.time-travellers.org/shane/papers/NFS_considered_h... The tl;dr basically boils down to the fact that PostgreSQL and MySQL (or really any good database engine running on *NIX systems) make very strong assumptions about the POSIX-ness of their underlying filesystem: flock() is fast, sync() calls actually mean data has hit disk, etc. Docker/CoreOS/etc. aren't a replacement for a good SAN or other reliable storage. If you value your data I'd suggest keeping your core database(s) on dedicated machines/VMs (ideally SSD-equipped and UPS-backed). If managing those is too much work, consider a managed cloud database; DynamoDB and RDS can stand in for Redis and Postgres, respectively.
- rckclmbr 12y ago
  So much this. RDS is a great solution to this problem which hasn't been solved yet. Basically zero maintenance.
  - HorizonXP 12y ago
    My immediate problem is that my software is running on a dedicated server hosted on-site; I have Internet access, but everything is hosted and run on a single massive VMWare ESXi server. I don't have the benefit of cloud-based services like RDS. I could modify my architecture to utilize that instead, and that's something I've thought about doing. As it stands, the VM server is UPS-backed, but does not run on SSDs. There is no SAN. If I were to fix the existing implementation, I would: a) Add a secondary VM server as a redundant backup. b) Add a SAN However, I don't think I can justify the capital expenditures for that. So what I'll likely do is replace the NFS server with a dedicated PostgreSQL server (VM), and perhaps start thinking about moving the majority of the infrastructure out of the building and into the cloud to take advantage of things like RDS. The latter is even more important for scalability as we add more customers.
- tw04 12y ago
  *Mounting your database that doesn't properly support NFS Oracle runs just fine on NFS. In fact, some of the largest implementations in the world utilize the protocol.
cpuguy83 12y ago
I've created a tool to help with migrating volume data: github.com/cpuguy83/docker-volumes and also this PR to help bring forward volume management within Docker: https://github.com/docker/docker/pull/8484 https://github.com/docker/docker/pull/8484

bjt 12y ago

No idea how Docker Swarm handles it, but that the data locality problem for ephemeral containers seems to be the central focus of https://clusterhq.com/ https://clusterhq.com/ (makers of Flocker).

wmf 12y ago

Deis is using Ceph to store persistent data for containers. I don't know if it will be suitable for databases, but I like the idea. http://deis.io/deis-0-13-0-ha-data-store/ http://deis.io/deis-0-13-0-ha-data-store/

lewq 12y ago

We are working on solving this specific problem with Flocker at https://github.com/clusterhq/flocker/ https://github.com/clusterhq/flocker/ - see http://docs.clusterhq.com/en/latest/introduction.html http://docs.clusterhq.com/en/latest/introduction.html for a description of how data migration works. We will be working on high availability after we've made some more progress on getting to a production-ready framework for volume management. We are super interested in building this as a plugin for Swarm, and are trying to work with Docker to make sure the plugins system is suitable for us to be able to contribute it for Swarm users in general.