6 ms·

Alright, all you brave people who run databases in docker: where do you store the actual data? - Host mounts? So what happens when a container gets rescheduled

by raarts 10y ago

Alright, all you brave people who run databases in docker: where do you store the actual data? - Host mounts? So what happens when a container gets rescheduled to another node? - Docker volumes? What happens when a container gets rescheduled to another node? - External SAN? Congratulations on your budget. That's not easily doable in public cloud I guess? - Shared filesystem like NFS, or Ceph? How's the performance for you? And the filesystem itself runs outside of your docker cloud I guess? - In the containers themselves? So you probably run some clustered database. So what if disaster strikes and all your containers go down? And get moved around? Also, databases in many cases need to be tuned for performance. How do you do that in a cloud? Maybe most of you are not running container scheduling. Which really is only taking containers half way.

Jdam 10y ago
Host mounts, because they are not rescheduled. If you say that no rescheduling only "takes containers half way", you're not fully aware of how containers are used nowadays. In my company for ex, reproducibility, immutability and a uniform deployment process is key.
wstrange 10y ago
Docker alone does not solve these problems. You need something like Kubernetes >- Host mounts? So what happens when a container gets rescheduled to another node? Persistent Volumes and Claims. Kube manages moving containers, and the associated storage. > - External SAN? Congratulations on your budget. That's not easily doable in public cloud I guess? Specify a storage class on the persistent volume claim. Most providers (Google, AWS, Azure,..) support SSD > So what if disaster strikes and all your containers go down? And get moved around? You let Kube handle rescheduling containers onto new nodes.
banachtarski 10y ago
I think you have some pretty big misconceptions which accounts for at least half your skepticism. Most databases are designed to restore from information that's stored on disk. If you use host volumes, there is no concept of a container "moving around." Just being dead or alive. If a container starts on a host that was previously running, that container now is the previous one.
nickbauman 10y ago
I use Linux containers extensively. They can be rolled over by the container orchestration service at any time so I do not use them for databases except in a dev cluster where we trash and reload test decks constantly.
jazoom 10y ago
Mount a volume on the host and make sure only one database container runs on that host. Done. Then it's exactly like a traditional setup except better because if the process crashes or config gets corrupted it can be replaced automatically in a few seconds, good as new. There's nothing brave about that.
- hmottestad 10y ago
  This is what we do. We use Docker Swarm and label our database host. Then add a constraint to the service/container i our docker-compose file so that the database is always scheduled to that host. Great thing about this setup is that we can now run the entire production system locally on our laptops and test servers with or without swarm. Everything is deployed in the same way and everyone uses the same versions and configurations (even for our database).
- Ronsenshi 10y ago
  Same here. Mounted on the host with the restriction to running containers.
justicezyx 10y ago
IIRC think Kubernets support pinned deployment. I.e., you can force scheduler to schedule running on fixed set of machines.
drvdevd 10y ago
I use ZFS under everything (including docker) and so I can use scheduled snapshots, backups, etc, to do the real data management, and not worry about losing data. I can use host mounts too, but I prefer to just back up and manage all of docker, out of band, without thinking too much about it. EDIT: granted this is for relatively small databases though. I think it could be scaled up however.
rad_gruchalski 10y ago
Mesos with placement constraints solves this problem. In case of AWS, store the data on an EBS volume, use constraints to dedicate agents with specific hardware to the DB containers. Always ensure that EBS is available to the agent on which the DB supposed to run. Easy. Same for Kafka, same for Cassandra, you name it...
lobster_johnson 10y ago
Kubernetes lets you do whatever you want. For example, if you tell it to mount an AWS EBS volume into the container, if the container is rescheduled on another host, the volume will be automatically remounted there. Kubernetes supports a whole range of volume types that all travel with the container, fully managed. You can even ask it to carve out pieces from a larger volume, so you don't have to create one volume per container. You can pin containers to individual hosts and use host volumes, but of course that rather defeats the purpose of using containers in the first place.
closeparen 10y ago
Most are running in a cloud environment with a dynamically attachable block store abstraction, so basically an external SAN. I'm very interested in approaching this problem on bare metal, though.

by raarts 10y ago

Jdam 10y ago

Host mounts, because they are not rescheduled. If you say that no rescheduling only "takes containers half way", you're not fully aware of how containers are used nowadays. In my company for ex, reproducibility, immutability and a uniform deployment process is key.

wstrange 10y ago

Docker alone does not solve these problems. You need something like Kubernetes >- Host mounts? So what happens when a container gets rescheduled to another node? Persistent Volumes and Claims. Kube manages moving containers, and the associated storage. > - External SAN? Congratulations on your budget. That's not easily doable in public cloud I guess? Specify a storage class on the persistent volume claim. Most providers (Google, AWS, Azure,..) support SSD > So what if disaster strikes and all your containers go down? And get moved around? You let Kube handle rescheduling containers onto new nodes.

banachtarski 10y ago

I think you have some pretty big misconceptions which accounts for at least half your skepticism. Most databases are designed to restore from information that's stored on disk. If you use host volumes, there is no concept of a container "moving around." Just being dead or alive. If a container starts on a host that was previously running, that container now is the previous one.

nickbauman 10y ago

I use Linux containers extensively. They can be rolled over by the container orchestration service at any time so I do not use them for databases except in a dev cluster where we trash and reload test decks constantly.

jazoom 10y ago

Mount a volume on the host and make sure only one database container runs on that host. Done. Then it's exactly like a traditional setup except better because if the process crashes or config gets corrupted it can be replaced automatically in a few seconds, good as new. There's nothing brave about that.

hmottestad 10y ago
This is what we do. We use Docker Swarm and label our database host. Then add a constraint to the service/container i our docker-compose file so that the database is always scheduled to that host. Great thing about this setup is that we can now run the entire production system locally on our laptops and test servers with or without swarm. Everything is deployed in the same way and everyone uses the same versions and configurations (even for our database).
Ronsenshi 10y ago
Same here. Mounted on the host with the restriction to running containers.

justicezyx 10y ago

IIRC think Kubernets support pinned deployment. I.e., you can force scheduler to schedule running on fixed set of machines.

drvdevd 10y ago

I use ZFS under everything (including docker) and so I can use scheduled snapshots, backups, etc, to do the real data management, and not worry about losing data. I can use host mounts too, but I prefer to just back up and manage all of docker, out of band, without thinking too much about it. EDIT: granted this is for relatively small databases though. I think it could be scaled up however.

rad_gruchalski 10y ago

Mesos with placement constraints solves this problem. In case of AWS, store the data on an EBS volume, use constraints to dedicate agents with specific hardware to the DB containers. Always ensure that EBS is available to the agent on which the DB supposed to run. Easy. Same for Kafka, same for Cassandra, you name it...

lobster_johnson 10y ago

Kubernetes lets you do whatever you want. For example, if you tell it to mount an AWS EBS volume into the container, if the container is rescheduled on another host, the volume will be automatically remounted there. Kubernetes supports a whole range of volume types that all travel with the container, fully managed. You can even ask it to carve out pieces from a larger volume, so you don't have to create one volume per container. You can pin containers to individual hosts and use host volumes, but of course that rather defeats the purpose of using containers in the first place.

closeparen 10y ago

Most are running in a cloud environment with a dynamically attachable block store abstraction, so basically an external SAN. I'm very interested in approaching this problem on bare metal, though.