5 ms·
Is anyone else very wary about keeping things with persistent state in containers (like databases)?
by elktea 10y ago
Is anyone else very wary about keeping things with persistent state in containers (like databases)?
- pat2man 10y agoThe persistent state is really just the data, not the process. With kubernetes you store the data on a persistent volume (which could be EBS, iSCSI, etc) and the process runs in the container. Host dies? Kubernetes can re-attach that volume on another host, start up a new container and you are back in business. *Note: looks like in this example they are not setting up a persistent volume.
- gtaylor 10y ago> Kubernetes can re-attach that volume on another host, start up a new container and you are back in business. There are still some bugs with this, particularly on AWS. Getting better with every release, though.
- mdaniel 10y agoWe just played with Elastic File System mounted into a Pod via `nfs` and it worked like a charm, with the additional "oh, wow" of being able to attach the same EFS to several Pods at the same time. I was also thrilled that they mounted with the root uid intact so there wasn't any kind of dumb permission juggling. I did say "played with" because we haven't beat on it to know if one could run Jira, Gitlab, Prometheus, that kind of workload. I wouldn't at this point try Postgres but maybe it'd work.
- manojlds 10y agoI use my postgres with EBS though.
- lobster_johnson 10y agoI wonder how suitable EFS is for Postgres. It's supposedly low-latency, high-throughput and supports Unix locking semantics and so on. On the other hand, it's NFS (one of the worst protocols out there), and there have been reports of less than impressive latencies. EFS is also a lot more expensive than EBS.
- mdaniel 10y agoEFS is also a lot more expensive than EBS. That may be true, but getting a k8s cluster unwedged from EBS volume state mismanagement is expensive, too. What I really want is the hutzpah to run GlusterFS but I am not yet brave enough to be in the keeping-a-production-FS-alive business.
- manojlds 10y agoCan you elaborate? I run my postgres on k8s with EBS.
- lobster_johnson 10y agoProbably the most severe issue: https://github.com/kubernetes/kubernetes/issues/29324 https://github.com/kubernetes/kubernetes/issues/29324. You won't encounter it until a pod is rescheduled, though.
- olalonde 10y agoAnother strategy is to use https://github.com/wal-e/wal-e https://github.com/wal-e/wal-e to continuously backup and restore from S3 on boot (e.g. https://github.com/deis/postgres https://github.com/deis/postgres). But restores will be slower and it probably doesn't come with strong durability guarantees.
- haimez 10y agoIdeally, yes. It's not actually the case though. Again, ideally. Well tested databases have evolved to have to survive arbitrary power cuts- even in the middle of a transaction; and so have slowly become reliable enough to trust to start from on-disk state only. Good f-in' luck bringing your db flavor of the month back online from disk state alone. Hope you didn't have any transactions open to your DB before it got rescheduled to a different node in your cluster.
- djsumdog 10y agoI work at a shop that has everything in docker containers. We have mostly coreos clusters, but we're slowly moving to mesos/marathon. And I do mean everything. We've got .NET apps running in docker containers, perl scripts .. they went hard in on docker. But our databases: Amazon RDS instances. It's one of the few things that isn't dockerized. I've personally been looking at a lot of docker solutions for my own projects and I'm still weary of a lot of the data storage solutions out there.