3 ms·

I wonder how much things like this are needed going forward. From my (rudimentary) knowledge about Kubernetes and stateful sets, I think that Kubernetes is able

I wonder how much things like this are needed going forward. From my (rudimentary) knowledge about Kubernetes and stateful sets, I think that Kubernetes is able to solve a lot of the issues surrounding failover, recovery of the old master, and partitioning; providing that we use Kubernetes in combination with networked storage that guarantees reliability. It appears that the way to setup PostgreSQL (or any replicated database) with Kubernetes is to treat index 0 in the stateful set as the master, and everything else as slaves. Each pod is to be connected to some networked storage. If the master goes down, then Kubernetes detects that through the health check, and simply reschedules the master on another node. The storage volume that the old master was using, is simply reattached to this other node. The master going down does not imply storage failure. Storage reliability then becomes a separate problem which is solved in another system (e.g. the RAID system or whatever). In this kind of setup, there is no failover support in the database itself. From the point of view of the database itself, it looks as if the underlying hardware/OS "automatically" recovered from failure. This way we don't have to mess with promoting slaves and stuff. Kubernetes already assigns static IPs to services, so pgpool and similar tools -- in so far they are only used to provide a stable network address for PostgreSQL -- become redundant. Network partitioning is "solved" by treating the Kubernetes state as the single authoritative description of the network state. What do people think about this? Obviously this setup won't work if you don't have networked storage that can be reattached to another node, but I'm thinking that maybe reattachable networked storage should be the future.

tokenizerrr 10y ago
> providing that we use Kubernetes in combination with networked storage that guarantees reliability What kind of networked storage do people like with kubernetes? I've recently set up a small cluster not in any cloud, and persistent cross-node storage is a concern. There's quite a few options such as glusterfs, but I'd be curious to know if anyone here knows about the tradeoffs.
- noway421 10y ago
  The set up op describing is more suited for clouds I guess, where networked storage is already a ready to use feature like aws ebs block storage.
- takeda 10y ago
  If you're in AWS then EFS is what is designed to do that, it is not cheap, but setting up a reliable solution is not easy either. If your goal is file based storage (not block based), application can make api calls to get data and is mostly read, then S3 might also be an option. Alternatively, roll-your-own on EC2 instances, which might not be as easy.
foxylion 10y ago
I think the largest disadvantage is, that you won't get real high availability. Promoting a slave to a master is done in a second. Restarting the master can take some time (especially when it crashed previously). There is also the problem of losing the storage volume of your master node (ebs block storage do neither have 100% availability, nor 100% durability). In this case you can't recover your master.
- smarterclayton 10y ago
  This is correct. The time to recreate a stateful set pod is: 1. Time to detect node failure 2. Time for cloud provider to indicate node down 3. Time to create new pod and wait for startup Today, 1/2 can be from 20-30s to infinite. Kubernetes does not drain a dead node unless it receives some external signal that the node is truly dead. Future work involves adding such failure detection with a fencer (that can ensure the node is partitioned, at which point it's safe to drain it). Just like normal HA, Kubernetes needs to know the node is actually down, vs just partitioned. I would recommend designing your stateful sets so that you can move master from index 0 to index N in the event of a missed heartbeat, and make sure your clients are accessing the master via some proxy (whether kubeproxy or another). If the node dies under partition, but clients are accessing via the service IP, starting deletion of the pod will update the service for any non-partitioned nodes.
jacques_chester 10y ago
Relaunching crashed processes is the visible part of HA and yes, Kubernetes and comparable systems do this well. I don't have direct experience with building HA data systems, but some other teams at my employers (Pivotal) do. We ship some of these as BOSH releases intended to allow service injection into Cloud Foundry (and pretty soon OpenShift and Kubernetes too[0]). For the purposes of what you describe, BOSH and k8s are comparable. Given that we have multiple teams that work continuously on building fully-packaged, fully automated HA data systems, I suspect that this stuff is harder than it looks. [0] https://www.openservicebrokerapi.org/ https://www.openservicebrokerapi.org/

tokenizerrr 10y ago

> providing that we use Kubernetes in combination with networked storage that guarantees reliability What kind of networked storage do people like with kubernetes? I've recently set up a small cluster not in any cloud, and persistent cross-node storage is a concern. There's quite a few options such as glusterfs, but I'd be curious to know if anyone here knows about the tradeoffs.

noway421 10y ago
The set up op describing is more suited for clouds I guess, where networked storage is already a ready to use feature like aws ebs block storage.
takeda 10y ago
If you're in AWS then EFS is what is designed to do that, it is not cheap, but setting up a reliable solution is not easy either. If your goal is file based storage (not block based), application can make api calls to get data and is mostly read, then S3 might also be an option. Alternatively, roll-your-own on EC2 instances, which might not be as easy.

foxylion 10y ago

I think the largest disadvantage is, that you won't get real high availability. Promoting a slave to a master is done in a second. Restarting the master can take some time (especially when it crashed previously). There is also the problem of losing the storage volume of your master node (ebs block storage do neither have 100% availability, nor 100% durability). In this case you can't recover your master.

smarterclayton 10y ago
This is correct. The time to recreate a stateful set pod is: 1. Time to detect node failure 2. Time for cloud provider to indicate node down 3. Time to create new pod and wait for startup Today, 1/2 can be from 20-30s to infinite. Kubernetes does not drain a dead node unless it receives some external signal that the node is truly dead. Future work involves adding such failure detection with a fencer (that can ensure the node is partitioned, at which point it's safe to drain it). Just like normal HA, Kubernetes needs to know the node is actually down, vs just partitioned. I would recommend designing your stateful sets so that you can move master from index 0 to index N in the event of a missed heartbeat, and make sure your clients are accessing the master via some proxy (whether kubeproxy or another). If the node dies under partition, but clients are accessing via the service IP, starting deletion of the pod will update the service for any non-partitioned nodes.

jacques_chester 10y ago

Relaunching crashed processes is the visible part of HA and yes, Kubernetes and comparable systems do this well. I don't have direct experience with building HA data systems, but some other teams at my employers (Pivotal) do. We ship some of these as BOSH releases intended to allow service injection into Cloud Foundry (and pretty soon OpenShift and Kubernetes too[0]). For the purposes of what you describe, BOSH and k8s are comparable. Given that we have multiple teams that work continuously on building fully-packaged, fully automated HA data systems, I suspect that this stuff is harder than it looks. [0] https://www.openservicebrokerapi.org/ https://www.openservicebrokerapi.org/