This is correct. The time to recreate a stateful set pod is: 1. Time to detect node failure 2. Time for cloud provider to indicate node down 3. Time to crea

This is correct. The time to recreate a stateful set pod is: 1. Time to detect node failure 2. Time for cloud provider to indicate node down 3. Time to create new pod and wait for startup Today, 1/2 can be from 20-30s to infinite. Kubernetes does not drain a dead node unless it receives some external signal that the node is truly dead. Future work involves adding such failure detection with a fencer (that can ensure the node is partitioned, at which point it's safe to drain it). Just like normal HA, Kubernetes needs to know the node is actually down, vs just partitioned. I would recommend designing your stateful sets so that you can move master from index 0 to index N in the event of a missed heartbeat, and make sure your clients are accessing the master via some proxy (whether kubeproxy or another). If the node dies under partition, but clients are accessing via the service IP, starting deletion of the pod will update the service for any non-partitioned nodes.