3 ms·

I don't get it, all the failure modes talked are not kubernetes specific. They happen if your running any ha database cluster? If you have async replication wi

I don't get it, all the failure modes talked are not kubernetes specific. They happen if your running any ha database cluster? If you have async replication with a large amount of lag, well of course your gonna lose data if master goes down if your not careful. Regardless if your using kubernetes or not... Can anyone explain why these failure modes are kubernetes specific? They just sound like things you have to think about regardless if your running a HA cluster...

drdaeman 9y ago
I think it's not K8s-specific but specific to anything when there is an external automated orchestration that can mess with the software at random (schedule and reschedule deployments etc etc). If you have N servers that are deployed by hand or, better say, with some automation ran by hand - you can be sure things are not moving at random. If you schedule a payload in K8s (or Docker Swarm or whatever) - the scheduler will only ask you for preferences, but will otherwise make decisions on its own. Like deciding to drain a node because disk is getting full or whatever. Or just schedule an updated service to another node because it felt like doing so. And most of software - Postgres included - doesn't have any idea about all of this. Some of the points raised (like leader election) are not specific to orchestration, but some (like persistent storage for containers) are sort of unique to containerized world. Unless there is a crazy sysadmin who can walk into server room and randomly shuffle drives. :)
_jal 9y ago
They're not kubernetes-specific, but apparently there are people who think deploying PG in it will magically Just Work(tm). What I took away from TFA is two-fold: clustered-DB administration is hard to automate (e.g., there's a reason DBAs exist), and a lot of the tooling DBAs use now has to be rebuilt to function in a k8 environment, or replaced. The first struck me as blindingly obvious; the second I find highly relevant, personally.
- zokier 9y ago
  > clustered-DB administration is hard to automate (e.g., there's a reason DBAs exist) This is why I'm excited about CockroachDB; it hopefully will make operating DB cluster easier.
  - bonesss 9y ago
    Without pretending I've ever done it in production: I'd have to assume that distributed databases will make the hop into a Kubernetes environment easier than databases that started monolithic and added replication over time. Most of the issues presented in the article relate to data loss during network segregation and leader election. Those are important, but distributed systems are generally a bit more explicit in their CAP compromises.
emmelaich 9y ago
It's not Kubernetes specific and that is actually addressed in the article! It's not postgres specific either. It will happen whenever you have a cluster or HA solution which is not intimately cognizant of the app it is supporting. Shit, it is hard enough to get any database extremely reliable even with their own solutions (Oracle RAC, Postgres SLONY, mysql replication). Adding Kubernetes / Docker to that mix makes for an interesting life. Caution is advised.

drdaeman 9y ago

I think it's not K8s-specific but specific to anything when there is an external automated orchestration that can mess with the software at random (schedule and reschedule deployments etc etc). If you have N servers that are deployed by hand or, better say, with some automation ran by hand - you can be sure things are not moving at random. If you schedule a payload in K8s (or Docker Swarm or whatever) - the scheduler will only ask you for preferences, but will otherwise make decisions on its own. Like deciding to drain a node because disk is getting full or whatever. Or just schedule an updated service to another node because it felt like doing so. And most of software - Postgres included - doesn't have any idea about all of this. Some of the points raised (like leader election) are not specific to orchestration, but some (like persistent storage for containers) are sort of unique to containerized world. Unless there is a crazy sysadmin who can walk into server room and randomly shuffle drives. :)

_jal 9y ago

They're not kubernetes-specific, but apparently there are people who think deploying PG in it will magically Just Work(tm). What I took away from TFA is two-fold: clustered-DB administration is hard to automate (e.g., there's a reason DBAs exist), and a lot of the tooling DBAs use now has to be rebuilt to function in a k8 environment, or replaced. The first struck me as blindingly obvious; the second I find highly relevant, personally.

zokier 9y ago
> clustered-DB administration is hard to automate (e.g., there's a reason DBAs exist) This is why I'm excited about CockroachDB; it hopefully will make operating DB cluster easier.
- bonesss 9y ago
  Without pretending I've ever done it in production: I'd have to assume that distributed databases will make the hop into a Kubernetes environment easier than databases that started monolithic and added replication over time. Most of the issues presented in the article relate to data loss during network segregation and leader election. Those are important, but distributed systems are generally a bit more explicit in their CAP compromises.

emmelaich 9y ago

It's not Kubernetes specific and that is actually addressed in the article! It's not postgres specific either. It will happen whenever you have a cluster or HA solution which is not intimately cognizant of the app it is supporting. Shit, it is hard enough to get any database extremely reliable even with their own solutions (Oracle RAC, Postgres SLONY, mysql replication). Adding Kubernetes / Docker to that mix makes for an interesting life. Caution is advised.