4 ms·

This is very interesting. I thought it was best to keep yr db out of the k8s cluster. But this seems to make Postgres "cloud native". The main reason for keepi

by cies 10y ago

This is very interesting. I thought it was best to keep yr db out of the k8s cluster. But this seems to make Postgres "cloud native". The main reason for keeping dbs out of k8s was that the storage (persistence) solutions in k8s were not up to the task yet. Now I wonder how is Patroni doing persistence? I cannot find anything on it in the Patroni docs. Maybe Patroni is so "self healing" that a proper storage solution is not an issue -- dunno, just guessing here.

vishesh92 10y ago
I am not really sure Patroni has anything to do with the persistence of data. It just uses etcd, zookeper or consul to elect a master in case of a failover and uses their key value store to save the information about the current master. Its upto you whether you keep your database on a container or not. And how the data is managed by the container. This talk by Josh Berkus explains how Patroni works pretty well. https://www.youtube.com/watch?v=OH9WSEiMsAw https://www.youtube.com/watch?v=OH9WSEiMsAw
- cies 10y ago
  Thanks. All the k8s talk in the README confused me I guess. And that's a great talk, made Patroni much clearer for me.
- jan_berlin 10y ago
  The great thing today is there are lots of options with Patroni, it is very much up to you and your requirements what kind of storage you select and how you want to recover from node and storage failure. Patroni with Spilo e.g. relies on EBS or local discs and can also ship WAL to S3. On Google it uses their equivalent solutions. On Kubernetes earlier posts are correct that you may rely on Kubernetes bringing your pod back with the same underlying volume thus providing some kind of availability, but for others this is not good enough, e.g. remember EBS is single AZ only. Patroni can help here, running and orchestrating either slaves or giving you automated recovery from S3.
rsanders 10y ago
There are many solutions for network block device style persistent storage in K8S now. In the cloud, you have solutions like Amazon's EBS and GCE's persistent disks. On VMware you have local VMDKs. On your own equipment, there's support for iSCSI, Ceph RBD, OpenStack Cinder, Portworx, ScaleIO, Quobyte, and just about anything else if you write a few scripts to format and mount volumes on the host (FlexVolume). Except for FlexVolume, K8S is able to dynamically provision volumes of a requested size for you. If you're feeling adventurous there's support for network filesystems like NFS, GlusterFS, CephFS, but I wouldn't recommend those for Postgres. https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes https://kubernetes.io/docs/concepts/storage/persistent-volum... In one of his demos, Josh Berkus recommends just using the K8S ephemeral "emptyDir" storage type on each node, and counting on Patroni to maintain data persistence through replication. e.g. any one Postgres node may die and lose its local data, but when its replacement is created, it will be populated with a fresh copy of the database wherever it is scheduled on the K8S cluster. That sounds workable for smaller databases, though it gives every single person I've suggested it to nightmares.
- abkfenris 10y ago
  It's a bit scary when you're using K8S ephemeral storage and depending on a basebackup from another pod, but you can also set it up to restore from your usual backup system (WAL-E, pghoard, barman, pgBackrest). That way any time you lose a pod, you get a test of your backup system on restore.
  - eicnix 10y ago
    There is currently some work being done for improving local storage usage in k8s. https://github.com/kubernetes/community/pull/306 https://github.com/kubernetes/community/pull/306 Additionally k8s can be configured to restart pods on the same node so you can use the local node storage for persistence and rely on your DB for replication.
  - icebraining 10y ago
    you can also set it up to restore from your usual backup system (WAL-E, pghoard, barman, pgBackrest) That sounds interesting, has anyone documented that kind of setup?
takeda 10y ago
Biggest issue, is not really k8s but docker, its drivers for storage are still quite young. I personally wouldn't place a database in container for that reason. Essentially when you do so any durability guarantees that a database provides are going out of the window. Things might seem to work fine, but when container get abruptly terminated in a wrong moment you might learn that all data is gone.
- hintbits 10y ago
  Why would the data disappear, given that it's stored on an external volume (i.e. EBS or Google Cloud Storage or just a directory on your laptop), not in the container FS?
  - takeda 10y ago
    I simplified, because in the end that would be the equivalent. The danger is that some bugs in a filesystem, manifests themselves only when there is abrupt termination, otherwise things appear normal. Stuff like data is being written in incorrect order, or data is synced to disk at wrong time, or not at all. On top of that user also expects performance, so some shortcuts can be taken that compromises the above (especially if the criteria is that docker is intended for stateless applications). When there bugs like that and things are abruptly terminated, the data on the disk can be heavily mangled to a point that can't be recovered so essentially it would be lost.
- smarterclayton 10y ago
  Kubernetes does not use Docker's storage drivers. There have been bugs with attach/detach of cloud block storage though that has impacted applications a few times over Kubernetes lifecycle, although all have been fixed fairly quickly.

by cies 10y ago

vishesh92 10y ago

I am not really sure Patroni has anything to do with the persistence of data. It just uses etcd, zookeper or consul to elect a master in case of a failover and uses their key value store to save the information about the current master. Its upto you whether you keep your database on a container or not. And how the data is managed by the container. This talk by Josh Berkus explains how Patroni works pretty well. https://www.youtube.com/watch?v=OH9WSEiMsAw https://www.youtube.com/watch?v=OH9WSEiMsAw

cies 10y ago
Thanks. All the k8s talk in the README confused me I guess. And that's a great talk, made Patroni much clearer for me.
jan_berlin 10y ago
The great thing today is there are lots of options with Patroni, it is very much up to you and your requirements what kind of storage you select and how you want to recover from node and storage failure. Patroni with Spilo e.g. relies on EBS or local discs and can also ship WAL to S3. On Google it uses their equivalent solutions. On Kubernetes earlier posts are correct that you may rely on Kubernetes bringing your pod back with the same underlying volume thus providing some kind of availability, but for others this is not good enough, e.g. remember EBS is single AZ only. Patroni can help here, running and orchestrating either slaves or giving you automated recovery from S3.

rsanders 10y ago

There are many solutions for network block device style persistent storage in K8S now. In the cloud, you have solutions like Amazon's EBS and GCE's persistent disks. On VMware you have local VMDKs. On your own equipment, there's support for iSCSI, Ceph RBD, OpenStack Cinder, Portworx, ScaleIO, Quobyte, and just about anything else if you write a few scripts to format and mount volumes on the host (FlexVolume). Except for FlexVolume, K8S is able to dynamically provision volumes of a requested size for you. If you're feeling adventurous there's support for network filesystems like NFS, GlusterFS, CephFS, but I wouldn't recommend those for Postgres. https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes https://kubernetes.io/docs/concepts/storage/persistent-volum... In one of his demos, Josh Berkus recommends just using the K8S ephemeral "emptyDir" storage type on each node, and counting on Patroni to maintain data persistence through replication. e.g. any one Postgres node may die and lose its local data, but when its replacement is created, it will be populated with a fresh copy of the database wherever it is scheduled on the K8S cluster. That sounds workable for smaller databases, though it gives every single person I've suggested it to nightmares.

abkfenris 10y ago
It's a bit scary when you're using K8S ephemeral storage and depending on a basebackup from another pod, but you can also set it up to restore from your usual backup system (WAL-E, pghoard, barman, pgBackrest). That way any time you lose a pod, you get a test of your backup system on restore.
- eicnix 10y ago
  There is currently some work being done for improving local storage usage in k8s. https://github.com/kubernetes/community/pull/306 https://github.com/kubernetes/community/pull/306 Additionally k8s can be configured to restart pods on the same node so you can use the local node storage for persistence and rely on your DB for replication.
- icebraining 10y ago
  you can also set it up to restore from your usual backup system (WAL-E, pghoard, barman, pgBackrest) That sounds interesting, has anyone documented that kind of setup?

takeda 10y ago

Biggest issue, is not really k8s but docker, its drivers for storage are still quite young. I personally wouldn't place a database in container for that reason. Essentially when you do so any durability guarantees that a database provides are going out of the window. Things might seem to work fine, but when container get abruptly terminated in a wrong moment you might learn that all data is gone.

hintbits 10y ago
Why would the data disappear, given that it's stored on an external volume (i.e. EBS or Google Cloud Storage or just a directory on your laptop), not in the container FS?
- takeda 10y ago
  I simplified, because in the end that would be the equivalent. The danger is that some bugs in a filesystem, manifests themselves only when there is abrupt termination, otherwise things appear normal. Stuff like data is being written in incorrect order, or data is synced to disk at wrong time, or not at all. On top of that user also expects performance, so some shortcuts can be taken that compromises the above (especially if the criteria is that docker is intended for stateless applications). When there bugs like that and things are abruptly terminated, the data on the disk can be heavily mangled to a point that can't be recovered so essentially it would be lost.
smarterclayton 10y ago
Kubernetes does not use Docker's storage drivers. There have been bugs with attach/detach of cloud block storage though that has impacted applications a few times over Kubernetes lifecycle, although all have been fixed fairly quickly.