4 ms·

I was deadly afraid to run databases in k8s in 2019. But I've been running CrunchyData's postgres-operator (no affiliation) since 2020 and I can't say anything

I was deadly afraid to run databases in k8s in 2019. But I've been running CrunchyData's postgres-operator (no affiliation) since 2020 and I can't say anything bad about it, just works. It's basically just a bunch of already well thought out open source tools like patroni, pgbackrest and postgresql combined into an operator. So it's not really magic. And the backup to s3 feature in pgbackrest is so solid that I've used the clone cluster and restore cluster feature a few times, just because it's so convenient. But this is in environments that serve only thousands of users, not at all comparable to huge startups, or environments where ms latency is important.

5Qn8mNbc2FNCiVV 4y ago
Can you tell me how the storage there works when using their replica feature? Can I just provide it storage from a k8s-host and it replicates it to the hosts where the other replicas live via software or do I need to provide it a StorageClass that supports distributed volumes? This is the one thing I cannot wrap my head around and apparently there isn't that much information (read benchmarks) on the implications of using different kinds of storages for databases on k8s (or at least I can't find anything)
- motoboi 4y ago
  In the cloud, volumes are just disks. They are attached to the instance were your pod get scheduled. You can in gke choose the disk type (IOPS). If configured like a daemonset, you are garanteed to always match disk to pod instance ordinal (postgres-0 get disk 0, postgres-1 get disk 1). So basically a way to run multiple postgres instances. But the devil is in the details.
  - 5Qn8mNbc2FNCiVV 4y ago
    I know that, I was asking more about the specifics of the Crunchydata postgres operator since they have replication built-in. If it was just them using the volumes and doing the replication on the application level (not using k8s storage management, idk how to better describe it) it'd be easy. If they expect me however to provide a Storageclass that provides replication on top of the volumes it'd be more annoying. I could try this out myself to be honest, but I didn't quite get to it until now
    - knewter 4y ago
      Pg manages replication. If you use something like longhorn to replicate the volume you're just wasting storage - run pg replicas and stream Wal to object storage with wal-g, which the operator probably does (I user stackgres, and it does)
    - INTPenis 4y ago
      I think you misunderstand, crunchydata only deploys postgresql and provisions streaming replication between postgres nodes using patroni, which provisions the postgresql.conf necessary when you choose more than one replica in the PostgresCluster object definition.
nwmcsween 4y ago
Know that you can't use crunchydata pgo in prod without a subscription as per their licensing terms for the docker containers
- noodlesUK 4y ago
  Could you link to something that sets this out? From what I am seeing (just reading in response to your comment, I don’t use pgo myself), both the container and pgo repos have an apache license.
  - oogali 4y ago
    It’s stated in https://www.crunchydata.com/developers/terms-of-use https://www.crunchydata.com/developers/terms-of-use Which is linked to from https://hub.docker.com/r/crunchydata/crunchy-postgres https://hub.docker.com/r/crunchydata/crunchy-postgres
    - linuxftw 4y ago
      It appears there are two operators by Crunchy Data. The PGO (FOSS one) and the commercial one. See this FAQ [1]. The PGO is Apache 2 licensed. 1: https://access.crunchydata.com/documentation/postgres-operator/v5/faq/ https://access.crunchydata.com/documentation/postgres-operat...
    - fulafel 4y ago
      Terms of service != licensing terms. The Docker repo terms of use seem talk about what it's "intended for" and is about support, suitability, security etc covering them against liability. Doesn't read like a license: > By participating in the Program and accepting these terms, you represent that you understand that (i) absent an active Crunchy Data Support Subscription, the Crunchy Developer Software is unsupported and (ii) Crunchy Developer Software is intended for development purposes only, (iii) Crunchy Developer Software may not address known security vulnerabilities, and (iv) Crunchy Data is relying on your representation as a condition of our providing you access to the Crunchy Developer Software. Whereas at their Github repo is there's an actual license (Apache) and says stuff like > Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
znpy 4y ago
I’m curious, have you had the chance of upgrading postgresql version usong that operator? If yes, how’s your experience?
- INTPenis 4y ago
  Hmm good question. I have two sets of clusters and both are on 13.9 so I'm honestly not sure if I ever did a postgres version upgrade. Without checking the docs I would imagine you could do a little hack where you restore from pgbackrest into a clean postgres 14 cluster.

5Qn8mNbc2FNCiVV 4y ago

Can you tell me how the storage there works when using their replica feature? Can I just provide it storage from a k8s-host and it replicates it to the hosts where the other replicas live via software or do I need to provide it a StorageClass that supports distributed volumes? This is the one thing I cannot wrap my head around and apparently there isn't that much information (read benchmarks) on the implications of using different kinds of storages for databases on k8s (or at least I can't find anything)

motoboi 4y ago
In the cloud, volumes are just disks. They are attached to the instance were your pod get scheduled. You can in gke choose the disk type (IOPS). If configured like a daemonset, you are garanteed to always match disk to pod instance ordinal (postgres-0 get disk 0, postgres-1 get disk 1). So basically a way to run multiple postgres instances. But the devil is in the details.
- 5Qn8mNbc2FNCiVV 4y ago
  I know that, I was asking more about the specifics of the Crunchydata postgres operator since they have replication built-in. If it was just them using the volumes and doing the replication on the application level (not using k8s storage management, idk how to better describe it) it'd be easy. If they expect me however to provide a Storageclass that provides replication on top of the volumes it'd be more annoying. I could try this out myself to be honest, but I didn't quite get to it until now
  - knewter 4y ago
    Pg manages replication. If you use something like longhorn to replicate the volume you're just wasting storage - run pg replicas and stream Wal to object storage with wal-g, which the operator probably does (I user stackgres, and it does)
  - INTPenis 4y ago
    I think you misunderstand, crunchydata only deploys postgresql and provisions streaming replication between postgres nodes using patroni, which provisions the postgresql.conf necessary when you choose more than one replica in the PostgresCluster object definition.

nwmcsween 4y ago

Know that you can't use crunchydata pgo in prod without a subscription as per their licensing terms for the docker containers

noodlesUK 4y ago
Could you link to something that sets this out? From what I am seeing (just reading in response to your comment, I don’t use pgo myself), both the container and pgo repos have an apache license.
- oogali 4y ago
  It’s stated in https://www.crunchydata.com/developers/terms-of-use https://www.crunchydata.com/developers/terms-of-use Which is linked to from https://hub.docker.com/r/crunchydata/crunchy-postgres https://hub.docker.com/r/crunchydata/crunchy-postgres
  - linuxftw 4y ago
    It appears there are two operators by Crunchy Data. The PGO (FOSS one) and the commercial one. See this FAQ [1]. The PGO is Apache 2 licensed. 1: https://access.crunchydata.com/documentation/postgres-operator/v5/faq/ https://access.crunchydata.com/documentation/postgres-operat...
  - fulafel 4y ago
    Terms of service != licensing terms. The Docker repo terms of use seem talk about what it's "intended for" and is about support, suitability, security etc covering them against liability. Doesn't read like a license: > By participating in the Program and accepting these terms, you represent that you understand that (i) absent an active Crunchy Data Support Subscription, the Crunchy Developer Software is unsupported and (ii) Crunchy Developer Software is intended for development purposes only, (iii) Crunchy Developer Software may not address known security vulnerabilities, and (iv) Crunchy Data is relying on your representation as a condition of our providing you access to the Crunchy Developer Software. Whereas at their Github repo is there's an actual license (Apache) and says stuff like > Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.

znpy 4y ago

I’m curious, have you had the chance of upgrading postgresql version usong that operator? If yes, how’s your experience?

INTPenis 4y ago
Hmm good question. I have two sets of clusters and both are on 13.9 so I'm honestly not sure if I ever did a postgres version upgrade. Without checking the docs I would imagine you could do a little hack where you restore from pgbackrest into a clean postgres 14 cluster.