9 ms·
CloudNativePG – Postgres Operator for Kubernetes
- uneekname 3y agoCould anyone speak to how this compares to other Postgres solutions for Kubernetes?
- rollulus 3y agoExactly what I was wondering about as well. I have no prod experience with it, but e.g. Zalando’s operator has been solid for me.
- zinclozenge 3y agoI've never used it myself, but while doing research I noticed that it received a lot of praise from users. One thing that did catch my attention is that it doesn't use statefulsets for the postgres pods. I mostly agree with their reasons, but I haven't taken the time to understand their implementation.
- remram 3y agoThe rationale is here: https://cloudnative-pg.io/documentation/1.20/controller/ https://cloudnative-pg.io/documentation/1.20/controller/
- uneekname 3y agoI found that to be well-written, so another point for CloudNativePG
- EdwardDiego 3y agoStrimzi (Kafka operator) is moving away from StatefulSets for the same reason.
- nz_cal 3y agoI’ve been using it in prod for a while now, pretty happy with it. Solid, integrated pgbouncer, crd based, good license. I do wish there was a simpler way to handle major version upgrades of pg. When I looked at some alternatives, these were my thoughts (may be out of date by now) - kubegres: maintained by one guy, lots of GitHub issues with no responses - crunchy data pgo: licensing is not obvious, seems to require license in some cases - stackgres: agpl, no thanks - zalando: they know pg extremely well, but it’s not kubernetes native. Doesn’t include pgbouncer. Doesn’t handle automatic failover when a node dies, and during testing it often got confused when killing a node.
- uneekname 3y agoI appreciate your perspective, thanks. Major version bumps do seem to be difficult sometimes with pg in my limited experience.
- gbartolini 3y agoIf you are interested, I suggest this article I wrote that covers the current state of major upgrades with CloudNativePG: https://www.enterprisedb.com/blog/current-state-major-postgresql-upgrades-cloudnativepg-kubernetes https://www.enterprisedb.com/blog/current-state-major-postgr...
- samokhvalov 3y ago> Doesn’t handle automatic failover when a node dies Patroni, the most popular autofailover solution for Postgres, is developed in Zalando. Of course it's included in Zalando operator by default.
- sieabahlpark 3y ago[dead]
- fijiaarone 3y agoIt’s the opposite. A Kubernetes solution for Postgres.
- uneekname 3y agoHaha, that might be a better way of putting it. "A solution for running Postgres in Kubernetes"
- perrygeo 3y agoA good comparison with other operators here [2022]: https://blog.palark.com/cloudnativepg-and-other-kubernetes-operators-for-postgresql/ https://blog.palark.com/cloudnativepg-and-other-kubernetes-o... Personally I've only used Kubegres (https://www.kubegres.io/ https://www.kubegres.io/), which didn't even make the above list. It's ok for a personal project. All k8s solutions for postgres take subtly different approaches. It seems that they've all converged on the Operator pattern. The basics are easy: run a database process which persists data to a cloud disk of your choice. The hard parts are how to update, migrate, backup, restore, monitor, failover, replicate, etc. These kubernetes "operators" promise to fulfill the role of a DBA but, just like hiring a DBA, it requires buy-in to their approach.
- EdwardDiego 3y agoFrom my experiences with Strimzi, I think of it as "half-managed", like, it'll make doing things like upgrades easy as, but yeah, it's just tooling that makes self-management easier.
- JimBlackwood 3y agoWe are currently moving to CNPG and have tried CrunchyData and Zalando in the process. The other two we abandoned while trying it out. Zalando: - Relies on WAL-E which is now obsolete - Documentation all over the place - Hacky setup that deviates from K8s standards (no easy way to set user through supplying secrets, for instance). In general, it feels like an operator to be used internally at Zalando according to their conventions that they just open sourced. It doesn’t seem like they want (or get time) to support other conventions. I don’t think this is a bad thing, it’s already great Zalando open sourced this. Just important to know when you decide to use it. CrunchyData: - Incomplete documentation (Certain values settings are missing from their API specs) - Hacky user setup. - Doesn’t support running without backups enabled. (Obviously, you’d never want to run without backups setup on prod. But when testing, it’s nice to not need to have a perfect setup from the start. Without backups, it will let the database pods fill up their PVC’s with a WAL. Even when not doing any writes. It fills up at about 10GB/day.) - Backups seem to randomly fail. It looks pretty OK otherwise. CNPG: - Adheres to K8S standards - Seem to realise that an Operator will (currently) not fully replace a DBA. Their kubectl plug-in is great to interact with the cluster. Obviously we still need to test rollovers and restoring from backups, but so far it’s been easy to setup. It does suffer from what most operators suffer from; their CRD is a mess. The UID of the Postgres Container is specified on the same indentation level as my switchOverDelay, superuserSecret and bootstrap spec. Would be nice if these would follow a more logical grouping (pod spec, users, switchover).
- smartbit 3y agoCan confirm Zalando & Crunchy are quite a mess, contrary to CNPG. See my other post https://news.ycombinator.com/item?id=37618886 https://news.ycombinator.com/item?id=37618886
- koolba 3y agoWhy is wal-e obsolete? I’m assuming you mean the golang replacement wal-g.
- JimBlackwood 3y agoSee the GitHub: https://github.com/wal-e/wal-e https://github.com/wal-e/wal-e Unmaintained would’ve made more sense to say, but the maintainer choose the words “obsolete” so I took those. :) Seems to be obsolete due to a lack of interest and contributions.
- dvh 3y agoWas it's always spelled kubernetes? In my "Mandela timeline" it was kubernets and just recently switched.
- martijnvds 3y agoK8s was always K-ubernete-s (8 letters)
- c0brac0bra 3y agoAlways AFAIK. It's a Greek word.
- wrboyce 3y agoNitpick, but I think it is from Ancient Greek. A cursory Google tells me the modern Greek descendant is κυβερνήτης/kyvernítis and has a slightly different meaning.
- cgio 3y agoκυβερνήτης is the ancient word too. Maybe they were trying to word play with cube, which by the way also comes from κύβος-kyvos and not kuvos. Exactly same meaning as far as I know, wonder what the cursory google came up as the difference in meaning. Also root for cybernetics...
- osigurdson 3y agoIt was never called "Kubernets".
- comradesmith 3y agoAlways kubernetes, but I’ve heard some engineers with accents say it in a way that sounds like “kubernets”
- ghshephard 3y agoIt's been K - 8 letters - S long enough that everyone at work just calls it "Kates" now, which initially annoyed me to no end, but has kind of grown on me.
- rgrieselhuber 3y agoIs it just me or is the color scheme on this website smoking crack?
- osigurdson 3y agoIt definitely looks different.
- deleted 3y ago[deleted]
- pphysch 3y agoThey're mixing white and black text on the same background. Mixing muddy red with dark blues. Yeah.
- susam 3y agoYou are probably viewing the dark colour scheme of the website. The website looks fine in light colour scheme. However in dark colour scheme, while it flips the background and text colour alright, it does not alter the accent colours sufficiently well. To see what I mean, compare https://i.imgur.com/hhL0fZH.png https://i.imgur.com/hhL0fZH.png (dark colour scheme) with https://i.imgur.com/rr24RkN.png https://i.imgur.com/rr24RkN.png (light colour scheme). Consider the colour of the text "Try Kubernetes Way" and the background colour. In light mode, we have foreground colour #DC2626 on background colour #FFFFFF. This has a contrast ratio of 4.82. WCAG 2.0 level AA recommends minimum contrast ratio of 3.0 for large text. So a contrast of 4.82 is decent enough. However in dark mode, we see #991B1B on #075985 which has a contrast ratio of only 1.09. This ratio is too low and fails all WCAG contrast requirements.
- out-of-ideas 3y ago> The Kubernetes way to me, this translates into: a really complicated mess of manifests of layers-of-layers of fun, but hidden in an attempt to make it easy; but when it does work in the simplest forms, it works great. if any lesson ive learned; let kubernetes manage only things that wont shoot yourself in the foot with; you wont know until you try :)
- tehlike 3y agoMarking your data pvc with retention: retain is a good start :)
- hardwaresofton 3y agoSurely you mean making the default for the storage class :) Running out of space > losing data
- tehlike 3y agoyes sir.
- deathanatos 3y agoDo I need an operator, though? I'd definitely take something over the insanity that is managed services and constant ticket filing with the cloud vendors, but my gut says I just need a Helm chart that makes it decently easy to, a.) bring up PG & some replicas that hopefully have a decent consistency story and b.) a backup job that can write to S3 et al. I need a better "seduce to use" for a CRD for Postgres, I guess. "It doesn’t rely on statefulsets [and manages disks itself]" and "The Kubernetes way" (and dark red on blue, yeesh) … just doesn't inspire me?
- cortesoft 3y agoThe operator also handles when a node fails or is taken down. An operator can handle replicating to a new replica on a new node and making sure everything stays consistent.
- cogman10 3y agoThe place the operator looks to be most useful is updating the db. It'll do "the right thing" updating the secondary pods first, doing a switch over of the primary, then updating the old primary pod. That sort of action is real hard to do with just a plain helm chart (especially if you didn't plan for it up front).
- lyjackal 3y agoThis. If it’s a database, I’d like the deployment infrastructure to be aware during upgrades and able to execute complex actions
- fivre 3y agoCRs that simply instantiate an instance aren't all that useful IMO. they're much more useful if you have something like Prometheus' case where they want to attach configuration to various Kubernetes resources and can't easily fit it in annotations. i don't know that i would actually recommend Helm for much though, since dealing with templates beyond a basic "sub string into field" use case is pain and misery. once you start dealing with named helper templates that operate at different scopes and all but the simplest control flow, the lack of tooling makes debugging templates a nightmare. the operator ecosystem has its own problems (i still can't tell what the extra Red Hat stuff beyond kubebuilder is really helping with and can't stand updating it), but having an actual programming language and all the accompanying type checking and testing tools available is a major benefit. if you don't need something complex, kustomize feels much less breakage-prone than Helm, though its patches aren't as intuitive to write.
- lucgagan 3y agoThat is very cool and much needed, but I am _paranoid_ about hosting the data myself. I've lost too many databases over the years to the most random mistakes. Granted, technology has matured a lot over the last decode, but still... the cost (the nights spent debugging and troubleshooting) of maintaining the database yourself is just not worth it when managed solutions exist.
- berkle4455 3y agoI think with any database the solution is simply backups no? backups that preferably aren’t tied to the hosted solution at all.
- panyam 3y agoActually not quite. Backups (assuming you are doing single node) still needs you to decide on your SLOs - RTO (how long it takes to restore) and RPO (how much data lose you can suffer) numbers. On the instant snazy end you have streaming backups and recovery and then on the other extreme you have backup once in N hours/days and restore taking how ever long it takes to restore (so you have customer outages you need to negotiate. Now let us involve multi node, (both replication and partitioning of shards). As shards go and up and down ensuring data is in sync etc is a hard consistency problem and needs man years of operational excellence and bug fixing. So when people think databases - they think of the cool stuff - the database engine that does relational algebra and handles SQL queries. That is (IMO) only 1% of a practical, performant, reliable database (offering).
- tetha 3y agoAlso what about the customer that deleted an important thing 6 weeks ago and absolutely needs it recovered? BTW, it's just one tentant in that DB, the other shouldn't be recovered, naturally.
- aprilllll 3y agoIn that case, it’d probably be best to just handle deletions at the application layer (e.g., setting a “deleted_at” timestamp field with scheduled permanent deletions later). And in terms of data compliance, it’s very important to make sure permanent deletions propagate through your backup systems within a reasonable amount of time - Google Cloud[1], for example, is ~180 days. [1] https://services.google.com/fh/files/misc/gcp_data_deletion_nda.pdf https://services.google.com/fh/files/misc/gcp_data_deletion_...
- yheowahwjsbf 3y ago> CloudNativePG does not use `StatefulSet`s for managing data persistence This is awesome. StatefulSets are an endless source of pain.
- claytongulick 3y agoI prefer to run PostgreSQL The Grumpy Greybeard Way. Like, on a (virtual) machine. With a config file. I'm doing a web app with terabytes of healthcare data with intense computation and geospatial queries with an average 60ms response time including latency. On a modest vm. With a config file.
- ibash 3y agoWith a WHAT?
- claytongulick 3y agoA config file. :-)
- samokhvalov 3y agoYou'll probably like this: https://github.com/vitabaks/postgresql_cluster https://github.com/vitabaks/postgresql_cluster
- claytongulick 3y agoI do like it! My current DR/HA is basically just hourly block level delta snapshots. It's super simple, cheap and easy to do if the business can tolerate up to an hour of potential data loss.
- yjftsjthsd-h 3y agoRunning a database on a VM doesn't sound very Greybeard™ of you; don't you know that abstracting like that kills performance and takes you away from the hardware? (Only half joking; obviously you totally can make it run fine on a VM, but it also runs fine in a container.)
- xyst 3y agoI look forward to trying this out with a simple app i am creating. Never really liked using managed solutions but for the longest time this was the accepted paradigm/sentiment because “containers are ephemeral”
- master_crab 3y agoKubernetes: a complicated orchestration system that runs stateless applications brilliantly (after six months of work from your infra engineers). Postgres: the very definition of a stateful service. Those two sound like perfect stablemates.
- figmert 3y agoYour definition of Kubernetes is accurate for maybe 5 years ago. These days Kubernetes is perfectly capable of running stateful services and supports them as a first class citizen.
- crabbone 3y agoNothing can be further from the truth. Kubernetes has nothing to offer to anyone that wants to work with storage (but there's a myriad of CSIs). Here, let me give you an example of a system that offers storage in a similar way to how Kubernetes offers... well, it isn't really that good at offering compute, but, at least it kind of does it. So, Ceph -- that's something that makes sense to run PostgreSQL on (it's a storage provider). Kubernetes isn't a storage provider. It doesn't know how to manage it even... I.e. if you think that you run PosgreSQL on Kubernetes -- you are mistaken. Something else does it. Kubernetes is a proxy there, at best (but probably is completely irrelevant).
- qaq 3y agoLook at benchmarks of Postgres on Ceph
- crabbone 3y agoPart of my job is to measure storage performance... I can tell you at leas this: there cannot be a meaningful "benchmark of Postgres on Ceph". Too many things will influence the benchmark way too much. You need to be a lot more specific when you talk about such benchmarks. Here are some things you will need to present: * Are OSDs connected to the node being tested through network or are they closer (NVMe / SAS / SATA?). If network, what's the bandwidth? What's the latency? What if it's stuff like reliable Ethernet that's used for iSCSI / NVMe over IP or something like that? * How much memory (relative to data at rest) does the node have? * What is the layout of memory buffers in PostgreSQL? * What is the setting used for synchronization in PostgreSQL? * How much replication is going to happen (Ceph pool size)? * Block sizes and frame sizes. * Type of workload. Surprisingly, some queries can exploit parallelism in I/O while other queries cannot. Surprisingly some queries will need a lot of synchronization while others don't. And there's more, it would be too tedious to try to give an exhaustive list of things to control for. And the problem is, at least those mentioned here can influence performance sometimes up to an order of magnitude, sometimes two orders of magnitude...
- crabbone 3y ago> CloudNativePG exclusively relies on the Kubernetes API server to maintain the state of a PostgreSQL cluster. This is scary af. Kubernetes API server is very finicky and unreliable. It's probably the first component that fails in Kubernetes no matter what the problem is (eg. I recently managed to run it into unrecoverable state by accidentally starting 10K jobs instead of 100). This is just outright bad idea... but, really "reliable" and Kubernetes don't belong together. So, if you wanted an unreliable PostgreSQL, which you have no idea how it's managed and how to recover... well, that sounds like fun!
- hnarn 3y agoCan someone explain why this is being downvoted?
- erulabs 3y agoBecause it’s reactionary and a bit silly, while providing no real value. Yes, orchestration requires the orchestration tool. No, k8s api is not unreliable or “finicky”. This is being downvoted because it’s the equivalent of saying “delivery requires the use of a car? But engines are so unreliable, bad idea!” Saying k8s and “reliable” don’t belong together is just flamebait - k8s runs reliably for tens of thousands of extremely large shops.
- crabbone 3y agoNo it's not. Kubernetes is not an equivalent of a car, it's an equivalent of Land Rover Discovery 5 in your scenario. There are plenty of cars which score differently on some pre-agreed reliability scale (and, specifically, this model of Land Rover scores very poorly). Kubernetes is light years away from being a reliable system. It's a system to support Web sites, which don't need to do anything that would require a lot of investment into reliability. Nobody in their right mind would use Kubernetes to run highly-reliable systems that, eg. put human lives at risk. PostgreSQL is in a completely different category from reliability standpoint: much more reliable. But, if you put it on Kubernetes footing, you give up that reliability. Again, maybe your business is to make Web sites -- then nobody cares, it's good enough. Running a database of a bank on top of Kubernetes -- well, I hope nobody actually tries that. And size has nothing to do with that. Reliability is about all sorts of metrics like mean time to failure / recovery and guarantees that the system makes like once a particular condition is encountered, the system will halt and so on. Kubernetes never claimed anything in that category, and, in practice, it doesn't offer any satisfactory guarantees of its own performance. Anecdotally, I saw Kubernetes fail due to its internal problems more times that I can count. Like, we are talking about at least hundreds of times. And by "fail" I mean that the system enters into a state that cannot be salvaged by a reboot or replacement a master node(s). Since my interaction with it involves a lot of testing, but I'm not testing Kubernetes, rather something else running on it, I'm all too used to deleting and deploying new clusters.
- rlex 3y agoI never ran postgres in prod in k8s, but i used cloudnative-pg for some light, non-critical loads, dev/stg stuff. It works and it's fast, recovery just works (unlike zalando, which refuses to operate with anything but AWS s3). I also reported few minor bugs to cnpg team and they fixed them in no time.
- nullify88 3y agoWe're using the Zalando operator. We have our wals and base backups shipped to Azure Storage Blob. Not sure where this S3 only thing comes from. Can you exllain further?
- boomskats 3y agoThe way I'm reading the comments here is 'AWS S3 as the only flavour of S3 that works', i.e. they're having issues backing up to to services like DigitalOcean Object Storage via S3, rather than S3 being the only backup protocol that's supported.
- rlex 3y agoyes, precisely. It's UI part that's broken, which cannot list snapshots. Issue is here, no fix since 2020, sadly: https://github.com/zalando/postgres-operator/issues/937 https://github.com/zalando/postgres-operator/issues/937
- rubenv 3y agoUsing the Zalando operator to send to Minio over here. No problems at all so perhaps what you ran into has since been fixed?
- dang 3y agoRelated: CloudNativePG: A New Open-Source Kubernetes Operator for Postgres - https://news.ycombinator.com/item?id=31366983 https://news.ycombinator.com/item?id=31366983 - May 2022 (11 comments) Related a bit more loosely: Why we moved from AWS RDS to Postgres in Kubernetes - https://news.ycombinator.com/item?id=32986325 https://news.ycombinator.com/item?id=32986325 - Sept 2022 (145 comments) Encrypting Postgres Data at Rest in Kubernetes - https://news.ycombinator.com/item?id=29057505 https://news.ycombinator.com/item?id=29057505 - Oct 2021 (40 comments) Show HN: StackGres: open-source Postgres-aaS on Kubernetes with 120 Extensions - https://news.ycombinator.com/item?id=28865432 https://news.ycombinator.com/item?id=28865432 - Oct 2021 (3 comments) Running Postgres in Kubernetes [pdf] - https://news.ycombinator.com/item?id=23682450 https://news.ycombinator.com/item?id=23682450 - June 2020 (100 comments) Should I Run Postgres on Kubernetes? - https://news.ycombinator.com/item?id=17512496 https://news.ycombinator.com/item?id=17512496 - July 2018 (2 comments)
- brightball 3y agoDefinitely useful for non-prod environments. I don’t want my production database sharing infrastructure with anything if I can avoid it though.
- yjftsjthsd-h 3y agoIf you really care, just taint the workers you want the database to run on.
- dmeijboom 3y agoWe’ve been using Zalando’s Postgres Operator before in production and we recently switched to cloudnative-pg. We didn’t experience any issues so far and I’m a big fan of their design choices (where you have a single user, single database for each micro-service that requires one).
- Already__Taken 3y agoI used zalando and surprisingly hit a bunch of config issues where the backups won't run on an s3 endpoint that isn't aws or wal-e now wont run because it's got no db credentials but does the seaweed endpoint. now my one db-one user-one pod "cluster" is dead because it won't elect itself leader. I simply cannot kick it correctly to revive it.
- rubenv 3y agoHow did you migrate between the two?
- rauanm 3y agoWe've started using CNPG cautiously in production and can say we're finally confident about running databases in kubernetes. (we had a bad run before with custom setups, and went back to using VMs again). I am very grateful to maintainers for not only open sourcing the industrial-grade operator, but also sharing so much of their expertise. Unexpected side-effect of adopting CNPG for us was that we now have a good starting point for running postgres with high availability. (obvs there's still a lot to learn, but CNPG docs is a treasure trove of operational knowledge)
- reacharavindh 3y agoThis is not a snark or a judgemental question. What is a reason someone would want to run Postgres on Kubernetes? I associate Kubernetes with stateless services that can canary deployed, blue-green deployments, auto restarts upon memory crashes, auto scaling etc. But, I cannot think of any practical reason one would want such events on a database that one relies on for keeping the state?! What am I missing? Are there any toy use cases where people are using Postgres for where they can afford(and want) kuberneyes to rolling deploy, and crash restart Postgres instances? In that case, why Postgres in the first place?
- gbartolini 3y agoThe reason could be the last 4 years of evolution in Kubernetes. Have you heard of DoK Community (Data on Kubernetes)? Might be a good place where to start.
- gbartolini 3y agoI'm the author of this blog article in which I try to recap why running Postgres in Kubernetes is a good thing: https://cloudnativenow.com/kubecon-cnc-eu-2022/why-run-postgres-in-kubernetes/ https://cloudnativenow.com/kubecon-cnc-eu-2022/why-run-postg...
- reacharavindh 3y agoI did read the article you linked. It does touch a little bit on potential benefits of having databases managed the same way the services using them are, and integration tests that include the database. However, I’m still scratching my head how any of that is better or not possible with a Postgres installation that is outside of kubernetes. Let’s take out the management part of databases themselves - assume a managed database service like RDS POSTGRES. why would one want to run Postgres on EKS over having their pods on EKS talking to RDS Postgres? I feel like I’m missing some technical reason/advantage that makes all these people choosing to run Postgres on Kunernetes with operators and what not. What does Kubernetes bring to this table over a separate Postgres that dont run the risk of kubernetes interfering with the reliability or operation?
- 3y ago