19 ms·
PostgreSQL 14 on Kubernetes
- leetrout 5y agoWhy would i want to run my database in k8s?
- voidfunc 5y agoWhy not? Its just a workload scheduler. Some of us have competent ops teams
- longcommonname 5y agoWhat level of competency do you need to run dbs?
- darthrupert 5y agoPoorly? Almost none. Well? Significant.
- edoceo 5y agoUnhelpful and arrogant. We are amateurs in the eyes of the gods.
- dijit 5y agoIt’s very easy to set something up for the first time. It’s a bit complex to keep something running for a long time. It’s very difficult to make something fault tolerant with no data loss or corruption. Arrogance is thinking that these things are easy because you can read a quickstart.
- tinus_hn 5y agoIt is true though, but it is also true that a ‘poorly’ run database server is good enough for many use cases. There is a reason people are DBAs and they are very well paid.
- janto 5y agoArrogant and foolish to declare no-one can be high skilled.
- darthrupert 5y agoYou're right. But so am I.
- ryanisnan 5y agoI think while true, this is an incredibly reductive take on what Kubernetes really is.
- manigandham 5y agoBecause kubernetes is a fantastic platform that provides a lot of node and app management functionality out of the box. Stateful services like databases are not really a challenge anymore given recent developments in both K8S and database containerization.
- tiew9Vii 5y agoWith a database I generally want to have it on its own dedicated host with fast NVME drives, memory, cpu etc, maybe some OS specific system level configuration options for performance etc. I don't want to binpack it on a cluster of nodes each contending for system resources. Also RDBMS databases are not really designed to be ephemeral/elastic, they are stateful/persistent by nature. What does Kubernetes offer in this scenario? I'm still hesitant to run things like DB's on K8's but have absolutely no problems running cloud native apps on K8's which can run ephemerally, be bin packed and only needing compute as all state is external to where the app is being executed.
- alphabettsy 5y agoStatefulSets, Node Groups, and Node Affinity/Taints solve some of those issues.
- karolist 5y agoA database is not a service well suited for ephemeral, quick scale down and scale up container workloads that K8s really excells at, and if you have nodes specifically configured to run databases and nothing else then you should simply run them there directly and not mangle K8s to replicate what will essentially be equivalent to running them there directly.
- manigandham 5y agoK8S runs workloads across servers in a declarative fashion. Ephemeral or stateful doesn't make a difference anymore. Whatever you use to manage the nodes directly is basically what K8S provides (and more) already, so you're really just replicating effort and complexity instead.
- evil-olive 5y agoif you run your stateless services on an orchestration system like k8s, you have two options: a) run your stateful services (such as Postgres) on the same orchestration system b) choose a second orchestration system (often a more "traditional" infrastructure provisioning system like Chef/Puppet/Ansible/SaltStack) that will only manage your stateful services, and deal with the care & feeding & mental overhead of having two things that do fairly similar jobs and need to interact with each other (because of course your k8s-hosted stateless services need to be able to look up Postgres in DNS) (option C would be to use a managed service like RDS...but options A and B assume you've already made a decision, for whatever reason, to run the stateful services yourself)
- morelisp 5y agoIf you choose a) you don't really get to avoid b), since something still has to bootstrap your Kubernetes nodes themselves. So it's more a question whether you'd rather write a Kubernetes manifest or a Salt/Ansible config; and with the complexity and limitations of a lot of storage operators, and no real "orchestration" benefit once you have pinned deployments to particular nodes anyway, sometimes it's the latter. The main force that's pushing us to move our storage into K8s isn't simplifying our node configuration, but to keep parity between our prod and more dynamic staging environments. For the latter k8s storage operators are great because we don't generally care about data durability or performance.
- cdogl 5y ago> If you choose a) you don't really get to avoid b), since something still has to bootstrap your Kubernetes nodes themselves Not quite true if you use a managed K8S offering - and it seems a self-managed cluster is something to be avoided until absolutely necessary. Even with a managed cluster like GKE there's plenty of operational overhead to keep everything up-to-date and working properly.
- morelisp 5y agoIf you're running on a managed K8s offering yet not using the managed storage, you're getting ripped off. (I'm well aware this describes a lot of companies / projects.)
- dagw 5y agoAt least in my case, because your ops team has decided that everything that gets deployed, gets deployed in k8s (unless you have a Very! good reason not to)
- dijit 5y agoKubernetes is great but not everything fits in kubernetes, this is a bad ops team if this is the case.
- jpgvm 5y agoNote the "except if you have a very good reason" caveat. I think this is a good ops team mantra. PostgreSQL unless you have a very good reason otherwise, and k8s unless you have a very good reason otherwise. Decisions like this allow for economies of scale with tooling, monitoring/observability, etc. Saying no to snowflakes is an important part of having a good ops team.
- dijit 5y agoGuess it depends on who is defining “very”. A persistent connection is antithetical to the design of kubernetes. Persistent storage is hard to get right in kubernetes and puts a lot of pressure on the cluster being perfect, which is something a good ops team will tell you is hard. If your application is stateless http services then kubernetes is great, but there are common things that don’t fit well, and forcing people into a solution which may require a complete rethinking is not very pragmatic.
- dagw 5y agoEh, for what it's worth they've deployed a couple of PostgreSQL instances for me using Kubernetes and it's worked exactly like I've wanted it to. But then again I've never had push those instances to breaking point either when it comes to IO or concurrency so who knows.
- tveita 5y agoA middle ground would be provisioning managed services through Kubernetes resources using a controller like Config Connector (GCP) or ACK (AWS) https://cloud.google.com/config-connector/docs/reference/resource-docs/sql/sqlinstance https://cloud.google.com/config-connector/docs/reference/res... https://aws-controllers-k8s.github.io/community/reference/rds/v1alpha1/dbinstance/ https://aws-controllers-k8s.github.io/community/reference/rd...
- hijinks 5y ago20 years ago people said the same thing about running DBs on virtual machines. Now almost all databases are run on VMs
- mbreese 5y agoBut that’s really not a great comparison, is it? Running on a VM is much more like running on a plain physical node. Running on K8s implies a whole new level of management overhead, that just isn’t there on a single node system. In this case, you should be asking “will we all be running databases in clustered mode in the future?” I think that answer still up for debate.
- jatone 5y agocontainers are actually a really good fit for dbs. they do one thing and one thing only and for a really really long time without many changes. and the answer is we will definitely be running DBs in clustered mode in the future. its not really a debate its an eventuality. without clusters you lose on reliability and maintenance QOL improvements. without clusters its very hard to upgrade systems effectively leaving systems to stagnate and decay both in performance (Hard/Software improvements aka kernel updates or inbternal DB improvements) and reliability.
- morelisp 5y agoThe question isn't whether DBs will be running in clusters - they all pretty much do, or can - but will those clusters be orchestrated by something first designed for dynamic scaling of stateless services, treating all nodes as varying-sized pools of otherwise mostly-homogeneous hyperconverged compute/storage? Or, how much are you willing to pay in $/performance/dev time to force them into such infrastructure? Or will they be on something more purpose-built, either designed from the start to handle storage more flexibly at a higher level? Or conversely something more specific and lower-level to extract maximum performance for their particular storage patterns?
- gizdan 5y ago> Running on a VM is much more like running on a plain physical node. Sure, and containers are just plain processes on a VM. The exact code that controls processes in a VM, also controls processes inside a container. As long as the underlying data volume is reliable, it really doesn't matter whether you're running it outside of a container or not. Volumes for K8s have a come a long way since its early days.
- rafaelturk 5y agoWe run all our databases in K8S, approx 80 DBs. We're a fintech that requires HA so all databases run with at least 3 notes. Maging DB clusters with K8S only requires 1 file. If we need to increase storage this can be achieved in minutes, if we need to take one node down we basically can do it anytime during the day.. Every now and then we discuss moving DBs to AWS RDS (or equivalent) however for our transaction volume RDS costs are prohibitive.
- cbushko 5y agoWhat DB are you running?
- dijit 5y agoI just checked your comment history and it appears that you like to troll. How can I know that this isn't one of those cases? I know some folks in fintech and while obviously they're not _you_, they're definitely not the sort to be wishy washy with data, if anything they're quite conservative when it comes to how the databases are managed. Any incidental complexity is incredibly discouraged.
- pixel_tracing 5y agoI think when I see tutorials like this all I see is the large bill I’ll be receiving $$$, can articles like these include costs??
- jiggawatts 5y agoYour costs will vary wildly depending on the cloud provider, VM sizes used, data volume, etc... A small database might be just $5 per month if its a thin slice of a big shared Kubernetes pie. Or it could be $500K per month if it's a huge set of replicated server storing petabytes, replicated for dev, test, and prod.
- mpwala 5y agoI always thought it's better to host distributed database, like Cockroachdb and Yugabytedb, in multiple kubernetes clusters. Just in case one of the cluster goes down.
- manigandham 5y agoA K8S cluster is already designed to be highly available and self healing. Stretching across clusters is usually a special case of scaling across datacenters/regions and brings a lot more overhead and complexity.
- hsbauauvhabzb 5y agoI’ll bite. Any good reading on highly available multi-region failover?
- manigandham 5y agoKubernetes doesn't "failover" like that. Each cluster does it's best to keep your workloads running but stretching across multi-clusters is your responsibility. There is K8S Federation [1](previously called multicluster) which is about deploying from a single host cluster to multiple other ones, but it's still unfinished, and either way K8S doesn't know much about your application itself to be able to perform any kind of failover switch. Usually if it's a stateless app then the solution is using a global load balancer or other networking at the edge to steer traffic to the responsive regions. Databases also tend to have their own replication and distribution strategies handled internally that should be used, along with the above mentioned networking. 1. https://github.com/kubernetes-sigs/kubefed https://github.com/kubernetes-sigs/kubefed
- aliswe 5y agoyeah, theres a point for anyyone where they would say "i accept that level of risk". for example, datacenter outages could be where many companies simply tap out.
- mikesabbagh 5y agocant think of a scenario where you need to query multiple databases in same query! Nice demo, but deploying it all in a cluster is distracting. Maybe do your next demo with docker-compose!
- tsimionescu 5y ago> Nice demo, but deploying it all in a cluster is distracting. Maybe do your next demo with docker-compose! You're suggesting that next time they should demonstrate Postgres on Kubernetes without the distracting Kubernetes part?
- whazor 5y agoI would like to see a comparison between this operator and the one from Zalando.
- csunbird 5y agolink to the zalando postgres operator: https://github.com/zalando/postgres-operator https://github.com/zalando/postgres-operator
- ___luigi 5y agoI would like to see a comparison between this operator and Stolon.
- sergioisidoro 5y agoI tried both while testing things for a personal project. In my very not exhaustive user perspective, I like Crunchy better. Zalando seems to fit the Zalando use case (not surprisingly), which is having multiple teams self-service their postgres clusters. They have a nice Web ui that is good for people who are not familiar with k8 to provision databases. However a lot of those features are a bit intrusive when you want to start a small project. The "teams" attributes are required in the clusters, for example, although you're not using them. I ended up using Cruncy for my personal project because its a bit simpler, looked like having a bit more activity on Github, and slightly better documentation from my perspective.
- rossmohax 5y agoBest way to run database is in static pods. You'll get all the benefit of kubernetes ecosystem (monitoring, logs, inventory, access control) without any drawbacks.
- peterthehacker 5y agoYou mean without a deployment or statefulset? How would you connect to the DB without a service providing a persistent DNS name?
- merb 5y agoyou can create a static pod manifest in /etc/kubernetes/manifests (https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/ https://kubernetes.io/docs/reference/setup-tools/kubeadm/imp...) these manifests can be targeted with a Service. However my preferred solution for databases inside k8s is still zalando-operator with spilo and local storage. it's rock solid. I once used kubedb and lost the database because of bugs, so I'm staying away from them. Crunchy looks solid aswell but it looks like it's a little bit more polished but harder to configure than zalando's solution which is more built to their needs.
- rafaelturk 5y agoYou can create a service point to the `pod` tag. Or a manifest.
- brainwipe 5y agoThis is an excellent, well explained article. For most of the small companies I work for, database hosting is treated like a utility so we end up using RDS etc. The benefit of arranging your own k8 database setup is outweighed by the cost of maintenance. On one hand it's nice that it's one less thing to worry about, on the other I miss this kind of deep nerdery.
- strzibny 5y agoWhile I am happy for the progress people make in bringing databases to container orchestrators, I don't see myself going off the dedicated virtual machine for the database. I find it easier/safer to run it this way. Shameless plug: if you want to start running your own PostgreSQL with SSL, SELinux, automatic system updates, attached storage, and more... I have a simple demo as part of https://deploymentfromscratch.com/ https://deploymentfromscratch.com/.
- vfiftyfive 5y agoThe truth is Kubernetes already has all the primitives to run database or any other stateful applications. What is challenging is how to make it reliable, performant and scalable for this kind of apps. If you manage to do it properly, then it will be far less costly and manageable than any DBaaS out there. There are 2 dimensions to the problem: 1) How to easily deploy and operate a "production" DB in K8S. As the toolset evolves, we've seen a lot of different operators and customization tools to facilitate this. Today it is rather trivial to write the manifests that will deploy a Redis cluster fit to your needs. 2) You need a REALLY good kubernetes-native data service layer that guarantees fast failover, sync replication and performance for your persistent volumes. StatefulSets alone won't really help in case of node failure...and don't even think about using NFS. Ondat/StorageOS are leaders in that space, it's really simple to deploy, so you can give it a try in less than 5min. Once you've made sure that your stateful configuration is right and that the network/data layer is reliable, then you can leverage all the visibility tools native to K8S (ELK/EFK, Prometheus, etc). Your stateful apps will be ready for CI and other DevOps integration using the K8S tools (Tekton, ArgoCD etc). This will make your devs happy as infra is not the blocker anymore...
- gigatexal 5y agowhile you can -- and I've worked at shops that have -- and there are DBs like Cockroach that are k8s native... I still wouldn't. Give me a some VMs and I'll create a cluster. Sure it will be overprovisioned and such but it's a database. It needs to be solid and consistent and slow to change. I still think it's best to allow K8s to coordinate ephermeral containers with little to no persistent storage and let the DBs be run more traditionally.
- tw04 5y agoI have to ask, do you have an interest in ONDAT/StorageOS? Because you say they're a leader and yet I've never run into them, your account is two weeks old, and I see they've just started a twitter account, so it all kind of smells like a PR push.
- ltbarcly3 5y agoRunning anything which depends on disk cache 'working' should not be run in a shared environment, unless your application can run arbitrarily slowly. If I understand how it works, allocating memory to a container does not allocate memory for caching of data, and any IO done by any container can eject data from the cache, which means any IO intensive process will step on any other.
- aliswe 5y agowhat's the minimum resource requirements for a minimal 1 micro-sized table, and very low traffic? Is it in the magnitude of 100s of MB and tens of a vcore ? or .. ?