6 ms·
So I thought it would be kind of cool to take Pgbouncer to the next level. I've been using it with large database clusters for a while and it's been spectacular
by levkk 5y ago
So I thought it would be kind of cool to take Pgbouncer to the next level. I've been using it with large database clusters for a while and it's been spectacular, but it does have some limitations that always seemed arbitrary.
This is my take on what a modern Postgres pooler can be. Besides supporting same session and transaction pooling modes as Pgbouncer, it also adds support for load balancing between replicas (round robin at the moment), failover in case a replica fails a health check, and the coolest thing yet I think: sharding at the pooler level.
It's written in Rust, which I think makes it much easier to iterate on and improve.
Note in case it wasn't obvious: this is super experimental, please don't use this in your production environment unless you're adventurous. In which case, please do and let me know how it goes!
- Lev
- dmingod666 5y agoThe performance is currently behind pg_bouncer though.. ofcouse the project is much more early stages, just want your thoughts. Where should new people contribute or can help most ( I do golang and am good with k8s stuff ) all the best!
- dkhenry 5y agoI am really excited to see this project. I can't tell you how many times my team at work has hit a limitation with pgbouncer and swore up and down we should just re-write it in rust. Looking forwards to where the project goes!
- potamic 5y agoSharding and failover support is really cool indeed! But I've always struggled to understand the use cases of pgbouncer. In most cases, you want to front your database with a service and don't want many clients connecting to it directly. And I've seen fairly large workloads easily managed over a small number of connections, so a good client-side pool implementation should easily suffice. In what sort of situations does pgbouncer come into play?
- djbusby 5y agoI've got a PG setup that has a bunch of clients in different languages (shell, Perl, PHP, Python, JS and Go (don't ask)). So, rather than pool in each language, pgBouncer was a quick and easy setup in front - and everyone gets it - even software we're not allowed to modify. E: Oh, and how could I forget all the mod_php based scripts for these web-app front-ends.
- mmontagna9 5y agoSome shops don't design like that :) some folks build giant monoliths and then have to scale/deploy that app with it's database connections broadly.
- suchire 5y agoIt’s pretty easy to exhaust a DB if you have many replicas of a service connecting, each at high concurrency. 20 replicas at 20 connections each is enough to hit limits (like on Cloud SQL for GCP) As you say, you can write a service to stand in front of the DB and mediate access, but PgBouncer is already written AND production ready
- potamic 5y agoAnother reason you would want a service in front is for schema management. I suppose it depends on the use case at hand. Also, 20 connections per service process seems pretty high. Unless there are many long running queries and clients are willing to wait long times for responses.
- bonesmoses 5y agoYou might want to consider a different name, as there's already a pgcat project which appears to be a logical replication extension to augment native logical replication. https://github.com/kingluo/pgcat https://github.com/kingluo/pgcat