9 ms·
Connection management in Postgres is definitely one of its sore spots. Each connection even an idle one consumes roughly 10 MB of overhead and introduces additi
by craigkerstiens 8y ago
Connection management in Postgres is definitely one of its sore spots. Each connection even an idle one consumes roughly 10 MB of overhead and introduces additional coordination overhead on queries. If you're running at any large scale production you want pgbouncer in place which can help manage the idle connections so they're not a hit to your system. Hopefully one day this improves directly in core.
- anarazel 8y ago> Each connection even an idle one consumes roughly 10 MB of overhead Citation needed :P Unless you're using a large shared_buffers setting without configuring huge pages, you should see smaller overhead. Obviously depends on what you're doing in each connection - if you consider accessing 10k tables in each session part of the connection overhead, sure, it'll be bigger. > introduces additional coordination overhead on queries. It's not really the established connection that matters. The number of in-progress transactions matters considerably (a snapshot is O(#active-transactions)). There's some influence of the maximum number of possible connections.
- jashmatthews 8y agoProduction example: per-process memory usage at ChartMogul is >70MB because we use a huge number of tables with partitioning.
- jpitz 8y ago> Citation needed :P There are plenty of citations out there, but Craig wrote most of them.
- iKSv2 8y agopgbouncer or pgpool (of the extensions I searched) do this. Which ones are more of a challenge? I mean I have read on some forum that PGPOOL -II would just shutdown and stop or misbehave. Any such experiences or gotchas with pgbouncer?
- rosser 8y agoWe use pgbouncer with pools of thousands of concurrent connections. It does what it says on the tin.
- deleted 8y ago[deleted]
- madisonhope 8y agoWe have experimented with both at scale and found PgBouncer to be a lot more stable and reliable. We haven't seen gotcha failures, or any failures really, in the last few years. It's also refreshingly simple to automate and scale, although there are shortcomings in the configuration design that ends up requiring some cleverness.
- iKSv2 8y ago"although there are shortcomings in the configuration design that ends up requiring some cleverness." Like?
- madisonhope 8y agoNamely, it doesn't support configuration via CLI or env vars, which means you have no option but to generate a file to be loaded in. To be fair, that's kind of a newer pattern. Also, multi-DB PgBouncer is a little more insecure than it leads on to be by default, along with a few other things you learn quickly as you handle larger and more complex use cases. It's not bad, just not as good as it could be in my opinion.
- tshannon 8y agoSorry if this is a stupid question, but don't most libraries do connection pooling on their own? Why is something like pgbouncer necessary?
- bobwaycott 8y agoMost libraries do not handle connection pooling on their own. They simply handle opening and closing a connection per query. Hence the write up on pgbouncer being a (relatively) transparent, drop-in solution to handling this when a library does not.
- merb 8y agowell transaction pooling is good for most databases. consider having 10 upstream connections per transaction pool and 10000s of connections to applications. pgbouncer can handle that and every application would only be assigned to a real database if a transaction/query happens and after that the upstream connection would be deassigned.