3 ms·

One thing I never understand about connection pool is how do they handle server-sent close connection (FIN) events? Like I can guarantee there are tons of CLOSE

by est 6y ago

One thing I never understand about connection pool is how do they handle server-sent close connection (FIN) events? Like I can guarantee there are tons of CLOSE_WAIT connections on my near-idle Python process for various of db pools, mysql or redis or pg. For sqlalchemy it just complaints "MySQL server has gone away (error(32, 'Broken pipe'))" and never know how to recover. For redis it just 100% the CPU and keep epoll the dead TCP. Example: https://github.com/andymccurdy/redis-py/pull/886 https://github.com/andymccurdy/redis-py/pull/886

sudhirj 6y ago
I'm not familiar with specific implementations, but from what I've seen with Redis and Postgres clients there's always a reconnect hook of some sort, which I assume pooling proxies give you as well. We've had plenty of broken pipe errors and we just reconnect.
hilbertseries 6y ago
Generally a connection pool also checks the health of its connections. In SQL connection pools this can be done by executing a select 1, if this fails then a new connection can be attempted.
outworlder 6y ago
> Like I can guarantee there are tons of CLOSE_WAIT connections on my near-idle Python process for various of db pools, mysql or redis or pg. Generally your client should close the connection when it's done. Server-side close should mostly be related to idle timeouts.
- est 6y ago
  > Generally your client should close the connection when it's done. Server-side close should mostly be related to idle timeouts. To achieve that, the client must have an extra thread listening for the close event, or a periodic check to purge the dead TCP connections. But there's a gotcha, some TCP connections without KEEPALIVE just timeout naturally, and FIN packets may never reach the client, you only know the connections is dead when you initiate a real request. This is when most connection pool fail apart.

by est 6y ago

sudhirj 6y ago

I'm not familiar with specific implementations, but from what I've seen with Redis and Postgres clients there's always a reconnect hook of some sort, which I assume pooling proxies give you as well. We've had plenty of broken pipe errors and we just reconnect.

hilbertseries 6y ago

Generally a connection pool also checks the health of its connections. In SQL connection pools this can be done by executing a select 1, if this fails then a new connection can be attempted.

outworlder 6y ago

> Like I can guarantee there are tons of CLOSE_WAIT connections on my near-idle Python process for various of db pools, mysql or redis or pg. Generally your client should close the connection when it's done. Server-side close should mostly be related to idle timeouts.

est 6y ago
> Generally your client should close the connection when it's done. Server-side close should mostly be related to idle timeouts. To achieve that, the client must have an extra thread listening for the close event, or a periodic check to purge the dead TCP connections. But there's a gotcha, some TCP connections without KEEPALIVE just timeout naturally, and FIN packets may never reach the client, you only know the connections is dead when you initiate a real request. This is when most connection pool fail apart.