2 ms·
Single biggest TCP issue I have had to debug and fix numerous times is about not doing connection reuse properly leading to tcp port exhaustion and causing seem
by vinay_ys 6y ago
Single biggest TCP issue I have had to debug and fix numerous times is about not doing connection reuse properly leading to tcp port exhaustion and causing seemly random delays causing timeout failures at higher level protocols, usually http. This one single issue has taken down multi-billion dollar production systems.
So, I hope people learn to check their http client/server implementations to have proper connection handling. Client should have a thoughtfully sized bounded connection pool with reasonably large idle timeout. It shouldn't close the connection after every application request (say, http request). There shouldn't be sockets in TIME_WAIT state accumulating at the client end.
Server should accept thoughtfully limited number of connections per client. Server should never close the connection except when it is shutting down.
There should be tcp keepalive messages to keep the connection alive with intermediate hop stateful firewalls (connection tracking table entries in firewalls expire when the connection is idle for too long) and to detect stale connections and re-establish them.
All of these things can be verified by analyzing at a packet capture. You can get a manageable sized pcap file by filtering on client/server ip/port-range pairs for at least 330 seconds.
Knowing tools to understand/debug tcp issues is an essential skill. sock stat command - ss, wireshark/tshark with Lua scripting is super useful. Knowing higher level application protocols like TLS and http is essential too.