3 ms·

Note: there is some justification in the slides; the article itself just lists the "recommendations" outright. (from the article) > net.ipv4.tcp_tw_reuse = 1

Note: there is some justification in the slides; the article itself just lists the "recommendations" outright. (from the article) > net.ipv4.tcp_tw_reuse = 1 (from the man page) > Allow to reuse TIME_WAIT sockets for new connections when it is safe from protocol viewpoint. It should not be changed without advice/request of technical experts. Why? Are sockets in TIME_WAIT a problem somehow? > net.ipv4.ip_local_port_range = 10240 65535 Again, why? My understanding is that this controls the range of ports that the kernel selects from for new sockets; e.g., if you make a TCP connection to google.com on port 443, on _your side_ the connection is <your ip> : <a port from that range>; the default range is [32768, 61000], and this is per destination IP. (You can have two connections to two separate IPs with the same local port.) The default range is nearly 30k ports wide. Are you opening >30k connections to a single host? > In the talk I described these tunables as our medicine cabinet, and to "consider these best before 2015". Does that not mean that these are expired now? (This article was written today, though?)

darkarmani 12y ago
TIME_WAIT: http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-li... "to prevent delayed segments from one connection being accepted by a later connection relying on the same quadruplet (source address, source port, destination address, destination port). The sequence number also needs to be in a certain range to be accepted. This narrows a bit the problem but it still exists, especially on fast connections with large receive windows."
josegonzalez 12y ago
The last point means that while Netflix trusts these recommendations to be good for them today, if you are looking at this in the future, you should verify that the performance gains are still there for those changes. Perhaps you are running in a system where one or more of these no longer gives a performance gain, or is even detrimental to your overall system health. Trust, but verify.
- deleted 12y ago
  [deleted]
brendangregg 12y ago
The problem is opening >30k connections to a single host during a TIME_WAIT period, 60 seconds. 500 connections per second. For backend servers, eg, an application server talking to a database, 500 connections per second is easy (although it's preferable if they can keep-alive). I liked Vincent Bernat's post about TIME_WAIT: http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-li...
- bboreham 12y ago
  Could we say it a bit stronger than that? Opening and closing connections from the same source to the same service is wasteful since you need multiple extra round-trips and it resets all the TCP dynamic tuning like CWND. I thought your talk was great; one minor niggle: you said that the result of too many sockets in TIME_WAIT would be dropped packets; it should refuse to open the new connection if no slots are available.
  - brendangregg 12y ago
    Yes, it's really wasteful to have unnecessary connect()/accept() calls, plus handshaking and buffer allocation, and TCP dynamic tuning, etc. And you're right, thanks, TIME_WAIT full should just error on the Linux client. I was thinking of a different kernel which has bugs in this area, and ends up dropping SYNs...
Dylan16807 12y ago
I'm not sure exactly what state the connections were in, but I've had very obnoxious situations where I shut down a daemon and can't bring it back up quickly because the port it listens on is 'in use'. Harmless-seeming timeouts can cause stupid problems.
caw 12y ago
> Again, why? My understanding is that this controls the range of ports that the kernel selects from for new sockets; e.g., if you make a TCP connection to google.com on port 443, on _your side_ the connection is <your ip> : <a port from that range>; the default range is [32768, 61000], and this is per destination IP. (You can have two connections to two separate IPs with the same local port.) The default range is nearly 30k ports wide. Are you opening >30k connections to a single host? This port range is sometimes known as the "ephemeral port range" and works as you described. How can you have 32 tabs open to news.ycombinator.com port 80? The source port on your machine are all different ports and from that range. The 30K range is not to a single host, it's just all open connections waiting for data to return. That is to say, the connections are established, being established, or being torn down. If the connection was completely torn down you'd be able to reuse the port and there would be no issue. So if your network working set is >28K ports, you may need to change this setting. Most people probably don't need to change this. If you do need to change this because you find your application is throwing errors about binding to ports in use, the above suggestion is fairly decent for setting and forgetting. The one problem with the above suggestion is if you have an application binding to a port somewhere in the range of 10240-32768 or 61001-65535 (http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_number... for examples, it's obviously not complete). You can't just say that 10240-65535 is fair game for ephemeral connections, because inevitably an ephemeral connection will block a known port bind attempt, and your service will fail to run.
- dap 12y ago
  The ephemeral port range is per host because TCP connections are identified by the four-tuple: local IP, local port, remote IP, remote port. You can have active connections from localhost:12345 to both some_server:80 and some_other_server:80 or even some_server:443.
  - caw 12y ago
    Didn't realize it was a 4 tuple, I was thinking of just the binding to the local port. Still, increasing the ephemeral range will cause problems with applications relying on ports outside the default ephemeral range.

darkarmani 12y ago

TIME_WAIT: http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-li... "to prevent delayed segments from one connection being accepted by a later connection relying on the same quadruplet (source address, source port, destination address, destination port). The sequence number also needs to be in a certain range to be accepted. This narrows a bit the problem but it still exists, especially on fast connections with large receive windows."

josegonzalez 12y ago

The last point means that while Netflix trusts these recommendations to be good for them today, if you are looking at this in the future, you should verify that the performance gains are still there for those changes. Perhaps you are running in a system where one or more of these no longer gives a performance gain, or is even detrimental to your overall system health. Trust, but verify.

deleted 12y ago
[deleted]

brendangregg 12y ago

The problem is opening >30k connections to a single host during a TIME_WAIT period, 60 seconds. 500 connections per second. For backend servers, eg, an application server talking to a database, 500 connections per second is easy (although it's preferable if they can keep-alive). I liked Vincent Bernat's post about TIME_WAIT: http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-li...

bboreham 12y ago
Could we say it a bit stronger than that? Opening and closing connections from the same source to the same service is wasteful since you need multiple extra round-trips and it resets all the TCP dynamic tuning like CWND. I thought your talk was great; one minor niggle: you said that the result of too many sockets in TIME_WAIT would be dropped packets; it should refuse to open the new connection if no slots are available.
- brendangregg 12y ago
  Yes, it's really wasteful to have unnecessary connect()/accept() calls, plus handshaking and buffer allocation, and TCP dynamic tuning, etc. And you're right, thanks, TIME_WAIT full should just error on the Linux client. I was thinking of a different kernel which has bugs in this area, and ends up dropping SYNs...

Dylan16807 12y ago

I'm not sure exactly what state the connections were in, but I've had very obnoxious situations where I shut down a daemon and can't bring it back up quickly because the port it listens on is 'in use'. Harmless-seeming timeouts can cause stupid problems.

caw 12y ago

> Again, why? My understanding is that this controls the range of ports that the kernel selects from for new sockets; e.g., if you make a TCP connection to google.com on port 443, on _your side_ the connection is <your ip> : <a port from that range>; the default range is [32768, 61000], and this is per destination IP. (You can have two connections to two separate IPs with the same local port.) The default range is nearly 30k ports wide. Are you opening >30k connections to a single host? This port range is sometimes known as the "ephemeral port range" and works as you described. How can you have 32 tabs open to news.ycombinator.com port 80? The source port on your machine are all different ports and from that range. The 30K range is not to a single host, it's just all open connections waiting for data to return. That is to say, the connections are established, being established, or being torn down. If the connection was completely torn down you'd be able to reuse the port and there would be no issue. So if your network working set is >28K ports, you may need to change this setting. Most people probably don't need to change this. If you do need to change this because you find your application is throwing errors about binding to ports in use, the above suggestion is fairly decent for setting and forgetting. The one problem with the above suggestion is if you have an application binding to a port somewhere in the range of 10240-32768 or 61001-65535 (http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_number... for examples, it's obviously not complete). You can't just say that 10240-65535 is fair game for ephemeral connections, because inevitably an ephemeral connection will block a known port bind attempt, and your service will fail to run.

dap 12y ago
The ephemeral port range is per host because TCP connections are identified by the four-tuple: local IP, local port, remote IP, remote port. You can have active connections from localhost:12345 to both some_server:80 and some_other_server:80 or even some_server:443.
- caw 12y ago
  Didn't realize it was a 4 tuple, I was thinking of just the binding to the local port. Still, increasing the ephemeral range will cause problems with applications relying on ports outside the default ephemeral range.