7 ms·

Automatically closing FIN_WAIT_2 is a violation of the TCP specification

238 points by jgrahamc 10y ago

wyldfire 10y ago
This is only really a problem when there's a collision of the quadruple identifying the connection. The article indicates as much - "If you want to reproduce this weird scenario consider running this script. It will greatly increase the probability of netcat hitting the conflicted socket". I will admit that I probably wouldn't expect an ETIMEDOUT over a loopback, but I think I would very quickly assume that one side or the other has a bug regarding leaked resources. > It seems that the design decisions made by the BSD Socket API have unexpected long lasting consequences. If you think about it - why exactly the socket can automatically expire the FIN_WAIT state, but can't move off from CLOSE_WAIT after some grace time. This is very confusing... And it should be! One side-effect of having a timeout drive you from CLOSE_WAIT to LAST_ACK without a close() would be that the remote side would not be able to see the application-level reaction to its active close. Determining whether this was a graceful application-level closure would not be possible anymore (though I'll admit I'm not sure how critical that is to the protocol).
- majke 10y ago
  Correct. To be hit by this problem you need leaking sockets AND a client that opens and closes sockets rapidly. What I didn't write is that we had a "bind before connect" on the client side, which means you could hit the CLOSE_WAIT _faster_ than after going through 30k+ sockets. Bind-before-connect sockets have random src port number, therefore depending on setup we hit the colliding socket in a thousand or so sockets, not in 30k+ sockets. Interesting comment on potential implications of automatically transitioning from CLOSE_WAIT.
mcguire 10y ago
Interesting. I would have thought that the server would have sent a reset when it received a SYN on a CLOSE_WAIT socket, not ignored it.
- JoeAltmaier 10y ago
  The OP describes a local-host scenario, which is a special case and could be cleaned up better, as you describe. The CLOSE_WAIT cannot simply be ignored in the general case. The other side could potentially be still out there, lagging badly. Its important to make a really diligent attempt to close, else other more horrible bugs may happen (two endpoints talking to your same port). I'd suggest the stack should never reallocate a port that is not 'clean'. The root of all this confusion is the small port number space (16 bits). Reusing them in a lossy environment (or buggy client environment) can result in all your sockets being in a bad state in a few seconds. Especially with random allocation - its the 'how many people in the room before two have the same birthday' puzzle - the answer is 'not many' before a collision (reallocation of dirty port) is likely. A solution would be to enlarge the port space - e.g. use uuids that don't ever repeat.
  - wyldfire 10y ago
    > The root of all this confusion is the small port number space (16 bits). IMO the leak is the real bug and the protocols work as intended. Keep in mind that the stack(s) distinguish TCP connections not by merely one 16-bit port number but by (local port, local addr, remote addr, remote port). This quadruple is effectively 96 bits of identification. Enlarging the port space has an astonishingly high cost. IMO, it would be easier to just define a new transport protocol or use a network protocol with more addresses (IPv6, e.g.).
    - JoeAltmaier 10y ago
      It may distinguish them (filter them for correctness) but port numbers must be unique across all legal tuples, right? So its still a 16-bit space. And the cost of adding 112 bits to each message is perhaps overstated in this modern world. Messages often dwarf that. Though agreed it would help to use a new protocol - once you have a uuid in the message header, you can design a protocol that omits all the other constant fields (ip etc) because the uuid can be associated at every stage with all that.
      - 10y ago
to3m 10y ago
Also in socket surprises, once you've moved on from CLOSE_WAIT to TIME_WAIT, and doing everything pretty much by the book, you can run out of ephemeral ports and hit this: https://goodenoughsoftware.net/2013/07/15/self-connects/ https://goodenoughsoftware.net/2013/07/15/self-connects/ (One way I like to test multithreaded stuff is to have a test mode where it runs for a while and quits, then run N copies of it at once, repeatedly, and leave the whole thing running for an hour/until it goes wrong. So that's how I encountered this in my case.)
- wyldfire 10y ago
  > Our mistake was the server was running on an ephemeral port. Typical idioms have the server as a connection responder and not an initiator. When the server is a responder it's typically on a well-known port for the sake of the clients knowing where to find it. Having a server on an ephemeral port sounds like a pretty uncommon use case.
  - to3m 10y ago
    One way you can have a listening port in the ephemeral range is to specify 0 as the port when binding, allowing the system to assign the port for you. You might do this if you're using the socket for some kind of parent<->child IPC. (Pipes might be more usual for this sort of thing, but sockets code is a bit easier to make portable between Windows and POSIX, and you also get the option of running both sides on different systems.)
peterwwillis 10y ago
Good flow chart/overview here: https://benohead.com/tcp-about-fin_wait_2-time_wait-and-close_wait/ https://benohead.com/tcp-about-fin_wait_2-time_wait-and-clos...
dap 10y ago
This is an interesting case, but I'm confused about some of the details. > A little known fact is that it's not possible to have any packet loss or congestion on the loopback interface. This seems a bit misleading, given the two counterexamples that the article describes after this. > If you think about it - why exactly the socket can automatically expire the FIN_WAIT state, but can't move off from CLOSE_WAIT after some grace time. This is very confusing... And it should be! On illumos, the FIN_WAIT_2 -> TIME_WAIT transition happens only after 60 seconds if the application has closed the socket file descriptor. In that case, by definition the application has no handle with which to perform operations on the socket. The resource belongs exclusively to the kernel. If the other system disappeared forever, and there were no timeout, that socket would be held open forever. By comparison, in CLOSE_WAIT, the application still has a handle on the socket, and it's responsible for the resource. The application can even keep sending more data in this case (as part of a graceful termination of a higher-level protocol). Or it could enable keep-alive. It's able to respond to the case where the other system has gone away, and it could break the application if the kernel closed the socket on its behalf. I think the behavior is non-obvious, but pretty reasonable.
- majke 10y ago
  >> A little known fact is that it's not possible to have any packet loss or congestion on the loopback interface. > This seems a bit misleading, given the two counterexamples that the article describes after this. Ok, allow me to be more precise. The loopback _link_, the virtual cable, the virtual interface thingy has zero packet loss. If you imagine a wire that represents loopback, it will never congest, never have interference, never have any loss. The packets will _always_ reach the other end. Now, what happens after they reach the other end, that's a separate story. The article indeed showed that it's possible that the kernel network stack dropped packets because of full buffers, or the CLOSE-WAIT thing. But it's the kernel network stack on the receiving end dropping packets actively, not the loopback "wire". > ...and it could break the application if the kernel closed the socket on its behalf. The article showed a case that the other peer just quit alltogether. What kernel thinks about CLOSE-WAIT socket is irrelevant. The connection is _dead_. You can send data over it, you can attempt to read, you can try to close it gracefully, no difference. The other end refused to cooperate. I argue that keeping the socket in CLOSE-WAIT for more than, say, 60 seconds is stupid, since the other party _will_ give up. Basically, any application has 60 seconds to flush the data and call the close() after receiving FIN. After that, it all gets weird. So why not enforce this 60 seconds and just move the socket off CLOSE_WAIT to some next stage? Now, what precisely file descriptor represents is another story. Maybe you're right, maybe a file descriptor can't point to a socket in TIME-WAIT or LAST-ACK states.
coldcode 10y ago
Even today something as basic as TCP can have interesting bugs.
- njharman 10y ago
  TCP is not simple. TCP is a spec, it could have design bugs. There are many implementations of that spec. New ones created, existing ones modified. It should not be surprising that they have bugs.
- gumby 10y ago
  I don't consider this a bug in TCP, it's a design mistake in the BSD socket interface. Nevertheless your point is a good one: even old and heavily-used systems will harbor latent bugs. There are surely unfixed design problems in TCP as well!
theptip 10y ago
Props to CloudFlare for putting out high-quality case studies. I consistently learn at least one new flag/command to add to the quiver every time I read one of these.
bluejekyll 10y ago
I always thought CLOSE_WAIT was due to unsent bytes after a close(). This definitely implies differently than my understanding, I'm surprised the article doesn't discuss SO_LINGER at all, from man pages: >SO_LINGER controls the action taken when unsent messages are queued on > socket and a close(2) is performed. If the socket promises reliable > delivery of data and SO_LINGER is set, the system will block the process > on the close attempt until it is able to transmit the data or until it > decides it is unable to deliver the information (a timeout period, termed > the linger interval, is specified in the setsockopt() call when SO_LINGER > is requested). If SO_LINGER is disabled and a close is issued, the sys- tem will process the close in a manner that allows the process to con- > tinue as quickly as possible. Am I wrong?
- majke 10y ago
  This article describes a case when close() is _not_ being called in the first place. But indeed, it's very interesting what exactly the application can do after it receives FIN. I suspect it has 60 seconds tops to send any remaining data, and SO_LINGER could be useful there.

238 points by jgrahamc 10y ago

This is only really a problem when there's a collision of the quadruple identifying the connection. The article indicates as much - "If you want to reproduce this weird scenario consider running this script. It will greatly increase the probability of netcat hitting the conflicted socket". I will admit that I probably wouldn't expect an ETIMEDOUT over a loopback, but I think I would very quickly assume that one side or the other has a bug regarding leaked resources. > It seems that the design decisions made by the BSD Socket API have unexpected long lasting consequences. If you think about it - why exactly the socket can automatically expire the FIN_WAIT state, but can't move off from CLOSE_WAIT after some grace time. This is very confusing... And it should be! One side-effect of having a timeout drive you from CLOSE_WAIT to LAST_ACK without a close() would be that the remote side would not be able to see the application-level reaction to its active close. Determining whether this was a graceful application-level closure would not be possible anymore (though I'll admit I'm not sure how critical that is to the protocol).

majke 10y ago
Correct. To be hit by this problem you need leaking sockets AND a client that opens and closes sockets rapidly. What I didn't write is that we had a "bind before connect" on the client side, which means you could hit the CLOSE_WAIT _faster_ than after going through 30k+ sockets. Bind-before-connect sockets have random src port number, therefore depending on setup we hit the colliding socket in a thousand or so sockets, not in 30k+ sockets. Interesting comment on potential implications of automatically transitioning from CLOSE_WAIT.

mcguire 10y ago

Interesting. I would have thought that the server would have sent a reset when it received a SYN on a CLOSE_WAIT socket, not ignored it.

JoeAltmaier 10y ago
The OP describes a local-host scenario, which is a special case and could be cleaned up better, as you describe. The CLOSE_WAIT cannot simply be ignored in the general case. The other side could potentially be still out there, lagging badly. Its important to make a really diligent attempt to close, else other more horrible bugs may happen (two endpoints talking to your same port). I'd suggest the stack should never reallocate a port that is not 'clean'. The root of all this confusion is the small port number space (16 bits). Reusing them in a lossy environment (or buggy client environment) can result in all your sockets being in a bad state in a few seconds. Especially with random allocation - its the 'how many people in the room before two have the same birthday' puzzle - the answer is 'not many' before a collision (reallocation of dirty port) is likely. A solution would be to enlarge the port space - e.g. use uuids that don't ever repeat.
- wyldfire 10y ago
  > The root of all this confusion is the small port number space (16 bits). IMO the leak is the real bug and the protocols work as intended. Keep in mind that the stack(s) distinguish TCP connections not by merely one 16-bit port number but by (local port, local addr, remote addr, remote port). This quadruple is effectively 96 bits of identification. Enlarging the port space has an astonishingly high cost. IMO, it would be easier to just define a new transport protocol or use a network protocol with more addresses (IPv6, e.g.).
  - JoeAltmaier 10y ago
    It may distinguish them (filter them for correctness) but port numbers must be unique across all legal tuples, right? So its still a 16-bit space. And the cost of adding 112 bits to each message is perhaps overstated in this modern world. Messages often dwarf that. Though agreed it would help to use a new protocol - once you have a uuid in the message header, you can design a protocol that omits all the other constant fields (ip etc) because the uuid can be associated at every stage with all that.
    - 10y ago

to3m 10y ago

Also in socket surprises, once you've moved on from CLOSE_WAIT to TIME_WAIT, and doing everything pretty much by the book, you can run out of ephemeral ports and hit this: https://goodenoughsoftware.net/2013/07/15/self-connects/ https://goodenoughsoftware.net/2013/07/15/self-connects/ (One way I like to test multithreaded stuff is to have a test mode where it runs for a while and quits, then run N copies of it at once, repeatedly, and leave the whole thing running for an hour/until it goes wrong. So that's how I encountered this in my case.)

wyldfire 10y ago
> Our mistake was the server was running on an ephemeral port. Typical idioms have the server as a connection responder and not an initiator. When the server is a responder it's typically on a well-known port for the sake of the clients knowing where to find it. Having a server on an ephemeral port sounds like a pretty uncommon use case.
- to3m 10y ago
  One way you can have a listening port in the ephemeral range is to specify 0 as the port when binding, allowing the system to assign the port for you. You might do this if you're using the socket for some kind of parent<->child IPC. (Pipes might be more usual for this sort of thing, but sockets code is a bit easier to make portable between Windows and POSIX, and you also get the option of running both sides on different systems.)

peterwwillis 10y ago

Good flow chart/overview here: https://benohead.com/tcp-about-fin_wait_2-time_wait-and-close_wait/ https://benohead.com/tcp-about-fin_wait_2-time_wait-and-clos...

dap 10y ago

This is an interesting case, but I'm confused about some of the details. > A little known fact is that it's not possible to have any packet loss or congestion on the loopback interface. This seems a bit misleading, given the two counterexamples that the article describes after this. > If you think about it - why exactly the socket can automatically expire the FIN_WAIT state, but can't move off from CLOSE_WAIT after some grace time. This is very confusing... And it should be! On illumos, the FIN_WAIT_2 -> TIME_WAIT transition happens only after 60 seconds if the application has closed the socket file descriptor. In that case, by definition the application has no handle with which to perform operations on the socket. The resource belongs exclusively to the kernel. If the other system disappeared forever, and there were no timeout, that socket would be held open forever. By comparison, in CLOSE_WAIT, the application still has a handle on the socket, and it's responsible for the resource. The application can even keep sending more data in this case (as part of a graceful termination of a higher-level protocol). Or it could enable keep-alive. It's able to respond to the case where the other system has gone away, and it could break the application if the kernel closed the socket on its behalf. I think the behavior is non-obvious, but pretty reasonable.

majke 10y ago
>> A little known fact is that it's not possible to have any packet loss or congestion on the loopback interface. > This seems a bit misleading, given the two counterexamples that the article describes after this. Ok, allow me to be more precise. The loopback _link_, the virtual cable, the virtual interface thingy has zero packet loss. If you imagine a wire that represents loopback, it will never congest, never have interference, never have any loss. The packets will _always_ reach the other end. Now, what happens after they reach the other end, that's a separate story. The article indeed showed that it's possible that the kernel network stack dropped packets because of full buffers, or the CLOSE-WAIT thing. But it's the kernel network stack on the receiving end dropping packets actively, not the loopback "wire". > ...and it could break the application if the kernel closed the socket on its behalf. The article showed a case that the other peer just quit alltogether. What kernel thinks about CLOSE-WAIT socket is irrelevant. The connection is _dead_. You can send data over it, you can attempt to read, you can try to close it gracefully, no difference. The other end refused to cooperate. I argue that keeping the socket in CLOSE-WAIT for more than, say, 60 seconds is stupid, since the other party _will_ give up. Basically, any application has 60 seconds to flush the data and call the close() after receiving FIN. After that, it all gets weird. So why not enforce this 60 seconds and just move the socket off CLOSE_WAIT to some next stage? Now, what precisely file descriptor represents is another story. Maybe you're right, maybe a file descriptor can't point to a socket in TIME-WAIT or LAST-ACK states.

coldcode 10y ago

Even today something as basic as TCP can have interesting bugs.

njharman 10y ago
TCP is not simple. TCP is a spec, it could have design bugs. There are many implementations of that spec. New ones created, existing ones modified. It should not be surprising that they have bugs.
gumby 10y ago
I don't consider this a bug in TCP, it's a design mistake in the BSD socket interface. Nevertheless your point is a good one: even old and heavily-used systems will harbor latent bugs. There are surely unfixed design problems in TCP as well!

theptip 10y ago

Props to CloudFlare for putting out high-quality case studies. I consistently learn at least one new flag/command to add to the quiver every time I read one of these.

bluejekyll 10y ago

I always thought CLOSE_WAIT was due to unsent bytes after a close(). This definitely implies differently than my understanding, I'm surprised the article doesn't discuss SO_LINGER at all, from man pages: >SO_LINGER controls the action taken when unsent messages are queued on > socket and a close(2) is performed. If the socket promises reliable > delivery of data and SO_LINGER is set, the system will block the process > on the close attempt until it is able to transmit the data or until it > decides it is unable to deliver the information (a timeout period, termed > the linger interval, is specified in the setsockopt() call when SO_LINGER > is requested). If SO_LINGER is disabled and a close is issued, the sys- tem will process the close in a manner that allows the process to con- > tinue as quickly as possible. Am I wrong?

majke 10y ago
This article describes a case when close() is _not_ being called in the first place. But indeed, it's very interesting what exactly the application can do after it receives FIN. I suspect it has 60 seconds tops to send any remaining data, and SO_LINGER could be useful there.