4 ms·

The many ways of handling TCP RST packets

99 points by luu 11y ago

peterwwillis 11y ago
"What could be a simpler networking concept than TCP's RST packet?" There are twelve RFCs that cover the TCP protocol. They had to write a guide to traversing them all (RFC7414), and that guide references 128 other RFCs. I doubt we can expect any aspect of a particular implementation to be simple. As more products emerge, their functionality diverges and inconsistencies appear. Unless you test your product against someone else's, assume there will probably be unexpected behavior, no matter what spec you're following. (It just gets worse with the intermediary devices, as they're not even a plain-old TCP stack anymore; now they can include a NAT implementation, routing, a firewall, an application proxy... the complexity becomes dizzying)
- feld 11y ago
  These intermediary devices vastly outnumber our routers and are a huge part of the "internet sucks" problem.
userbinator 11y ago
I don't have a huge knowledge of TCP, but IMHO the sequence number should just be ignored for RST --- that half of the connection is unusable for sending data after a RST, so the sequence number is irrelevant. I think the connection should close, unconditionally. The argument about RST spoofing seems somewhat contrived - if you have the means to inject arbitrary packets, chances are you also have the means to read them, so what sequence numbers are being used is known. Any flaw in this reasoning? Despite the diagram showing very different behaviour for all the OSs, the fact that they should be able to successfully communicate with each other, and the Internet is the biggest example of that, says to me that it might not really matter all that much... What would really suck is if an operating system were to totally ignore the rule to reply with an RST when receiving a packet on a closed socket. Isn't this behaviour explicitly encouraged to prevent port scanning and related attacks?
- hwh 11y ago
  "if you have the means to inject arbitrary packets, chances are you also have the means to read them" - this is not true in some (most?) settings. Current mode of operation of the Internet as we know it will allow incoming traffic allow to claim to stem from any public IP address. It's up to the interconnecting parties to ensure that no forged source IP adresses are used on packets leaving their network towards others, but that's all there is at the moment. See http://www.internetsociety.org/doc/addressing-challenge-ip-spoofing http://www.internetsociety.org/doc/addressing-challenge-ip-s...
- jsnell 11y ago
  I don't know of any real TCP stacks that accept all RSTs with no validation checks. The trend has definitely been in the opposite direction... That TCP implementations are interoperable almost all the time is a major feat of design and engineering. For the specific case of RST handling, there's two reasons why it does. First, the RSTs sent in reply to packets received on closed sockets allow the system to converge. That's a clever bit of design, and if it's taken out (as appears to have been the case in the last example), the teardown becomes fragile. Second, if a connection doesn't get closed properly all that happens normally is some resource wastage. The main purpose of the connection (transfering data) worked correctly, so the failure after that won't get noticed.
- toast0 11y ago
  > The argument about RST spoofing seems somewhat contrived - if you have the means to inject arbitrary packets, chances are you also have the means to read them, so what sequence numbers are being used is known. hwh addressed this, but to add more context, there are many places where you can inject spoofed packets with absolutely no visibility into the real data flow. If you're connected to an ISP that doesn't do BCP 38 filtering (for whatever reason), you can send packets to any destination from any source, but can only see packets destined for you (and possibly other people in your/nearby subnets). Even if you are subject to BCP 38 filtering, that may be at the network edge, so you can likely spoof packets from any source to a destination within your network. So, if you have the capability to send a spoofed RST packet and any RST works, you just need to know the connection 4-tuple {Peer1IP, Peer2IP, Peer1Port, Peer2Port} in many common protocols, one of the peer's ip and port is well known, and you may be able to easily determine the other peer's IP and guess the port; 64k packets sent to disrupt someone's IRC connection isn't a lot. If you also have to guess the sequence number, it's a lot harder. >> What would really suck is if an operating system were to totally ignore the rule to reply with an RST when receiving a packet on a closed socket. > Isn't this behaviour explicitly encouraged to prevent port scanning and related attacks? There's a difference between sending a response to a SYN packet, and sending a response to a packet that was sent on a previously established socket. The packet trace shown is really pathological -- because the client is only willing to send one RST, the server is likely to do a lot of retransmits and only figure out the connection is closed when retransmit timers expire; this is wasted effort and resources on the server side, but also for the client's network. At least the client should keep the socket in a FIN_WAIT state for some period of time to send RSTs, but it should also send RSTs when an otherwise unexpected packet comes in that appears to be from an established connection, that way it will stop getting data it can't process. It's reasonable to rate limit sending of RSTs, if you're concerned about DDoS via reflection, or other resource usage issues.
AstroJetson 11y ago
"What could be a simpler networking concept than TCP's RST packet?" I just checked my collection of Steven's TCP/IP Illustrated and there are a couple of pages on RST. So it's not that simple. Kudo's to digging into RST as deeply as you did, I found the article to be a good read. I do think it's very cool that the people that did the RFC's really thought about the corner cases and how the protocol could take care of them.
derFunk 11y ago
I just had a short romance with RST/ACK. It happened during a keepalive http session to AWS ELB, where the nginx backend had keepalive disabled completely due to a misconfiguration. The ELB's idle_time was set to 10seconds (instead of default 60). Just because the idle_time was set to a low value the effect of seeing the RST/ACK could be seen. It needed Wireshark to track this down. On the http level it appeared that a request had been sent, but the received response was empty and didn't even reach nginx at all (because ELB closed the stream).

99 points by luu 11y ago

"What could be a simpler networking concept than TCP's RST packet?" There are twelve RFCs that cover the TCP protocol. They had to write a guide to traversing them all (RFC7414), and that guide references 128 other RFCs. I doubt we can expect any aspect of a particular implementation to be simple. As more products emerge, their functionality diverges and inconsistencies appear. Unless you test your product against someone else's, assume there will probably be unexpected behavior, no matter what spec you're following. (It just gets worse with the intermediary devices, as they're not even a plain-old TCP stack anymore; now they can include a NAT implementation, routing, a firewall, an application proxy... the complexity becomes dizzying)

feld 11y ago
These intermediary devices vastly outnumber our routers and are a huge part of the "internet sucks" problem.

userbinator 11y ago

I don't have a huge knowledge of TCP, but IMHO the sequence number should just be ignored for RST --- that half of the connection is unusable for sending data after a RST, so the sequence number is irrelevant. I think the connection should close, unconditionally. The argument about RST spoofing seems somewhat contrived - if you have the means to inject arbitrary packets, chances are you also have the means to read them, so what sequence numbers are being used is known. Any flaw in this reasoning? Despite the diagram showing very different behaviour for all the OSs, the fact that they should be able to successfully communicate with each other, and the Internet is the biggest example of that, says to me that it might not really matter all that much... What would really suck is if an operating system were to totally ignore the rule to reply with an RST when receiving a packet on a closed socket. Isn't this behaviour explicitly encouraged to prevent port scanning and related attacks?

hwh 11y ago
"if you have the means to inject arbitrary packets, chances are you also have the means to read them" - this is not true in some (most?) settings. Current mode of operation of the Internet as we know it will allow incoming traffic allow to claim to stem from any public IP address. It's up to the interconnecting parties to ensure that no forged source IP adresses are used on packets leaving their network towards others, but that's all there is at the moment. See http://www.internetsociety.org/doc/addressing-challenge-ip-spoofing http://www.internetsociety.org/doc/addressing-challenge-ip-s...
jsnell 11y ago
I don't know of any real TCP stacks that accept all RSTs with no validation checks. The trend has definitely been in the opposite direction... That TCP implementations are interoperable almost all the time is a major feat of design and engineering. For the specific case of RST handling, there's two reasons why it does. First, the RSTs sent in reply to packets received on closed sockets allow the system to converge. That's a clever bit of design, and if it's taken out (as appears to have been the case in the last example), the teardown becomes fragile. Second, if a connection doesn't get closed properly all that happens normally is some resource wastage. The main purpose of the connection (transfering data) worked correctly, so the failure after that won't get noticed.
toast0 11y ago
> The argument about RST spoofing seems somewhat contrived - if you have the means to inject arbitrary packets, chances are you also have the means to read them, so what sequence numbers are being used is known. hwh addressed this, but to add more context, there are many places where you can inject spoofed packets with absolutely no visibility into the real data flow. If you're connected to an ISP that doesn't do BCP 38 filtering (for whatever reason), you can send packets to any destination from any source, but can only see packets destined for you (and possibly other people in your/nearby subnets). Even if you are subject to BCP 38 filtering, that may be at the network edge, so you can likely spoof packets from any source to a destination within your network. So, if you have the capability to send a spoofed RST packet and any RST works, you just need to know the connection 4-tuple {Peer1IP, Peer2IP, Peer1Port, Peer2Port} in many common protocols, one of the peer's ip and port is well known, and you may be able to easily determine the other peer's IP and guess the port; 64k packets sent to disrupt someone's IRC connection isn't a lot. If you also have to guess the sequence number, it's a lot harder. >> What would really suck is if an operating system were to totally ignore the rule to reply with an RST when receiving a packet on a closed socket. > Isn't this behaviour explicitly encouraged to prevent port scanning and related attacks? There's a difference between sending a response to a SYN packet, and sending a response to a packet that was sent on a previously established socket. The packet trace shown is really pathological -- because the client is only willing to send one RST, the server is likely to do a lot of retransmits and only figure out the connection is closed when retransmit timers expire; this is wasted effort and resources on the server side, but also for the client's network. At least the client should keep the socket in a FIN_WAIT state for some period of time to send RSTs, but it should also send RSTs when an otherwise unexpected packet comes in that appears to be from an established connection, that way it will stop getting data it can't process. It's reasonable to rate limit sending of RSTs, if you're concerned about DDoS via reflection, or other resource usage issues.

AstroJetson 11y ago

"What could be a simpler networking concept than TCP's RST packet?" I just checked my collection of Steven's TCP/IP Illustrated and there are a couple of pages on RST. So it's not that simple. Kudo's to digging into RST as deeply as you did, I found the article to be a good read. I do think it's very cool that the people that did the RFC's really thought about the corner cases and how the protocol could take care of them.

derFunk 11y ago

I just had a short romance with RST/ACK. It happened during a keepalive http session to AWS ELB, where the nginx backend had keepalive disabled completely due to a misconfiguration. The ELB's idle_time was set to 10seconds (instead of default 60). Just because the idle_time was set to a low value the effect of seeing the RST/ACK could be seen. It needed Wireshark to track this down. On the http level it appeared that a request had been sent, but the received response was empty and didn't even reach nginx at all (because ELB closed the stream).