4 ms·

TCP is UNreliable

26 points by LiveTheDream 13y ago

tptacek 13y ago
I voted this up (tried to submit, actually), but is this article correct, or is something in this person's stack not actually programming sockets correctly? When you call shutdown() to arrange a FIN for a socket, especially after you've sent data and received no application-layer acknowledgement, you're supposed to verify that the connection closed properly. You do this by select()ing on the socket and then read()ing 0 bytes. The error condition (the RST segment they're getting) is signaled by the "Connection closed by peer" errno --- and -1 from read(). The reliability signals this app needs are built into both TCP and the socket programming interface, from what I can tell. The application-layer acknowledgement it wants appears duplicative.
- abofh 13y ago
  Without looking at the code, it's hard to say, typically errors like the one described won't be caught by return values of read/write, but by return values of close() -- which many many people don't bother to check. (Not saying that's what happened, but that's my bet)
  - tptacek 13y ago
    Did they close() the socket? If you write data to a socket, and receive no signal that it's been processed, shouldn't you shutdown(2), select(2), then read(2) before you close(2) it?
  - deleted 13y ago
    [deleted]
- BetaCygni 13y ago
  > after you've sent data and received no application-layer acknowledgement This would be the most robust fix. Have the server return "OK" or something after it has processed the data. It could have silently crashed but still accept connections.
  - tptacek 13y ago
    I'm not sure I see how this is more robust than simply checking to make sure you had an orderly close. The orderly close check should also catch the case where the application crashes after its TCP stack has received the data but before the application processed it. I'm not saying there's never any point to application acknowledgements; look at any careful SMTP server implementation for a study of how to handle these problems. TCP can promise a reliable transport but obviously not a reliable disk.
    - noselasd 13y ago
      Though, the contract you have with TCP , is between TCP at your end and the TCP at the peer, not that it also have delivered the data to the application of the peer. If the peer crashes just as it read() the data, but didn't process it, it will look just as a normal clean close on the sender machine if there arn't more data laying around in the TCP buffers.
      - tptacek 13y ago
        To be clear, you're talking about the case where the application dequeues the data from its receive queue and then crashes. If you close a socket without reading queued data, you get an RST.
- oesmith 13y ago
  > The reliability signals this app needs are built into both TCP and the socket programming interface, from what I can tell. The application-layer acknowledgement it wants appears duplicative. An ACK only suggests that a segment has reached the recieve buffer successfully. If the receiving application crashes between buffering incoming data and successfully processing it, that's expressly not TCP's problem. The RST will signal that the receiver has died, but does not imply whether or not the application has successfully processed any buffered data up to that point. AFAIK, an application-layer acknowledgement is the only way to solve this problem.
  - tptacek 13y ago
    I agree that there are circumstances where you need application-layer acks, SMTP being a canonical example. However, there are other cases where you don't need lock-step app-layer acknowledgements; for instance, if your messages are idempotent. I'm really just reacting to the forensic work in this article, which suggests (maybe I'm misreading) a problem that TCP doesn't actually have.
    - oesmith 13y ago
      Yes, if all you want is to confirm that a TCP connection has closed and flushed correctly, then AFAIK, that's entirely possible with a [shutdown, select, read] sequence. I struggle to understand why an RPC system (as detailed in the article) wouldn't want an app-level acknowledgement though...
notacoward 13y ago
This doesn't sound like a TCP problem. Mistake #1: The sender optimistically sends a first packet before getting a SYN reply. That implies that the sender issued a non-blocking connect, and then wrote without waiting for the connect to complete. Otherwise, the sender (kernel) wouldn't have any data to send right away. Mistake #2: The sender sends a FIN, but doesn't block because all data has already been acknowledged. This is supposedly a problem because the app never receives the data, but a TCP ack has never meant that data got beyond the receiving kernel to an app. So yes, it's true that neither side ever got an error and unfortunate that it's possible for the receiving app to get no data in that case, but that doesn't make it a TCP problem. Somebody at a higher level - JVM or the app itself - over-optimized their "connection and first data" code without understanding the actual behavior of the TCP protocol and socket API. Then they "fixed" it by working with their own protocol instead of against the standard one. Nothing to see here. It was just a false alarm. Move on.
- gonzo 13y ago
  It's not a TCP problem. It's a problem with the linux implementation of TCP.
  - noselasd 13y ago
    No, it's not even that. The linked article (http://www.cliffc.org/blog/2013/08/15/tcp-is-unreliable/ http://www.cliffc.org/blog/2013/08/15/tcp-is-unreliable/) at the bottom tells you what's actually going on. The only thing slightly fishy is that the peer application didn't receive any connection attempt, but if you're anyway quite overloaded, e.g. running out of file descriptors, that may well happen.
    - notacoward 13y ago
      The FIN/SO_LINGER problem is real, and often misunderstood, but I don't think that was quite the whole story here. SO_LINGER would flag the error some of the time, but not always (see Mistake #2). Even if it did, avoiding the problem altogether (see Mistake #1) would be better than flagging it after the fact. Even with SO_LINGER on and the kernel's RST issues fixed, the app would still be erroneous.
songgao 13y ago
Interesting! One thing that I don't understand (which might be a little off-topic): If it's an RPC call, shouldn't there be a response for each RPC request to ack that the RPC request is well processed by the application layer? Otherwise how do you make sure the application layer won't screw up? How do you notify the client that the request is not invalid?
noselasd 13y ago
Note that he links to http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-s... which more or less explains what's going on.

26 points by LiveTheDream 13y ago

I voted this up (tried to submit, actually), but is this article correct, or is something in this person's stack not actually programming sockets correctly? When you call shutdown() to arrange a FIN for a socket, especially after you've sent data and received no application-layer acknowledgement, you're supposed to verify that the connection closed properly. You do this by select()ing on the socket and then read()ing 0 bytes. The error condition (the RST segment they're getting) is signaled by the "Connection closed by peer" errno --- and -1 from read(). The reliability signals this app needs are built into both TCP and the socket programming interface, from what I can tell. The application-layer acknowledgement it wants appears duplicative.

abofh 13y ago
Without looking at the code, it's hard to say, typically errors like the one described won't be caught by return values of read/write, but by return values of close() -- which many many people don't bother to check. (Not saying that's what happened, but that's my bet)
- tptacek 13y ago
  Did they close() the socket? If you write data to a socket, and receive no signal that it's been processed, shouldn't you shutdown(2), select(2), then read(2) before you close(2) it?
- deleted 13y ago
  [deleted]
BetaCygni 13y ago
> after you've sent data and received no application-layer acknowledgement This would be the most robust fix. Have the server return "OK" or something after it has processed the data. It could have silently crashed but still accept connections.
- tptacek 13y ago
  I'm not sure I see how this is more robust than simply checking to make sure you had an orderly close. The orderly close check should also catch the case where the application crashes after its TCP stack has received the data but before the application processed it. I'm not saying there's never any point to application acknowledgements; look at any careful SMTP server implementation for a study of how to handle these problems. TCP can promise a reliable transport but obviously not a reliable disk.
  - noselasd 13y ago
    Though, the contract you have with TCP , is between TCP at your end and the TCP at the peer, not that it also have delivered the data to the application of the peer. If the peer crashes just as it read() the data, but didn't process it, it will look just as a normal clean close on the sender machine if there arn't more data laying around in the TCP buffers.
    - tptacek 13y ago
      To be clear, you're talking about the case where the application dequeues the data from its receive queue and then crashes. If you close a socket without reading queued data, you get an RST.
oesmith 13y ago
> The reliability signals this app needs are built into both TCP and the socket programming interface, from what I can tell. The application-layer acknowledgement it wants appears duplicative. An ACK only suggests that a segment has reached the recieve buffer successfully. If the receiving application crashes between buffering incoming data and successfully processing it, that's expressly not TCP's problem. The RST will signal that the receiver has died, but does not imply whether or not the application has successfully processed any buffered data up to that point. AFAIK, an application-layer acknowledgement is the only way to solve this problem.
- tptacek 13y ago
  I agree that there are circumstances where you need application-layer acks, SMTP being a canonical example. However, there are other cases where you don't need lock-step app-layer acknowledgements; for instance, if your messages are idempotent. I'm really just reacting to the forensic work in this article, which suggests (maybe I'm misreading) a problem that TCP doesn't actually have.
  - oesmith 13y ago
    Yes, if all you want is to confirm that a TCP connection has closed and flushed correctly, then AFAIK, that's entirely possible with a [shutdown, select, read] sequence. I struggle to understand why an RPC system (as detailed in the article) wouldn't want an app-level acknowledgement though...

notacoward 13y ago

This doesn't sound like a TCP problem. Mistake #1: The sender optimistically sends a first packet before getting a SYN reply. That implies that the sender issued a non-blocking connect, and then wrote without waiting for the connect to complete. Otherwise, the sender (kernel) wouldn't have any data to send right away. Mistake #2: The sender sends a FIN, but doesn't block because all data has already been acknowledged. This is supposedly a problem because the app never receives the data, but a TCP ack has never meant that data got beyond the receiving kernel to an app. So yes, it's true that neither side ever got an error and unfortunate that it's possible for the receiving app to get no data in that case, but that doesn't make it a TCP problem. Somebody at a higher level - JVM or the app itself - over-optimized their "connection and first data" code without understanding the actual behavior of the TCP protocol and socket API. Then they "fixed" it by working with their own protocol instead of against the standard one. Nothing to see here. It was just a false alarm. Move on.

gonzo 13y ago
It's not a TCP problem. It's a problem with the linux implementation of TCP.
- noselasd 13y ago
  No, it's not even that. The linked article (http://www.cliffc.org/blog/2013/08/15/tcp-is-unreliable/ http://www.cliffc.org/blog/2013/08/15/tcp-is-unreliable/) at the bottom tells you what's actually going on. The only thing slightly fishy is that the peer application didn't receive any connection attempt, but if you're anyway quite overloaded, e.g. running out of file descriptors, that may well happen.
  - notacoward 13y ago
    The FIN/SO_LINGER problem is real, and often misunderstood, but I don't think that was quite the whole story here. SO_LINGER would flag the error some of the time, but not always (see Mistake #2). Even if it did, avoiding the problem altogether (see Mistake #1) would be better than flagging it after the fact. Even with SO_LINGER on and the kernel's RST issues fixed, the app would still be erroneous.

songgao 13y ago

Interesting! One thing that I don't understand (which might be a little off-topic): If it's an RPC call, shouldn't there be a response for each RPC request to ack that the RPC request is well processed by the application layer? Otherwise how do you make sure the application layer won't screw up? How do you notify the client that the request is not invalid?

noselasd 13y ago

Note that he links to http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-s... which more or less explains what's going on.