16 ms·

Lots of the complexity of TCP at a protocol level comes from treating it as a single duplex connection instead of a pair of simplex connections. And the article

by tonyg 3y ago

Lots of the complexity of TCP at a protocol level comes from treating it as a single duplex connection instead of a pair of simplex connections. And the article here shows that this complexity leaks up to the API level, too.

amelius 3y ago
Isn't there a good reason for it though?
- jerf 3y ago
  Well, breaking the socket into two pieces would add complexity. It would create a new requirement for non-trivial programs to add bookkeeping to track the two bits together and make it easier for the two bits to become accidentally permanently estranged from each other. It would add new states to the already complicated TCP set of states. Then again, in most of my "non-trivial" network code I already end up cracking the socket into two pieces anyhow, and while I take advantage of the pun that closing either of them closes the other it would be trivial for me to add this behavior to the separated sockets if that's what I wanted. Partially this is a result of my "non-trivial" network code being in Go so I end up running a goroutine each for send and receive, but still I imagine this would be a common structure one way or another in a lot of non-trivial network servers. But the additional states on the TCP socket would definitely bite some people. Even in its current simpler state it still confuses people and there's a post about how some server turned out to be leaking sockets because it didn't properly close them shows up with some frequency on HN.
  - dfawcus 3y ago
    Nah - one would just add a new syscall, to split the fd. int sock_fd, split_pair[2]; char c; int n = socket_split(sock_fd, split_pair); close(sock_fd); n = write(split_pair[1], "a", 1); close(split_pair[1]); n = read(split_pair[0], &c, 1); read(split_pair[0], &c, 1); /* returns 0 / EOF */ The effect being to split the fds, and give an output similar to pipe(2). Moreover it would also duplicate the underlying open file structure, such that one is read only, the other write only, and both referencing the same socket. Then the read fd would reference one "open file", the write-fd the other. (One can observe a similar issue if one opens FIFO's in read/write mode, but since they can also be opened read-only or write-only, the issue does not arise.)
    - jerf 3y ago
      I wonder if perhaps you replied to the wrong comment? I never questioned that it could be done, I was discussing the results from the point of view of it already being done (and for my discussion's purpose, mandatory).
      - dfawcus 3y ago
        I was disagreeing with your first paragraph. If the splitting is optional, as it would have to be now, and as I proposed it would not add complexity since it would be opt in. Theoretically if it had split from the start, it may have added a minor amount of complexity to some programs, but not a significant amount. I don't get the need to add TCP states, as the TCP machine will not change, and the user space app shouldn't really be tracking/replicatting the TCP states. Anyway, your assumed (and mandatory) point of view was not obvious to me, I interpreted it from the position of "if that was done now". Which would add a trivial amount of logic to the kernel, mainly reusing existing logic, structures, and counters. So I guess we were slightly at cross purposes.
  - jmalicki 3y ago
    > It would add new states to the already complicated TCP set of states. Huh? Half-closed connections already exist. The complexity in the state machine is there, and more TCP users should be aware of it (the source of soo.... many... bugs)
    - jerf 3y ago
      While the current "half closed state" and "one of the read/write halves is closed and you don't realize it" may sound similar, they're two different "halves". Splitting the socket fully into two separate file handles would mean that users could close one of them without realizing it, and thus be able to write a new sort of bug where they would be able to ask stack overflow "I'm able to read from my socket but not write from it, what gives?" This wouldn't necessarily manifest as a new TCP state in the kernel state table. You could do this to TCP today at the API level alone. Which is further evidence it's a different "half". And if it sounds silly that anyone would make such a mistake, like many such hypothetical mistakes it's important to remember that in the wild it's not always next to each other. My least favorite networking experience ever involved being three levels deep in abstraction in some Perl libraries, every level of which felt like it can and should make all sorts of helpful little calls, the end result being all kinds of fun to debug as soon as any of them were wrong. The thing that closed one of the halves of the socket and the person confused by the result can be arbitrarily separated in the code. Or, to put it another way, you can't grow the API's complexity without growing the API's complexity. If today closing closes both reading and writing but a new API severs that guarantee and creates at least two new states, that inevitably results in a new way for things to go wrong. You can't have one without the other.
      - jmalicki 3y ago
        > Splitting the socket fully into two separate file handles would mean that users could close one of them without realizing it, and thus be able to write a new sort of bug where they would be able to ask stack overflow "I'm able to read from my socket but not write from it, what gives?" shutdown(sockfd, 1) gives you "I'm able to read from my socket but not write from it, what gives?" The real bugs with TCP come from "why am I not able to download the file because the server did close(sockfd) rather than shutdown(sockfd, 1), and my TCP stack ACK'd, causing the server to RST, causing my machine to give me an error before I read all of the data". This is already waaaay harder than the counterfactual you're imagining, and the bugs are here, today.
- tonyg 3y ago
  As far as I'm aware, it was driven by the "piggybacking" of ACKs on packets travelling in the opposite direction, which turns out to have been a slight misfeature.
- Veserv 3y ago
  Yes. TCP connection establishment inherently forms a two-way channel between two entitys. There is little reason to duplicate the connection/routing (in the generic sense) establishment for each direction in the general case where you might want a two-way channel. The problem in the presented example is that there are, fundamentally, 3 communicating entitys A->B->C, which logically demands two channels. A quirk of the example where A and C happen to be on the same system and coincidentally can use the channel disjointly (read-only and write-only) in conjunction with TCP being inherently two-way allows it to be kludged into a single channel, but frankly it is your fault for relying on bizarre implementation details [1]. If you want a proper abstraction, what you would actually do is multiplex multiple logical streams over a single duplex channel. The code would then operate purely on the logical streams (pipe-like) and everything would be clean. The implementation would then only need to establish connection once, removing the inherent overhead associated with establishing multiple network protocol connections, while still giving you a clean and consistent abstraction (barring the standard leakiness of the TCP abstraction). [1] https://xkcd.com/1172/ https://xkcd.com/1172/
MuffinFlavored 3y ago
> instead of a pair of simplex connections To clarify, would one be send and the other be receive or are you saying one would be for the application data and the 2nd would be for the protocol SYN/ACK and all that?
- tonyg 3y ago
  The former.
rstuart4133 3y ago
> Lots of the complexity of TCP at a protocol level comes from treating it as a single duplex connection instead of a pair of simplex connections. I've created a few communication protocols in my time (all people of a certain age have, because we pre-date TCP/IP), and I am reasonable familiar with TCP itself. I don't see the extra complexity you speak of. At most, it is de-multiplexing the ACK fields in the header from the rest. If you want to whinge about unnecessary complexity in the TCP protocol, I'd suggest starting with it being byte orientated rather than packet orientated. Secondly Dan is out of touch. He's right in saying it's a complex solution for a shell pipeline, but the TCP is designed to operate in a less reliable world. In TCP's world you can no longer assume a message you sent was acted upon by the other end. The network could have died, or the system crashed before the data made it to disk. Thus the shell command: echo Run for your life or you will die | mail friend@doomed.com is perfectly reasonable. You know the message will make it or you will be told something went wrong. But if the "mail" command is running on a different machine, unsupervised, then you have no idea the message made it unless the mail programs sends a message back saying "email safely and durably received". That requires a back channel. You almost always require a back channel for that reason and what's worse, they are a pita to set up using unidirectional pipes. That makes a unidirectional pipe is a terrible API for sending messages to a remote location. It looks like TCP's successor will be QUIC. In QUIC there isn't just two streams, you can create lots of sub-streams, using datagrams if you want. Now that is a lot of extra complexity. But from the applications point of view it's a very nice feature. It's a bit like the second channel TCP gives you, as it's surprising how often you need a more than one channel between two programs. In fact the very first use TCP was put to was FTP and guess what - FTP creates two TCP connections. And it's example of how painful it can be to set that up as NAT broke the original FTP for a while. We now finally have an off the shelf solution in QUIC. It only took 30+ years to arrive. It looks to me like the world has voted. The outcome is a firm Nay to Dan's suggestion, and it's now moving in the opposite direction.
- dfawcus 3y ago
  "And it's example of how painful it can be to set that up as NAT broke the original FTP for a while." NAT broke FTP for about the 5 seconds it took for one to realise they needed to type in the PASV command - which existed right from the start. It was in RFC 542 (1973) - pre TCP, and RFC 765 (1980) - first post TCP. NAT dates from around 1994.

by tonyg 3y ago

amelius 3y ago

Isn't there a good reason for it though?

jerf 3y ago
Well, breaking the socket into two pieces would add complexity. It would create a new requirement for non-trivial programs to add bookkeeping to track the two bits together and make it easier for the two bits to become accidentally permanently estranged from each other. It would add new states to the already complicated TCP set of states. Then again, in most of my "non-trivial" network code I already end up cracking the socket into two pieces anyhow, and while I take advantage of the pun that closing either of them closes the other it would be trivial for me to add this behavior to the separated sockets if that's what I wanted. Partially this is a result of my "non-trivial" network code being in Go so I end up running a goroutine each for send and receive, but still I imagine this would be a common structure one way or another in a lot of non-trivial network servers. But the additional states on the TCP socket would definitely bite some people. Even in its current simpler state it still confuses people and there's a post about how some server turned out to be leaking sockets because it didn't properly close them shows up with some frequency on HN.
- dfawcus 3y ago
  Nah - one would just add a new syscall, to split the fd. int sock_fd, split_pair[2]; char c; int n = socket_split(sock_fd, split_pair); close(sock_fd); n = write(split_pair[1], "a", 1); close(split_pair[1]); n = read(split_pair[0], &c, 1); read(split_pair[0], &c, 1); /* returns 0 / EOF */ The effect being to split the fds, and give an output similar to pipe(2). Moreover it would also duplicate the underlying open file structure, such that one is read only, the other write only, and both referencing the same socket. Then the read fd would reference one "open file", the write-fd the other. (One can observe a similar issue if one opens FIFO's in read/write mode, but since they can also be opened read-only or write-only, the issue does not arise.)
  - jerf 3y ago
    I wonder if perhaps you replied to the wrong comment? I never questioned that it could be done, I was discussing the results from the point of view of it already being done (and for my discussion's purpose, mandatory).
    - dfawcus 3y ago
      I was disagreeing with your first paragraph. If the splitting is optional, as it would have to be now, and as I proposed it would not add complexity since it would be opt in. Theoretically if it had split from the start, it may have added a minor amount of complexity to some programs, but not a significant amount. I don't get the need to add TCP states, as the TCP machine will not change, and the user space app shouldn't really be tracking/replicatting the TCP states. Anyway, your assumed (and mandatory) point of view was not obvious to me, I interpreted it from the position of "if that was done now". Which would add a trivial amount of logic to the kernel, mainly reusing existing logic, structures, and counters. So I guess we were slightly at cross purposes.
- jmalicki 3y ago
  > It would add new states to the already complicated TCP set of states. Huh? Half-closed connections already exist. The complexity in the state machine is there, and more TCP users should be aware of it (the source of soo.... many... bugs)
  - jerf 3y ago
    While the current "half closed state" and "one of the read/write halves is closed and you don't realize it" may sound similar, they're two different "halves". Splitting the socket fully into two separate file handles would mean that users could close one of them without realizing it, and thus be able to write a new sort of bug where they would be able to ask stack overflow "I'm able to read from my socket but not write from it, what gives?" This wouldn't necessarily manifest as a new TCP state in the kernel state table. You could do this to TCP today at the API level alone. Which is further evidence it's a different "half". And if it sounds silly that anyone would make such a mistake, like many such hypothetical mistakes it's important to remember that in the wild it's not always next to each other. My least favorite networking experience ever involved being three levels deep in abstraction in some Perl libraries, every level of which felt like it can and should make all sorts of helpful little calls, the end result being all kinds of fun to debug as soon as any of them were wrong. The thing that closed one of the halves of the socket and the person confused by the result can be arbitrarily separated in the code. Or, to put it another way, you can't grow the API's complexity without growing the API's complexity. If today closing closes both reading and writing but a new API severs that guarantee and creates at least two new states, that inevitably results in a new way for things to go wrong. You can't have one without the other.
    - jmalicki 3y ago
      > Splitting the socket fully into two separate file handles would mean that users could close one of them without realizing it, and thus be able to write a new sort of bug where they would be able to ask stack overflow "I'm able to read from my socket but not write from it, what gives?" shutdown(sockfd, 1) gives you "I'm able to read from my socket but not write from it, what gives?" The real bugs with TCP come from "why am I not able to download the file because the server did close(sockfd) rather than shutdown(sockfd, 1), and my TCP stack ACK'd, causing the server to RST, causing my machine to give me an error before I read all of the data". This is already waaaay harder than the counterfactual you're imagining, and the bugs are here, today.
tonyg 3y ago
As far as I'm aware, it was driven by the "piggybacking" of ACKs on packets travelling in the opposite direction, which turns out to have been a slight misfeature.
Veserv 3y ago
Yes. TCP connection establishment inherently forms a two-way channel between two entitys. There is little reason to duplicate the connection/routing (in the generic sense) establishment for each direction in the general case where you might want a two-way channel. The problem in the presented example is that there are, fundamentally, 3 communicating entitys A->B->C, which logically demands two channels. A quirk of the example where A and C happen to be on the same system and coincidentally can use the channel disjointly (read-only and write-only) in conjunction with TCP being inherently two-way allows it to be kludged into a single channel, but frankly it is your fault for relying on bizarre implementation details [1]. If you want a proper abstraction, what you would actually do is multiplex multiple logical streams over a single duplex channel. The code would then operate purely on the logical streams (pipe-like) and everything would be clean. The implementation would then only need to establish connection once, removing the inherent overhead associated with establishing multiple network protocol connections, while still giving you a clean and consistent abstraction (barring the standard leakiness of the TCP abstraction). [1] https://xkcd.com/1172/ https://xkcd.com/1172/

MuffinFlavored 3y ago

> instead of a pair of simplex connections To clarify, would one be send and the other be receive or are you saying one would be for the application data and the 2nd would be for the protocol SYN/ACK and all that?

tonyg 3y ago
The former.

rstuart4133 3y ago

> Lots of the complexity of TCP at a protocol level comes from treating it as a single duplex connection instead of a pair of simplex connections. I've created a few communication protocols in my time (all people of a certain age have, because we pre-date TCP/IP), and I am reasonable familiar with TCP itself. I don't see the extra complexity you speak of. At most, it is de-multiplexing the ACK fields in the header from the rest. If you want to whinge about unnecessary complexity in the TCP protocol, I'd suggest starting with it being byte orientated rather than packet orientated. Secondly Dan is out of touch. He's right in saying it's a complex solution for a shell pipeline, but the TCP is designed to operate in a less reliable world. In TCP's world you can no longer assume a message you sent was acted upon by the other end. The network could have died, or the system crashed before the data made it to disk. Thus the shell command: echo Run for your life or you will die | mail friend@doomed.com is perfectly reasonable. You know the message will make it or you will be told something went wrong. But if the "mail" command is running on a different machine, unsupervised, then you have no idea the message made it unless the mail programs sends a message back saying "email safely and durably received". That requires a back channel. You almost always require a back channel for that reason and what's worse, they are a pita to set up using unidirectional pipes. That makes a unidirectional pipe is a terrible API for sending messages to a remote location. It looks like TCP's successor will be QUIC. In QUIC there isn't just two streams, you can create lots of sub-streams, using datagrams if you want. Now that is a lot of extra complexity. But from the applications point of view it's a very nice feature. It's a bit like the second channel TCP gives you, as it's surprising how often you need a more than one channel between two programs. In fact the very first use TCP was put to was FTP and guess what - FTP creates two TCP connections. And it's example of how painful it can be to set that up as NAT broke the original FTP for a while. We now finally have an off the shelf solution in QUIC. It only took 30+ years to arrive. It looks to me like the world has voted. The outcome is a firm Nay to Dan's suggestion, and it's now moving in the opposite direction.

dfawcus 3y ago
"And it's example of how painful it can be to set that up as NAT broke the original FTP for a while." NAT broke FTP for about the 5 seconds it took for one to realise they needed to type in the PASV command - which existed right from the start. It was in RFC 542 (1973) - pre TCP, and RFC 765 (1980) - first post TCP. NAT dates from around 1994.