4 ms·

The identity of a TCP/IP connection is a quintuple containing { src addr, src port, dest addr, dest port, proto }. It feels like the article would be better if

The identity of a TCP/IP connection is a quintuple containing { src addr, src port, dest addr, dest port, proto }. It feels like the article would be better if before or after all the network stack spelunking and quizzing that it was mentioned somewhere. We don't need to rely on the implementation, though there's always interesting corner cases worth exploring.

loeg 4y ago
I agree that is super important and maybe worth mentioning but the point of the quiz is to demonstrate that Linux's implementation is actually more constrained than the traditional "unique src addr/port dst addr/port 4-tuple" (for TCP).
- DSMan195276 4y ago
  Yeah I feel like people are missing the point that it's not a 4-tuple thing, it's an ordering issue. Since the source port (sometimes) gets picked before the interface or destination is, you can get an EADDRNOTAVAIL result even when there's technically a potential for it to end-up as a unique 4-tuple. Doing the assignment in a different order or more explicitly can allow it to work by making sure that the kernel always knows it will be unique.
  - grayfaced 4y ago
    It's interesting design decision to preemptively fail when the connect might later be ambiguous. However in memory allocation, Linux will gladly allocate memory it doesn't have, return success and then later when you attempt to use it, it will kill the process with OOM killer.
    - loeg 4y ago
      > However in memory allocation, Linux will gladly allocate memory it doesn't have, return success and then later when you attempt to use it, it will kill the process with OOM killer. In userspace, with overcommit enabled, yes. In the kernel, you will often do the opposite -- speculatively allocate memory first, then take locks or other serialization primitives, then attempt to insert the object into some container, but fail if there was a collision. (The idea is to keep the relatively slow memory allocation outside of the locked region.)
    - DSMan195276 4y ago
      Very late on this one :P But I think the difference is the fact that ports are unique, so if the kernel assigns you a bad source port by accident then it can't change it later. You would need to know this is what happened and bind to a new source port (and do you attempt automatic again?). Where-as with overcommit, your program doesn't know or care what particular memory page you're using, and the kernel can easily fix things up in the background (Ex. use swap, or kill some other process) without you ever knowing about it.
deleted 4y ago
[deleted]
twic 4y ago
What is "proto" in your tuple? If we're talking about TCP, then it's always TCP, isn't it? I think that formulation was invented to explain the fact that TCP and UDP can have distinct otherwise identical sockets, but i don't think it's a great way to do that. TCP and UDP just have completely separate spaces of socket addresses, the same way TCP and NetBIOS or TCP and UNIX domain sockets do.
- wyldfire 4y ago
  Indeed - I fudged things a bit by talking about TCP there. It would have been clearer if I just discussed IP instead. > TCP and UDP just have completely separate spaces of socket addresses But so does SCTP, and ICMP and IGMP and ... -- so rather than enumerate the protocols we can just describe this property of IP.
  - derefr 4y ago
    Yeah, but for ICMP and so forth, ports aren't a thing. So you don't really have a universal 5-tuple. For a router or other middlebox, or an OS kernel, to do things like outbound-initiated-flow firewall-rule exceptions correctly, it must keep N different flow-state tables, one per transport-layer (L4) protocol; where each flow-state table's "primary key" is over a set of columns unique to that table / L4 protocol. TCP and UDP just happen to be both the best-known L4 protocols, and to both use {srcIP, srcPort, dstIP, dstPort} as their "primary key" for flows; but this doesn't hold for other L4 protocols. (Which is in turn why L4 protocols "must" be handled in kernel-land, for kernel firewalls, traffic-shapers, etc. to work: L4 flow-state doesn't have a universal schema for these services to work with; and because these services are implemented in static-compiled languages, they have to be built with compile-time knowledge of each known L4 protocol, so that they can have concrete implementations for each L4 protocol written or generated for each service. There's no way to just bring in (through some hypothetical FUSE-like "userland L4 protocol server" abstraction) more L4 protocols, and expect those kernel facilities to work with them. [And all the same goes for ASICs in L4 network routers — only moreso.] Which is why we got the L4 protocol ossification we did. Modern protocols like SCTP and QUIC being implemented on top of UDP, is a direct result of there being no universal 5-tuple!)

loeg 4y ago

I agree that is super important and maybe worth mentioning but the point of the quiz is to demonstrate that Linux's implementation is actually more constrained than the traditional "unique src addr/port dst addr/port 4-tuple" (for TCP).

DSMan195276 4y ago
Yeah I feel like people are missing the point that it's not a 4-tuple thing, it's an ordering issue. Since the source port (sometimes) gets picked before the interface or destination is, you can get an EADDRNOTAVAIL result even when there's technically a potential for it to end-up as a unique 4-tuple. Doing the assignment in a different order or more explicitly can allow it to work by making sure that the kernel always knows it will be unique.
- grayfaced 4y ago
  It's interesting design decision to preemptively fail when the connect might later be ambiguous. However in memory allocation, Linux will gladly allocate memory it doesn't have, return success and then later when you attempt to use it, it will kill the process with OOM killer.
  - loeg 4y ago
    > However in memory allocation, Linux will gladly allocate memory it doesn't have, return success and then later when you attempt to use it, it will kill the process with OOM killer. In userspace, with overcommit enabled, yes. In the kernel, you will often do the opposite -- speculatively allocate memory first, then take locks or other serialization primitives, then attempt to insert the object into some container, but fail if there was a collision. (The idea is to keep the relatively slow memory allocation outside of the locked region.)
  - DSMan195276 4y ago
    Very late on this one :P But I think the difference is the fact that ports are unique, so if the kernel assigns you a bad source port by accident then it can't change it later. You would need to know this is what happened and bind to a new source port (and do you attempt automatic again?). Where-as with overcommit, your program doesn't know or care what particular memory page you're using, and the kernel can easily fix things up in the background (Ex. use swap, or kill some other process) without you ever knowing about it.

deleted 4y ago

[deleted]

twic 4y ago

What is "proto" in your tuple? If we're talking about TCP, then it's always TCP, isn't it? I think that formulation was invented to explain the fact that TCP and UDP can have distinct otherwise identical sockets, but i don't think it's a great way to do that. TCP and UDP just have completely separate spaces of socket addresses, the same way TCP and NetBIOS or TCP and UNIX domain sockets do.

wyldfire 4y ago
Indeed - I fudged things a bit by talking about TCP there. It would have been clearer if I just discussed IP instead. > TCP and UDP just have completely separate spaces of socket addresses But so does SCTP, and ICMP and IGMP and ... -- so rather than enumerate the protocols we can just describe this property of IP.
- derefr 4y ago
  Yeah, but for ICMP and so forth, ports aren't a thing. So you don't really have a universal 5-tuple. For a router or other middlebox, or an OS kernel, to do things like outbound-initiated-flow firewall-rule exceptions correctly, it must keep N different flow-state tables, one per transport-layer (L4) protocol; where each flow-state table's "primary key" is over a set of columns unique to that table / L4 protocol. TCP and UDP just happen to be both the best-known L4 protocols, and to both use {srcIP, srcPort, dstIP, dstPort} as their "primary key" for flows; but this doesn't hold for other L4 protocols. (Which is in turn why L4 protocols "must" be handled in kernel-land, for kernel firewalls, traffic-shapers, etc. to work: L4 flow-state doesn't have a universal schema for these services to work with; and because these services are implemented in static-compiled languages, they have to be built with compile-time knowledge of each known L4 protocol, so that they can have concrete implementations for each L4 protocol written or generated for each service. There's no way to just bring in (through some hypothetical FUSE-like "userland L4 protocol server" abstraction) more L4 protocols, and expect those kernel facilities to work with them. [And all the same goes for ASICs in L4 network routers — only moreso.] Which is why we got the L4 protocol ossification we did. Modern protocols like SCTP and QUIC being implemented on top of UDP, is a direct result of there being no universal 5-tuple!)