4 ms·

This a nice summary; I was actually thinking about doing my own writeup of this very thing. I have a few thoughts beyond what was covered in the blog post: Fo

by simmons 12y ago

This a nice summary; I was actually thinking about doing my own writeup of this very thing. I have a few thoughts beyond what was covered in the blog post: For many protocols and modes of communication, UDP is really lousy for the developer -- we end up re-inventing TCP (congestion control, retransmission, etc.). Unfortunately, if we want to traverse NAT, we often must rely on UDP hole punching and thus use UDP. There's a TCP hole punching technique called "TCP simultaneous open", but it seems too flaky to expect to work in the real world. I've been thinking about trying UDT [1] as a user-mode TCP-like connection layer over UDP. In addition to the ugly UPnP technique for asking a router to automatically port-forward, there have been attempts to standardize a much simpler solution in NAT-PMP and it's successor, PCP. (PCP even works for IPv6 stateful firewalls.) Many people do have security concerns about this capability. I don't know to what extent NAT-PMP/PCP has made inroads into the router market -- it would be interesting to conduct some sort of survey. [1] http://udt.sourceforge.net/ http://udt.sourceforge.net/

simmons 12y ago
Also, the article frequently mentions a 120-second timeout interval, within which the connection needs to be refreshed. In Linux (and thus most consumer routers?), I believe the timeout is actually 30 seconds for UDP. :(
- zrm 12y ago
  The 120 second interval is the minimum from RFC4787 Sec. 4.3 REQ-5. (There is an exception that permits shorter intervals but only for specific ports, in particular port 53 for DNS.) If Linux is using 30 seconds then it's in violation and someone should submit a bug report.
  - dvanduzer 12y ago
    Privileged ports were a broken security model to begin with. Consider a a distributed system that relies on that minimum 120 second interval when determining whether to send a keepalive. Failure to send one within this interval would create an otherwise easily avoided network partition. Consider someone adapting this system to operate across a restricted network which only permits traffic on privileged ports to traverse its border. Suddenly idle connections can be purged well under that previously safe two minute window. Instant partition. Whether Linux implements the conditional to distinguish between privileged and unprivileged ports is irrelevant. Any serious systems designer must consider the timeout interval to be 30 seconds.
    - roghummal 12y ago
      This has zero to do with the ports being "privileged" and everything to do with their state as a "well-known port range" used by "IANA-registered application(s)". The idea being you can adjust the timeout because you can infer the way the port is being used.
    - zrm 12y ago
      It isn't about privileged ports. It's about the protocols those ports are assigned to, and it's really only referring to port 53 for DNS because it's the only protocol that requires it. Using a shorter mapping timeout for port 22 or port 80 is not permitted. The reason for DNS to be special is that DNS has to use a random source port for every query to reduce vulnerability to the Kaminsky attack. Most other UDP applications use a single socket (and thus port and thus NAT mapping) for all peers. So if you have many DNS clients behind a NAT you're going to run out of source ports if you try to keep mappings for 120 seconds. Meanwhile the DNS protocol only requires the mapping to persist until the query response is received which is almost immediately. So DNS gets a special rule. Which isn't a 30 second timeout. There are real NATs in production that use 5 second mapping timeouts on port 53, which is completely reasonable and necessary if your site is making 10,000 DNS queries a second. The sensible way to deal with all of this is to assume the 120 second timeout initially and if you encounter a hard disconnect that allows an immediate reconnection then reduce the keepalive interval. And don't run non-DNS protocols on port 53 unless you're prepared for the consequences.
      - dvanduzer 12y ago
        Any NAT deployment that large will included local DNS caching servers. Out of the box consumer NAT configurations (WRT-54G, etc) typically run a DNS cache also. Source port randomization in response to Kaminsky's paper in 2008 is definitely the sort of thing that would fill NAT tables faster. But RFC4787 was published in 2007. The behavior chosen by the implementers of the Linux netfilter code (see simmons' post) elegantly covers most cases without getting as specific as REQ-5 in the RFC, and not nearly as specific as you're describing. Network operators just manually configure a faster timeout for port 53. That's what the RFC permits: tuning in response to real conditions. A hard disconnect is a major event that usually goes all the way back up the stack. A systems designer wants to avoid such a thing, and the sensible thing is to look at the source (of the most commonly deployed implementations).
  - simmons 12y ago
    I just took a look at Linux's nf_conntrack_proto_udp.c to see what's happening. It looks like Linux starts the UDP timeout at 30 seconds, but extends it to 180 seconds when it sees traffic both ways, assuming it's "some kind of stream."
latitude 12y ago
You don't need to reinvent TCP. You can simply put TCP over UDP. Works like magic :) TCP hole punching works just fine in practice, but it's certainly a part of the "long tail" of empirical hacks that allow bumping NAT traversal rates into 95+% range. Detecting NATs that decrement ports is another. Ports that are incremented/decremented by more than 1 is yet another. And there's also UPnP that gives you another percent or so. I was doing this as a part of the Hamachi project [0], almost 10 years ago, and this long tail was absolutely instrumental to being able to serve hundreds of thousands of clients off just few measly Dell servers for $300/mo in hosting costs. [0] http://swapped.cc/hamachi http://swapped.cc/hamachi
- jon-wood 12y ago
  Its somewhat OT, but thanks for Hamachi! It made many a late night gaming session possible in my teens and early twenties, and was always an astoundingly simple piece of software to use considering everything it did.
  - latitude 12y ago
    My pleasure, glad to have been of some help :)
- api 12y ago
  Cool to get a ping from Hamachi's author. I deliberately did not use Hamachi while designing ZeroTier since I didn't want to pollute my headspace with something so similar. :) ZeroTier right now is sort of an OSS Hamachi, since I figured this would be a good starting application, but I've got different longer-term goals in mind. (Won't be abandoning that application though.)

by simmons 12y ago

simmons 12y ago

Also, the article frequently mentions a 120-second timeout interval, within which the connection needs to be refreshed. In Linux (and thus most consumer routers?), I believe the timeout is actually 30 seconds for UDP. :(

zrm 12y ago
The 120 second interval is the minimum from RFC4787 Sec. 4.3 REQ-5. (There is an exception that permits shorter intervals but only for specific ports, in particular port 53 for DNS.) If Linux is using 30 seconds then it's in violation and someone should submit a bug report.
- dvanduzer 12y ago
  Privileged ports were a broken security model to begin with. Consider a a distributed system that relies on that minimum 120 second interval when determining whether to send a keepalive. Failure to send one within this interval would create an otherwise easily avoided network partition. Consider someone adapting this system to operate across a restricted network which only permits traffic on privileged ports to traverse its border. Suddenly idle connections can be purged well under that previously safe two minute window. Instant partition. Whether Linux implements the conditional to distinguish between privileged and unprivileged ports is irrelevant. Any serious systems designer must consider the timeout interval to be 30 seconds.
  - roghummal 12y ago
    This has zero to do with the ports being "privileged" and everything to do with their state as a "well-known port range" used by "IANA-registered application(s)". The idea being you can adjust the timeout because you can infer the way the port is being used.
  - zrm 12y ago
    It isn't about privileged ports. It's about the protocols those ports are assigned to, and it's really only referring to port 53 for DNS because it's the only protocol that requires it. Using a shorter mapping timeout for port 22 or port 80 is not permitted. The reason for DNS to be special is that DNS has to use a random source port for every query to reduce vulnerability to the Kaminsky attack. Most other UDP applications use a single socket (and thus port and thus NAT mapping) for all peers. So if you have many DNS clients behind a NAT you're going to run out of source ports if you try to keep mappings for 120 seconds. Meanwhile the DNS protocol only requires the mapping to persist until the query response is received which is almost immediately. So DNS gets a special rule. Which isn't a 30 second timeout. There are real NATs in production that use 5 second mapping timeouts on port 53, which is completely reasonable and necessary if your site is making 10,000 DNS queries a second. The sensible way to deal with all of this is to assume the 120 second timeout initially and if you encounter a hard disconnect that allows an immediate reconnection then reduce the keepalive interval. And don't run non-DNS protocols on port 53 unless you're prepared for the consequences.
    - dvanduzer 12y ago
      Any NAT deployment that large will included local DNS caching servers. Out of the box consumer NAT configurations (WRT-54G, etc) typically run a DNS cache also. Source port randomization in response to Kaminsky's paper in 2008 is definitely the sort of thing that would fill NAT tables faster. But RFC4787 was published in 2007. The behavior chosen by the implementers of the Linux netfilter code (see simmons' post) elegantly covers most cases without getting as specific as REQ-5 in the RFC, and not nearly as specific as you're describing. Network operators just manually configure a faster timeout for port 53. That's what the RFC permits: tuning in response to real conditions. A hard disconnect is a major event that usually goes all the way back up the stack. A systems designer wants to avoid such a thing, and the sensible thing is to look at the source (of the most commonly deployed implementations).
- simmons 12y ago
  I just took a look at Linux's nf_conntrack_proto_udp.c to see what's happening. It looks like Linux starts the UDP timeout at 30 seconds, but extends it to 180 seconds when it sees traffic both ways, assuming it's "some kind of stream."

latitude 12y ago

You don't need to reinvent TCP. You can simply put TCP over UDP. Works like magic :) TCP hole punching works just fine in practice, but it's certainly a part of the "long tail" of empirical hacks that allow bumping NAT traversal rates into 95+% range. Detecting NATs that decrement ports is another. Ports that are incremented/decremented by more than 1 is yet another. And there's also UPnP that gives you another percent or so. I was doing this as a part of the Hamachi project [0], almost 10 years ago, and this long tail was absolutely instrumental to being able to serve hundreds of thousands of clients off just few measly Dell servers for $300/mo in hosting costs. [0] http://swapped.cc/hamachi http://swapped.cc/hamachi

jon-wood 12y ago
Its somewhat OT, but thanks for Hamachi! It made many a late night gaming session possible in my teens and early twenties, and was always an astoundingly simple piece of software to use considering everything it did.
- latitude 12y ago
  My pleasure, glad to have been of some help :)
api 12y ago
Cool to get a ping from Hamachi's author. I deliberately did not use Hamachi while designing ZeroTier since I didn't want to pollute my headspace with something so similar. :) ZeroTier right now is sort of an OSS Hamachi, since I figured this would be a good starting application, but I've got different longer-term goals in mind. (Won't be abandoning that application though.)