4 ms·

I debugged and identified the exact same problem a few weeks ago. I don't have any solutions, but can confirm what you're seeing. I suspect most clients aren't

I debugged and identified the exact same problem a few weeks ago. I don't have any solutions, but can confirm what you're seeing. I suspect most clients aren't creating enough tcp connections in the window to cause a collision. (In our case, we discovered the issue from our load box during performance testing)

nielsole 6y ago
Collisions can already occur at two connections
- beowulfe 6y ago
  AFAIK, all the major tcp/ip stacks do round robin port assignment. Given relatively short, and relatively few, connections you should not have any collisions.
  - nielsole 6y ago
    Imagine the client is actually two machines behind a NAT (corporate network or school) which each open 1 connection. You can even reproduce this easily with two docker containers on the same host and curl using the "--local-port" option.
  - toast0 6y ago
    If your customers are behind CGNAT, and you have a good number of them, it would be pretty easy to run into collisions. CGNAT doesn't have any problem using the same source ip:port for connections to different destination ip:port, because there's no reason not to. I agree though, that most OS stacks are very conservative about using the same source ip:port for connections to anything else. It works ok until you need to make lots of connections, then you have to manage ports in userspace.
    - jcrawfordor 6y ago
      fwiw CGNAT is fairly uncommon in the United States except for cellphones. This creates one of those bad situations where you can have something that falls flat in the face of CGNAT but also "works fine" for all of your developer/employee traffic.
linuxftw 6y ago
I also saw this recently, although on another provider. The solution in the article's case (as stated) is to not terminate multiple public IPs across one or more NLB which target the same instance. If you must target the same instance more than once, give that instance an IP address for each public IP.

nielsole 6y ago

Collisions can already occur at two connections

beowulfe 6y ago
AFAIK, all the major tcp/ip stacks do round robin port assignment. Given relatively short, and relatively few, connections you should not have any collisions.
- nielsole 6y ago
  Imagine the client is actually two machines behind a NAT (corporate network or school) which each open 1 connection. You can even reproduce this easily with two docker containers on the same host and curl using the "--local-port" option.
- toast0 6y ago
  If your customers are behind CGNAT, and you have a good number of them, it would be pretty easy to run into collisions. CGNAT doesn't have any problem using the same source ip:port for connections to different destination ip:port, because there's no reason not to. I agree though, that most OS stacks are very conservative about using the same source ip:port for connections to anything else. It works ok until you need to make lots of connections, then you have to manage ports in userspace.
  - jcrawfordor 6y ago
    fwiw CGNAT is fairly uncommon in the United States except for cellphones. This creates one of those bad situations where you can have something that falls flat in the face of CGNAT but also "works fine" for all of your developer/employee traffic.

linuxftw 6y ago

I also saw this recently, although on another provider. The solution in the article's case (as stated) is to not terminate multiple public IPs across one or more NLB which target the same instance. If you must target the same instance more than once, give that instance an IP address for each public IP.