5 ms·

they could fix the problem by assigning each NLB AZ IP an aliased IP on your target machine. however, this would be an absolute mess to configure. For example

they could fix the problem by assigning each NLB AZ IP an aliased IP on your target machine. however, this would be an absolute mess to configure. For example if the 3 NLB AZ IPs were: 1.2.3.6 1.2.3.7 1.2.3.8 then if you set up 3 IP aliases on your target instance: 192.168.2.6 192.168.2.7 192.168.2.8 then if the NLB always mapped: 1.2.3.6 -> 192.168.2.6 1.2.3.7 -> 192.168.2.7 1.2.3.8 -> 192.168.2.8 then there would be no clashes.

toast0 6y ago
Since you seem knowledgable and I'm not an AWS customer... Can you confirm the general workings of the Network Load Balancer? It seems like it's basically doing NAT to take client:port <-> server:port to client:port <-> instance:port, including adjusting instance NAT mappings so the outgoing traffic gets NATed back? Sort of like direct server return load balancing, but weird because the instance doesnt see the service IP and NAT takes care of it, somehow? In that case, yeah, you either need to have a rfc1918 service address mapped for each public service address or just use the public address. Elsewhere in thread, it sounds like it's possible to set this up using the public address but kubernetes doesn't like that; in which case, it seems appropriate to fix kubernetes.
- benmmurphy 6y ago
  There are two modes to NLB: 'instance' mode which is what they are describing in the article. I think what happens is the packets hit the AWS network and then somehow AWS decides whether it is an existing flow or chooses a target instance to create a new flow. then it just routes the packet by only modifying the destination address. it can do this because the whole of the AWS network is basically a lie and ip packets can just magically pop up in your interface and they don't have to go through a normal routing process. then on the way back it is able to track the flow of the connection and know that has to rewrite the source address of the packet. i think this is basically what you describe above. then there is 'ip' mode which i think it looks like it works similar to how haproxy would proxy TCP traffic. all of the packets you receive look like they come from the NLB and not the client. because there can be many clients this means the NLB networking layer needs to rewrite the source address and the source port which i guess is a more traditional kind of NAT. this also means there is a limit to the number of concurrent flows you can have. in the documentation i think AWS says there is 55,000 connection limit.
  - toast0 6y ago
    Ok, ip mode seems usable if you don't have a lot of connections per NLB/instance pair, but kind of meh. Does the instance port have to match the service port? In instance mode, you could setup so us-east1 is like 441, us-east2 is 442, us-west1 is 451 etc. And then you wouldn't have tcp 4-tuple collisions on the instance. Similarly for ip mode, you could put the same host in for ports 440-449 and get 10x 55,000 connections, or however many ports you need to hit your numbers. (Port numbers chosen for example only, check /etc/services if you don't want to stomp on allocations for services you're probably not using) Thanks!
    - benmmurphy 6y ago
      That AZ-port solution would work as well and would be easier to configure than different IPs. But they don't let you map port or IP based on AZ for a single NLB. You could setup different independent NLBs in each zone and then setup target groups for each NLB AZ and I think this might have been what you were proposing. I'm pretty sure you can setup a different port for each target group and it doesn't have to match the traffic port on the NLB. For IP mode you are correct that you can run on multiple ports to get around the connection limit. The documentation even says it is based on IP address and port. > When the target type is ip, the load balancer can support 55,000 simultaneous connections or about 55,000 connections per minute to each unique target (IP address and port). If you exceed these connections, there is an increased chance of port allocation errors. If you get port allocation errors, add more targets to the target group. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html https://docs.aws.amazon.com/elasticloadbalancing/latest/netw...

toast0 6y ago

Since you seem knowledgable and I'm not an AWS customer... Can you confirm the general workings of the Network Load Balancer? It seems like it's basically doing NAT to take client:port <-> server:port to client:port <-> instance:port, including adjusting instance NAT mappings so the outgoing traffic gets NATed back? Sort of like direct server return load balancing, but weird because the instance doesnt see the service IP and NAT takes care of it, somehow? In that case, yeah, you either need to have a rfc1918 service address mapped for each public service address or just use the public address. Elsewhere in thread, it sounds like it's possible to set this up using the public address but kubernetes doesn't like that; in which case, it seems appropriate to fix kubernetes.

benmmurphy 6y ago
There are two modes to NLB: 'instance' mode which is what they are describing in the article. I think what happens is the packets hit the AWS network and then somehow AWS decides whether it is an existing flow or chooses a target instance to create a new flow. then it just routes the packet by only modifying the destination address. it can do this because the whole of the AWS network is basically a lie and ip packets can just magically pop up in your interface and they don't have to go through a normal routing process. then on the way back it is able to track the flow of the connection and know that has to rewrite the source address of the packet. i think this is basically what you describe above. then there is 'ip' mode which i think it looks like it works similar to how haproxy would proxy TCP traffic. all of the packets you receive look like they come from the NLB and not the client. because there can be many clients this means the NLB networking layer needs to rewrite the source address and the source port which i guess is a more traditional kind of NAT. this also means there is a limit to the number of concurrent flows you can have. in the documentation i think AWS says there is 55,000 connection limit.
- toast0 6y ago
  Ok, ip mode seems usable if you don't have a lot of connections per NLB/instance pair, but kind of meh. Does the instance port have to match the service port? In instance mode, you could setup so us-east1 is like 441, us-east2 is 442, us-west1 is 451 etc. And then you wouldn't have tcp 4-tuple collisions on the instance. Similarly for ip mode, you could put the same host in for ports 440-449 and get 10x 55,000 connections, or however many ports you need to hit your numbers. (Port numbers chosen for example only, check /etc/services if you don't want to stomp on allocations for services you're probably not using) Thanks!
  - benmmurphy 6y ago
    That AZ-port solution would work as well and would be easier to configure than different IPs. But they don't let you map port or IP based on AZ for a single NLB. You could setup different independent NLBs in each zone and then setup target groups for each NLB AZ and I think this might have been what you were proposing. I'm pretty sure you can setup a different port for each target group and it doesn't have to match the traffic port on the NLB. For IP mode you are correct that you can run on multiple ports to get around the connection limit. The documentation even says it is based on IP address and port. > When the target type is ip, the load balancer can support 55,000 simultaneous connections or about 55,000 connections per minute to each unique target (IP address and port). If you exceed these connections, there is an increased chance of port allocation errors. If you get port allocation errors, add more targets to the target group. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html https://docs.aws.amazon.com/elasticloadbalancing/latest/netw...