5 ms·

One overlooked point is ephemeral source port exhaustion. If a load balancer forwards a HTTP connection to a backend system, it needs a TCP source port for the

One overlooked point is ephemeral source port exhaustion. If a load balancer forwards a HTTP connection to a backend system, it needs a TCP source port for the duration of that connection (not destination port, which is probably 80 or 443). That limits the number of outgoing connections to less than 65535. A common workaround is to use more outgoing IP addresses to the backends as source IPs, thus multiplying the available number of source ports to 65535 times number_of_ips. HTTP/2 solves this, as you can multiplex requests to backend servers over a single TCP socket. So there is actually a point of using HTTP/2 for load_balancer <-> backend_system connections.

mike_d 2y ago
This has been solved for 10+ years. Properly configure your load balancer with HTTP keep alive and piplining.
- littlecranky67 2y ago
  HTTP keep-alive still limits the number of outgoing connections to 65535. Pipelining suffers from the known same issues addressed in the article. But I agree, it is a solved problem unless you really have a lot of incoming connections. When you use multiple outgoing ip addresses that fixes that even for very busy load balancers, and since IPv6 is common today you will likely have a /64 to draw addresses from.
  - mike_d 2y ago
    On modern systems you have about 28k ephemeral ports available. 65,535 is the total number of ports (good luck trying to use them all). Either way, if you have more than 20k connections open to a single backend (remember linux does connection tracking using the 4 tuple, so you can reuse a source port to different destinations) you are doing something seriously wrong and should hire competent network engineering folks.
    - littlecranky67 2y ago
      > Either way, if you have more than 20k connections open to backends you are doing something seriously wrong I don't see how that is a fringe or rare case. With a loadbalancer (using no pipelining or multiplexing), the number of simultaenous outgoing http connections to backend systems is at least the number of simulatenous open incoming http connections. Having more than 28k simultanous incoming http requests is not a lot for a busy load balancer. Now with pipelining (or limiting to 28k outgoing connections), the loadbalancer has to queue requests and multiplex them to the backends when connections become available. Pipelining suffers from head-of-line blocking, increasing possible latency caused by the loadbalancer further. In any case, you will increase latency to the end-user by queing. If you use HTTP/2 multiplexing, you can go past those 28k incoming connections without queing on the loadbalancer side.
      - mike_d 2y ago
        > (using no pipelining or multiplexing), "Doctor it hurts when I hit myself" > the number of simultaenous outgoing http connections to backend systems is at least the number of simulatenous open incoming http connections No it isn't. You establish a pool of long lived connections per backend. The load balancer should be coalescing in flight requests. At that traffic volume you should also be doing basic in-memory caches to sink things like favicon requests. I am not going to respond further as this chain is getting quite off topic. There are plenty of good resources available from relevant Google searches, but if you really still have questions about how load balancers work my email is in my profile.
        littlecranky67 2y ago
        > You establish a pool of long lived connections per backend Yes, and you would do the same with HTTP/2. You haven't addressed the head-of-line blocking problem caused by HTTP/1.1 pipelining, which HTTP/2 completely solves. Head-of-line blocking becomes an increasing issue when your HTTP connections are long lived, such as when using websockets or large-media transfers or streaming.

mike_d 2y ago

This has been solved for 10+ years. Properly configure your load balancer with HTTP keep alive and piplining.

littlecranky67 2y ago
HTTP keep-alive still limits the number of outgoing connections to 65535. Pipelining suffers from the known same issues addressed in the article. But I agree, it is a solved problem unless you really have a lot of incoming connections. When you use multiple outgoing ip addresses that fixes that even for very busy load balancers, and since IPv6 is common today you will likely have a /64 to draw addresses from.
- mike_d 2y ago
  On modern systems you have about 28k ephemeral ports available. 65,535 is the total number of ports (good luck trying to use them all). Either way, if you have more than 20k connections open to a single backend (remember linux does connection tracking using the 4 tuple, so you can reuse a source port to different destinations) you are doing something seriously wrong and should hire competent network engineering folks.
  - littlecranky67 2y ago
    > Either way, if you have more than 20k connections open to backends you are doing something seriously wrong I don't see how that is a fringe or rare case. With a loadbalancer (using no pipelining or multiplexing), the number of simultaenous outgoing http connections to backend systems is at least the number of simulatenous open incoming http connections. Having more than 28k simultanous incoming http requests is not a lot for a busy load balancer. Now with pipelining (or limiting to 28k outgoing connections), the loadbalancer has to queue requests and multiplex them to the backends when connections become available. Pipelining suffers from head-of-line blocking, increasing possible latency caused by the loadbalancer further. In any case, you will increase latency to the end-user by queing. If you use HTTP/2 multiplexing, you can go past those 28k incoming connections without queing on the loadbalancer side.
    - mike_d 2y ago
      > (using no pipelining or multiplexing), "Doctor it hurts when I hit myself" > the number of simultaenous outgoing http connections to backend systems is at least the number of simulatenous open incoming http connections No it isn't. You establish a pool of long lived connections per backend. The load balancer should be coalescing in flight requests. At that traffic volume you should also be doing basic in-memory caches to sink things like favicon requests. I am not going to respond further as this chain is getting quite off topic. There are plenty of good resources available from relevant Google searches, but if you really still have questions about how load balancers work my email is in my profile.
      - littlecranky67 2y ago
        > You establish a pool of long lived connections per backend Yes, and you would do the same with HTTP/2. You haven't addressed the head-of-line blocking problem caused by HTTP/1.1 pipelining, which HTTP/2 completely solves. Head-of-line blocking becomes an increasing issue when your HTTP connections are long lived, such as when using websockets or large-media transfers or streaming.