3 ms·

28000 ephemeral ports is enough for the c10k problem, but 100k+ seems a stretch. At what point is increasing the number of ports from 28k to a higher number the

28000 ephemeral ports is enough for the c10k problem, but 100k+ seems a stretch. At what point is increasing the number of ports from 28k to a higher number the right answer? Reuse as described here sounds like a useful optimization but at some point (or due to pathological workload) even that will be exhausted I’d think? What to do then

toast0 5y ago
> At what point is increasing the number of ports from 28k to a higher number the right answer? You can increase it to 65535, although you'll get funny looks from some people. Beyond that, you'd need TCP and UDP extensions, which seems unlikely to be deployed. Usually it's OK, you rarely need to connect more than 65,000 times to a single host; but if you do, you can try to convince them to open more listening ports, or use more listening IPs or more connecting IPs --- one way to encourage IPv6. Really, the socket API should be updated to a single syscall that binds and connects (or listens) and lets you express what you actually need. Whether it's to have a whole port reserved, or just need a currently unique 4-tuple with no local information selection, or you've got one piece of the local information; or maybe you want a connection that will hash to the same CPU you're currently on. If you control all of the outgoing connections, you can do this work in userspace, but it's challenging; if you're running multiple processes that share the 4-tuple address space, it's a lot harder (PS, try to partition the space so processes don't compete). Turning two (or more) syscalls into a single one is useful anyway, especially if you've got meltdown mitigations turned on.
- ignoramous 5y ago
  Since you worked at WhatsApp, you'd have seen it first hand how Rick Reed et al pushed the boundaries of FreeBSD network stack and Erlang BEAM... 2M+ (TCP) long-lived connections per server, though presumably to different (CGNAT?) destinations back in 2012: https://blog.whatsapp.com/1-million-is-so-2011/ https://blog.whatsapp.com/1-million-is-so-2011/ Acknowledging that WhatsApp may have never hit the limits Cloudflare has, did FreeBSD make it easier than it would have been on Linux [0]? Is it common for network load balancers to instead bypass the kernel (xdp, netmap, snabbswitch etc) and take over connection management entirely (perhaps, along with a hand-rolled TCP/IP flow manager)? [0] I know Linux isn't all that bad, Marek: https://blog.cloudflare.com/how-to-receive-a-million-packets/ https://blog.cloudflare.com/how-to-receive-a-million-packets... (:
  - toast0 5y ago
    That's 2M+ inbound connections[1]. Inbound connections are relatively easy, because client IPs are pretty diverse, even when there's CGNAT. This is way easier than a proxy load like Cloudflare does. I briefly ran some HAProxy servers that helped finalize the migration to Facebook; that was 1:1 each inbound connection (that managed to send a valid enough hello packet in a minute) got an outbound connection --- that was a lot harder. I was able to do a pretty good amount of tuning for that specific case by mucking about in the (FreeBSD) kernel; IMHO, the Linux kernel isn't as easy to muck about with, but then I wasn't ever eased into it; and it's possible it wouldn't have needed as much mucking about, also FreeBSD 13 has some interesting looking improvements which may have addressed the bottlenecks I was seeing, but I no longer have access to hyperscale optimization problems. :) TLDR; to get haproxy + FreeBSD 12 to do the max 1:1 tcp proxying, use only as many CPUs as RSS-able NIC RX queues (ideally don't have 2x14-core servers when you can only use 16 cpus, but getting new hardware wasn't an option), run one haproxy process per CPU (pinned), use RSS on inbound sockets and select outgoing ip + port so that outgoing connections use the same cpu (search for my patches on the haproxy list), then tweak the kernel a bit, I think my patches are floating around on FreeBSD phabricator or a mailing list somewhere, but they're not great; I did some tweaking to the tcb hashing and IIRC a little bit of mucking around with port selection/verfication, but it's all a bit hazy, the bottleneck was definitely kernel locks around inserting new outbound connections. WhatsApp pre-Facebook didn't use network load balancers; DNS round robin was good enough for us. We tried our hosting provider's load balancers for part of our service, but the uptime of the load balancers was worse than the uptime of the two servers behind it, so we didn't consider it for anything else; connection limits were too low for chat anyway. Facebook uses proxygen for network load balancers, but I don't recall hearing anything about kernel bypass. They're mostly http(s), and they do request inspection and some responses from the edge, so IMHO, it doesn't fit the usual case for hyperoptimizing the user/kernel interface: it's more appropriate to focus on user/kernel interface when you're cpu limited, but your application is doing very little. If you had traditional layer 2 load balancer (aka, the good ones that let you do direct server return), I'd imagine it looks a lot closer to a kernel table based NAT than an HAProxy; and that way you'd have most (if not all) of the traffic handled by the kernel rather than ever going to the application. I wanted to build something fun like that for the proxy, but I also wanted something that I could setup and tune, and not feel guilty about leaving a mess for someone when I left. I really really actually wanted to build a kernel bypassed userspace tcp stack for proxying, in Erlang, where I could hot-load changes, cause it's a giant pain to drain hundreds of thousands of connections so you can update the kernel without disrupting users; way more fun to hot load your tcp stack. Alas. [1] I think Rick posted somewhere when we accidentally ran past 3 Million connections on a chatd (nope, it was only 2.8M[2]), but the connection numbers went down a lot as we did better client to server encryption and authentication, and as the chat protocol got more complex, so we ran out of CPU because of our server logic. Facebook hosted WhatsApp server nodes are way smaller machines (or were, when I left; single socket, less cores, less ram), so the server count jumped quite a bit. [2] http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf http://www.erlang-factory.com/upload/presentations/558/efsf2... slide 16 Edit: Sorry, this comment is full of runon sentences and kind of mostly just a braindump of anything related to the prompt. Happy to take questions, either here (if at least tangentally related to the thread topic of lots of outbound connections) or by email, and maybe I'll make better sentences with followup questions :)
uberduper 5y ago
Additional IPs on the interface to use as a source.
- a-dub 5y ago
  aliasing. that's the solution i always knew to use. think it was added in linux 2.x. edit: guess it's been deprecated in recent kernels and it was added to linux 1.2 in 1995. apparently now the ip networking stack just understands multiple addresses per interface. defeating time_wait is a bad idea, the source ephemeral port is how routers, firewalls, nats and other hosts identify a stream. reusing too soon can confuse all of them.

toast0 5y ago

> At what point is increasing the number of ports from 28k to a higher number the right answer? You can increase it to 65535, although you'll get funny looks from some people. Beyond that, you'd need TCP and UDP extensions, which seems unlikely to be deployed. Usually it's OK, you rarely need to connect more than 65,000 times to a single host; but if you do, you can try to convince them to open more listening ports, or use more listening IPs or more connecting IPs --- one way to encourage IPv6. Really, the socket API should be updated to a single syscall that binds and connects (or listens) and lets you express what you actually need. Whether it's to have a whole port reserved, or just need a currently unique 4-tuple with no local information selection, or you've got one piece of the local information; or maybe you want a connection that will hash to the same CPU you're currently on. If you control all of the outgoing connections, you can do this work in userspace, but it's challenging; if you're running multiple processes that share the 4-tuple address space, it's a lot harder (PS, try to partition the space so processes don't compete). Turning two (or more) syscalls into a single one is useful anyway, especially if you've got meltdown mitigations turned on.

ignoramous 5y ago
Since you worked at WhatsApp, you'd have seen it first hand how Rick Reed et al pushed the boundaries of FreeBSD network stack and Erlang BEAM... 2M+ (TCP) long-lived connections per server, though presumably to different (CGNAT?) destinations back in 2012: https://blog.whatsapp.com/1-million-is-so-2011/ https://blog.whatsapp.com/1-million-is-so-2011/ Acknowledging that WhatsApp may have never hit the limits Cloudflare has, did FreeBSD make it easier than it would have been on Linux [0]? Is it common for network load balancers to instead bypass the kernel (xdp, netmap, snabbswitch etc) and take over connection management entirely (perhaps, along with a hand-rolled TCP/IP flow manager)? [0] I know Linux isn't all that bad, Marek: https://blog.cloudflare.com/how-to-receive-a-million-packets/ https://blog.cloudflare.com/how-to-receive-a-million-packets... (:
- toast0 5y ago
  That's 2M+ inbound connections[1]. Inbound connections are relatively easy, because client IPs are pretty diverse, even when there's CGNAT. This is way easier than a proxy load like Cloudflare does. I briefly ran some HAProxy servers that helped finalize the migration to Facebook; that was 1:1 each inbound connection (that managed to send a valid enough hello packet in a minute) got an outbound connection --- that was a lot harder. I was able to do a pretty good amount of tuning for that specific case by mucking about in the (FreeBSD) kernel; IMHO, the Linux kernel isn't as easy to muck about with, but then I wasn't ever eased into it; and it's possible it wouldn't have needed as much mucking about, also FreeBSD 13 has some interesting looking improvements which may have addressed the bottlenecks I was seeing, but I no longer have access to hyperscale optimization problems. :) TLDR; to get haproxy + FreeBSD 12 to do the max 1:1 tcp proxying, use only as many CPUs as RSS-able NIC RX queues (ideally don't have 2x14-core servers when you can only use 16 cpus, but getting new hardware wasn't an option), run one haproxy process per CPU (pinned), use RSS on inbound sockets and select outgoing ip + port so that outgoing connections use the same cpu (search for my patches on the haproxy list), then tweak the kernel a bit, I think my patches are floating around on FreeBSD phabricator or a mailing list somewhere, but they're not great; I did some tweaking to the tcb hashing and IIRC a little bit of mucking around with port selection/verfication, but it's all a bit hazy, the bottleneck was definitely kernel locks around inserting new outbound connections. WhatsApp pre-Facebook didn't use network load balancers; DNS round robin was good enough for us. We tried our hosting provider's load balancers for part of our service, but the uptime of the load balancers was worse than the uptime of the two servers behind it, so we didn't consider it for anything else; connection limits were too low for chat anyway. Facebook uses proxygen for network load balancers, but I don't recall hearing anything about kernel bypass. They're mostly http(s), and they do request inspection and some responses from the edge, so IMHO, it doesn't fit the usual case for hyperoptimizing the user/kernel interface: it's more appropriate to focus on user/kernel interface when you're cpu limited, but your application is doing very little. If you had traditional layer 2 load balancer (aka, the good ones that let you do direct server return), I'd imagine it looks a lot closer to a kernel table based NAT than an HAProxy; and that way you'd have most (if not all) of the traffic handled by the kernel rather than ever going to the application. I wanted to build something fun like that for the proxy, but I also wanted something that I could setup and tune, and not feel guilty about leaving a mess for someone when I left. I really really actually wanted to build a kernel bypassed userspace tcp stack for proxying, in Erlang, where I could hot-load changes, cause it's a giant pain to drain hundreds of thousands of connections so you can update the kernel without disrupting users; way more fun to hot load your tcp stack. Alas. [1] I think Rick posted somewhere when we accidentally ran past 3 Million connections on a chatd (nope, it was only 2.8M[2]), but the connection numbers went down a lot as we did better client to server encryption and authentication, and as the chat protocol got more complex, so we ran out of CPU because of our server logic. Facebook hosted WhatsApp server nodes are way smaller machines (or were, when I left; single socket, less cores, less ram), so the server count jumped quite a bit. [2] http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf http://www.erlang-factory.com/upload/presentations/558/efsf2... slide 16 Edit: Sorry, this comment is full of runon sentences and kind of mostly just a braindump of anything related to the prompt. Happy to take questions, either here (if at least tangentally related to the thread topic of lots of outbound connections) or by email, and maybe I'll make better sentences with followup questions :)

uberduper 5y ago

Additional IPs on the interface to use as a source.

a-dub 5y ago
aliasing. that's the solution i always knew to use. think it was added in linux 2.x. edit: guess it's been deprecated in recent kernels and it was added to linux 1.2 in 1995. apparently now the ip networking stack just understands multiple addresses per interface. defeating time_wait is a bad idea, the source ephemeral port is how routers, firewalls, nats and other hosts identify a stream. reusing too soon can confuse all of them.