3 ms·

Since you worked at WhatsApp, you'd have seen it first hand how Rick Reed et al pushed the boundaries of FreeBSD network stack and Erlang BEAM... 2M+ (TCP) long

Since you worked at WhatsApp, you'd have seen it first hand how Rick Reed et al pushed the boundaries of FreeBSD network stack and Erlang BEAM... 2M+ (TCP) long-lived connections per server, though presumably to different (CGNAT?) destinations back in 2012: https://blog.whatsapp.com/1-million-is-so-2011/ https://blog.whatsapp.com/1-million-is-so-2011/ Acknowledging that WhatsApp may have never hit the limits Cloudflare has, did FreeBSD make it easier than it would have been on Linux [0]? Is it common for network load balancers to instead bypass the kernel (xdp, netmap, snabbswitch etc) and take over connection management entirely (perhaps, along with a hand-rolled TCP/IP flow manager)? [0] I know Linux isn't all that bad, Marek: https://blog.cloudflare.com/how-to-receive-a-million-packets/ https://blog.cloudflare.com/how-to-receive-a-million-packets... (:

toast0 5y ago
That's 2M+ inbound connections[1]. Inbound connections are relatively easy, because client IPs are pretty diverse, even when there's CGNAT. This is way easier than a proxy load like Cloudflare does. I briefly ran some HAProxy servers that helped finalize the migration to Facebook; that was 1:1 each inbound connection (that managed to send a valid enough hello packet in a minute) got an outbound connection --- that was a lot harder. I was able to do a pretty good amount of tuning for that specific case by mucking about in the (FreeBSD) kernel; IMHO, the Linux kernel isn't as easy to muck about with, but then I wasn't ever eased into it; and it's possible it wouldn't have needed as much mucking about, also FreeBSD 13 has some interesting looking improvements which may have addressed the bottlenecks I was seeing, but I no longer have access to hyperscale optimization problems. :) TLDR; to get haproxy + FreeBSD 12 to do the max 1:1 tcp proxying, use only as many CPUs as RSS-able NIC RX queues (ideally don't have 2x14-core servers when you can only use 16 cpus, but getting new hardware wasn't an option), run one haproxy process per CPU (pinned), use RSS on inbound sockets and select outgoing ip + port so that outgoing connections use the same cpu (search for my patches on the haproxy list), then tweak the kernel a bit, I think my patches are floating around on FreeBSD phabricator or a mailing list somewhere, but they're not great; I did some tweaking to the tcb hashing and IIRC a little bit of mucking around with port selection/verfication, but it's all a bit hazy, the bottleneck was definitely kernel locks around inserting new outbound connections. WhatsApp pre-Facebook didn't use network load balancers; DNS round robin was good enough for us. We tried our hosting provider's load balancers for part of our service, but the uptime of the load balancers was worse than the uptime of the two servers behind it, so we didn't consider it for anything else; connection limits were too low for chat anyway. Facebook uses proxygen for network load balancers, but I don't recall hearing anything about kernel bypass. They're mostly http(s), and they do request inspection and some responses from the edge, so IMHO, it doesn't fit the usual case for hyperoptimizing the user/kernel interface: it's more appropriate to focus on user/kernel interface when you're cpu limited, but your application is doing very little. If you had traditional layer 2 load balancer (aka, the good ones that let you do direct server return), I'd imagine it looks a lot closer to a kernel table based NAT than an HAProxy; and that way you'd have most (if not all) of the traffic handled by the kernel rather than ever going to the application. I wanted to build something fun like that for the proxy, but I also wanted something that I could setup and tune, and not feel guilty about leaving a mess for someone when I left. I really really actually wanted to build a kernel bypassed userspace tcp stack for proxying, in Erlang, where I could hot-load changes, cause it's a giant pain to drain hundreds of thousands of connections so you can update the kernel without disrupting users; way more fun to hot load your tcp stack. Alas. [1] I think Rick posted somewhere when we accidentally ran past 3 Million connections on a chatd (nope, it was only 2.8M[2]), but the connection numbers went down a lot as we did better client to server encryption and authentication, and as the chat protocol got more complex, so we ran out of CPU because of our server logic. Facebook hosted WhatsApp server nodes are way smaller machines (or were, when I left; single socket, less cores, less ram), so the server count jumped quite a bit. [2] http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf http://www.erlang-factory.com/upload/presentations/558/efsf2... slide 16 Edit: Sorry, this comment is full of runon sentences and kind of mostly just a braindump of anything related to the prompt. Happy to take questions, either here (if at least tangentally related to the thread topic of lots of outbound connections) or by email, and maybe I'll make better sentences with followup questions :)

toast0 5y ago

That's 2M+ inbound connections[1]. Inbound connections are relatively easy, because client IPs are pretty diverse, even when there's CGNAT. This is way easier than a proxy load like Cloudflare does. I briefly ran some HAProxy servers that helped finalize the migration to Facebook; that was 1:1 each inbound connection (that managed to send a valid enough hello packet in a minute) got an outbound connection --- that was a lot harder. I was able to do a pretty good amount of tuning for that specific case by mucking about in the (FreeBSD) kernel; IMHO, the Linux kernel isn't as easy to muck about with, but then I wasn't ever eased into it; and it's possible it wouldn't have needed as much mucking about, also FreeBSD 13 has some interesting looking improvements which may have addressed the bottlenecks I was seeing, but I no longer have access to hyperscale optimization problems. :) TLDR; to get haproxy + FreeBSD 12 to do the max 1:1 tcp proxying, use only as many CPUs as RSS-able NIC RX queues (ideally don't have 2x14-core servers when you can only use 16 cpus, but getting new hardware wasn't an option), run one haproxy process per CPU (pinned), use RSS on inbound sockets and select outgoing ip + port so that outgoing connections use the same cpu (search for my patches on the haproxy list), then tweak the kernel a bit, I think my patches are floating around on FreeBSD phabricator or a mailing list somewhere, but they're not great; I did some tweaking to the tcb hashing and IIRC a little bit of mucking around with port selection/verfication, but it's all a bit hazy, the bottleneck was definitely kernel locks around inserting new outbound connections. WhatsApp pre-Facebook didn't use network load balancers; DNS round robin was good enough for us. We tried our hosting provider's load balancers for part of our service, but the uptime of the load balancers was worse than the uptime of the two servers behind it, so we didn't consider it for anything else; connection limits were too low for chat anyway. Facebook uses proxygen for network load balancers, but I don't recall hearing anything about kernel bypass. They're mostly http(s), and they do request inspection and some responses from the edge, so IMHO, it doesn't fit the usual case for hyperoptimizing the user/kernel interface: it's more appropriate to focus on user/kernel interface when you're cpu limited, but your application is doing very little. If you had traditional layer 2 load balancer (aka, the good ones that let you do direct server return), I'd imagine it looks a lot closer to a kernel table based NAT than an HAProxy; and that way you'd have most (if not all) of the traffic handled by the kernel rather than ever going to the application. I wanted to build something fun like that for the proxy, but I also wanted something that I could setup and tune, and not feel guilty about leaving a mess for someone when I left. I really really actually wanted to build a kernel bypassed userspace tcp stack for proxying, in Erlang, where I could hot-load changes, cause it's a giant pain to drain hundreds of thousands of connections so you can update the kernel without disrupting users; way more fun to hot load your tcp stack. Alas. [1] I think Rick posted somewhere when we accidentally ran past 3 Million connections on a chatd (nope, it was only 2.8M[2]), but the connection numbers went down a lot as we did better client to server encryption and authentication, and as the chat protocol got more complex, so we ran out of CPU because of our server logic. Facebook hosted WhatsApp server nodes are way smaller machines (or were, when I left; single socket, less cores, less ram), so the server count jumped quite a bit. [2] http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf http://www.erlang-factory.com/upload/presentations/558/efsf2... slide 16 Edit: Sorry, this comment is full of runon sentences and kind of mostly just a braindump of anything related to the prompt. Happy to take questions, either here (if at least tangentally related to the thread topic of lots of outbound connections) or by email, and maybe I'll make better sentences with followup questions :)