Think differently about this problem. Not all applications are web pages where the lifecycle of a given connection is "establish TCP, establish TLS, receive se

Think differently about this problem. Not all applications are web pages where the lifecycle of a given connection is "establish TCP, establish TLS, receive series of requests and produce series of responses mostly by hitting external caches or DBs" [or the QUIC variant of this]. That problem space was one of the very first scale challenges to arrive in 1998 and was one of the very first that actually got addressed. It's not the challenge space now and has not been, basically, for 20+ years, except perhaps the whole scaling-of-database problem, which has been dealt with by sharding, and the distribution of flows problem, which has mostly been solved with clever applications of intensely performant _scale-up_ hardware in the form of modern switch NPUs running variations of ECMP and clever uses of anycast, DNS load balancing, routing, etc. All of this stuff was pretty common by the mid-2000s. But bottlenecks in systems always exist. They move around. They can be anywhere in the stack. Connections is the simplest one that got a lot of attention for twenty years - real pre-emptive threads (in Linux, and Solaris's weird diversion into m:n), select() scalability (both in terms of bookkeeping and in terms of basic limits - that is, the lack thereof) giving rise to kqueue on freebsd, WFMO() on NT and years of attempts on Linux to get something that actually worked, which took awhile, _then_ the c10k problem, and so on. After connectivity you have issues in layer 3 - TCP offload, cost of TLS, etc. Userland to kernel copies (basic stuff like sendfile() to different userland driver schemes). And so on. Physical layer - servers move from NICs with lots of copies, to ring based with scatter-gather, to TSO, crypto offload, to ... while going from 10mb to 100, 1000, 2.5g, 10g, 40g, 100G and sooner or later 400G on server will not be as uncommon as it is now. But as your networking capacity and throughput scale, you start bumping into other things. Once you're talking high speed links and various schemes to get the kernel out of the way, you are mostly - not always, but mostly - talking about data movement problems. Elephant flows have their own system level problems in networks, and for data moving and staging you actually don't want the host _cpu_ involved if you can avoid it, let alone the kernel. Now you are in the area of doing (remote)->NIC----PCIe---->NVME (or --->GPU) directly, if you can. Now your NVME storage device becomes the bottleneck. 90s era supercomputing clusters had all of these problems with slightly different technologies, AI clusters have them today. These are not connection limited, they do not scale with people. Their primary scale challenge is utilization/CAPEX, but that's a longer discussion.