I didn't mean to minimize the difficulties of maintaining that many TCP connections (much less getting useful work out of them). I read the original article whe

I didn't mean to minimize the difficulties of maintaining that many TCP connections (much less getting useful work out of them). I read the original article when it was on HN, but must have mentally thrown most of it away due to the freebsd bias. So I just reread it, and the fact that they are getting those numbers utilizing much of the OS buffer management and Nginx, is impressive by itself. But their difficulties sort of plays into my original assumptions. Basically, if you want cutting edge I/O perf your better off dumping most general purpose OS's I/O stacks unless you want to spend a lot of time re-engineering them to work around bottlenecks. sendfile() is good, but the general concept tends to waste far to much time doing filesystem traversals, buffer management, dma scatter gather lists, and a bunch of other crap that gets in the way of getting a blob of data from the disk, encrypting it, and passing it off to a send offload to handle breaking up and apply the TCP/IP headers/checksums. Frankly the minimum MSS size is something that ipv6 should have fixed, given that no one is on 9600bps modems, but didn't. Good for them for realizing that modern machines have a little less than a GB of bandwidth per pcie lane per direction, and memory bandwidth to match. If you don't mess up the CPU side of things you can even touch all that data once or twice and still maintain pretty amazing I/O numbers. EDIT: Also in the case of x86 NUMA, you _REALLY_ want to make sure that the nvme/source disk, the memory buffer your writing to and the network adapter are on the same node with the core doing the encryption/etc. That is pretty easy if the "application" controls buffer allocation/pooling, but much harder with a general purpose OS which will fragment the memory pools.