See drewg's comment. He is being modest in that they are also doing CPU encryption at 100G, so every single stream is different which doubles the usage of memo

See drewg's comment. He is being modest in that they are also doing CPU encryption at 100G, so every single stream is different which doubles the usage of memory bandwidth. I will get to 100G with Skylake, mainly because we have to rework our storage BoM rather than CPU improvements. Intel's focus has been off, they've somewhat misread where the market was going, but even today you have 40 PCIe 3.0 lanes (39GB/s), 67GB/s DDR4 memory bandwidth, and typically more than enough cores and threads to do whatever you want in a single E5 Xeon socket. Computers are _really_ fast, software is slow :) So right out the gate, you have plenty of speeds and feeds to get stuff off disk and out the wire. That's exactly my workload, which entails pulling data off storage, into the VFS while kqueue is managing a pool of connected sockets, and when they are ready for more data, it goes out right from the page cache with sendfile. Netflix contributed some amazing work that makes FreeBSD particularly optimized for this workload. In general the trick is to do less, batch more, and try not to copy things around. For example DPDK or Netmap packet forwarding, clear an entire soft ring at a time instead of one packet at a time. Using netmap, you can change ownership of the data by pointer swapping to move it from the rx ring to tx. The ACM Queue article on netmap is particularly good reading. Basically, pass by reference, but by understanding the memory layout of the system. aio and sendfile kind of suck on Linux. epoll kind of sucks. Linux hugepages really suck, and the Linux VM seems to be biased toward massive concurrency or something that I don't really understand. None of these are monumental technical problems, but there is malignant culture at these hyperscalers because just being a bandwagon fan doesn't make you a winning team. Linux users tend to trust manufacturers to do all the device driver development correctly, and vendors like RedHat to drive general forward progress. How many patches does Amazon have in Xen? Linux? At the BSD companies I've mentioned, a few dozen people across all of them have pulled this stuff off, and it's all there in the base system. We rip apart vendor drivers or entire subsystems when that's the prescription. We're pretty happy to share details and help others succeed at conferences or even by partnering up team to team across companies.