4 ms·
One thing to remember is that high-speed packet filtering is an unusual workflow and CloudFlare operates at a much greater scale than most of us see: most Linux
by acdha 11y ago
One thing to remember is that high-speed packet filtering is an unusual workflow and CloudFlare operates at a much greater scale than most of us see: most Linux devices are not connected to 10G, much less 100G, networks and they're usually doing more work than looking at a packet to decide whether to accept or reject it. The fact that the APIs and the kernel stack were designed many years before those kind of speeds were possible doesn't matter because most sites don't have that much traffic and most server applications will bottleneck at doing other work well before that point.
The example in the article found a single core handling 1.4M packets per second. If you're running a web-server shoveling data out to clients those packets are going to be close to the maximum size which, if I haven't screwed up the math, looks something like this:
1.4M * 1400 bytes (assuming a low MTU) * 8 (bytes -> bits) = 15Gbps
That's not to say that there isn't still plenty of room to improve and, as lukego noted, there's a lot of work in progress (see e.g. https://lwn.net/Articles/615238/ https://lwn.net/Articles/615238/ on work to batch operations to avoid paying some of the processing costs for every packet) but for the average server you'd find bottlenecks on something like a database, application logic, request handling, client network capacity, etc. before the network stack overhead is your greatest challenge. The people who encounter this tend to be CDN vendors like CloudFlare and security people who need to filter, analyze, or generate traffic on levels which are at least the the scale of a large company (e.g. https://github.com/robertdavidgraham/masscan https://github.com/robertdavidgraham/masscan).
- lukego 11y agoPeople are also spending hundreds of billions of dollars on equipment like routers ever year. These could be Linux boxes if the kernel had sensible performance. I am kind of amazed that the Linux kernel did not become the dominant data-plane for the networking industry ahead of proprietary implementations from Cisco, Juniper, etc. Hopefully Snabb Switch will have better luck there... ;-)
- asdfaoeu 11y agoASICs are always going to be way faster than a Linux kernel.
- __d 11y agoMost switch/router dataplane processing is done in hardware, with control by custom drivers in the control-plane OS. Cisco IOS is variously hosted on a proprietary real-time OS, QNX or Linux. Juniper JunOS is FreeBSD-based. Arista AOS is Linux-based. With the advent of SDN, the packet-rate limitations of a general-purpose OS lead to things like DPDK and Snabb, but they both run within a Linux host environment (DPDK can use FreeBSD as well; not sure re: Snabb).
- scurvy 11y agoSorry if I sound mean, but this is just a long apologist post about how things are just so hard. Really? Why? Why can't Linux match BSD's performance? Also, 10gb servers are not rare by any means. Take a look around the next time you walk in a colo. 10 gb servers everywhere.
- justincormack 11y agoWell the BSDs have issues too. FreeBSD/Netflix has been doing in kernel SSL to try to fix some issues, and there is Netmap available for packet processing in userspace. So they are running into similar issues as Linux.
- gonzo 11y agoNetflix is doing in-kernel SSL so they don't have to hit the web server to encrypt the disk blocks for their streaming movies. That doesn't mean it's a (Free)BSD issue, just that Netflix chose a particular architecture and are doing the work to make it go fast. Netmap is in FreeBSD, and available for linux. I think it might be in Dragonfly at this point as well.
- tedunangst 11y agoI'm not certain if "apologist post" refers to the article or the comment you replied to, but neither mentions BSD. ?
- acdha 11y ago> Sorry if I sound mean, but this is just a long apologist post about how things are just so hard. Really? Why? Why can't Linux match BSD's performance? Mostly I just wish you'd read it again: you appear to have missed the part where I said that this is a real problem which needed working on. The point I was making is that it's not a problem for most Linux users. Linux includes millions of devices attached to sub-100Mb networks but even if you want to look solely at things in modern data centers ask yourself how many of them are running network-limited applications or are providing services to internet users over an uplink which is actually fast enough to stress modern hardware. For all but the most demanding users the Linux vs. BSD decision will be made on other factors.
- zurn 11y agoWhen 100M ethernet was introduced in the 90s, PCs had Pentiums, at < 100 M instructions per second. Both x86 CPUs and Ethernet have gotten ~1000x faster in the same time - cf 100G Ethernet and processors with 30x frequency, ~2x more IPC, and 20x more cores.
- tedunangst 11y agoYou haven't accounted for memory bandwidth and latency. And counting increased core count means you can't actually assume the same old single threaded algorithm will scale linearly.
- zurn 11y agoThose are real-world speed bumps that make sense when you want to do stateful processing w/ large working sets, but not really for this "just get packets through the stack" toy benchmark that fits in cache and the workload is parallel-friendly. And also the article is talking about handling just 10G, not 100G!
- X-Istence 11y agoI work at an ISP, we are starting to connect stuff at 40G (times 2 for redundancy) because we want a single machine to do more work. Improving the kernel would greatly speed up many of our applications with no real downsides. With 40G to the edge, we also need to be able to properly firewall/filter/and all that fun stuff the traffic!
- gonzo 11y ago> One thing to remember is that high-speed packet filtering is an unusual workflow and CloudFlare operates at a much greater scale than most of us see: most Linux devices are not connected to 10G You're about to see a whole bunch more 10G in the coming 5 years.
- gonzo 11y ago> One thing to remember is that high-speed packet filtering is an unusual workflow and CloudFlare operates at a much greater scale than most of us see: most Linux devices are not connected to 10G You're about to see a whole bunch more 10G in the coming 5 years.