7 ms·

The CPU Cost of Networking on a Linux Host

192 points by sohkamyung 7y ago

ra1n85 7y ago
For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architectures. Almost no packets are forwarded by the same processors running management and control plane functions anymore. This is mainly because the required traffic rates today need dedicated silicon purpose built for the task (the Broadcom Tomahawk3 can do 12.8 Terabits/sec above a relatively small packet size). I don't know how things will shake out for the Linux world and x86 packet forwarding given the trend and lack of real performance in the kernel. Right now, your best bet when it comes to Linux and high network throughput/packet processing requirements is to just bypass the kernel entirely with DPDK, a "smart" NIC, or XDP.
- diegocg 7y ago
  These is the most up-to-date article I've seen on these matters https://lwn.net/Articles/787754/ https://lwn.net/Articles/787754/ https://lwn.net/Articles/629155/ https://lwn.net/Articles/629155/
- EvanAnderson 7y ago
  I wonder if there's an economic argument, at useful scales, to using FPGA's in general purpose servers to accelerate network performance. The purpose-built ASICs would win on cost-per-unit every time, but the FPGA would have some adaptability to new protocols or algorithms that the ASIC wouldn't.
  - wmf 7y ago
    Yes, Azure uses FPGAs for networking. Programmable NPUs like Netronome or Pensando are a middle ground that are more cost-efficient than FPGAs for most needs.
  - yummypaint 7y ago
    Ive been waiting for mainstream workstation motherboards with this capability for years. Presumably a pci card is how this would be handled in practice for now. My naive take is that toolchains are still too convoluted and bogged down with licensing schemes to deliver the kind of real-time highly integrated adaptability this would require for individual users. Would be great if the barrier to entry has in fact dropped enough that university-sized networks could consider them.
  - karatekidd32v 7y ago
    There's some cool tech going on at barefoot (https://barefootnetworks.com/ https://barefootnetworks.com/) I don't know much about the technical details, but the pitch I've heard is that it gives you ASIC level performance with more flexibility to reprogram the chip (not full FPGA).
    - ra1n85 7y ago
      Yeah, it's an interesting approach. They're basically allowing you to define packet processing with P4 on their Tofino family of chipsets: https://p4.org/ https://p4.org/ That said, there's only so much you can do in a chip before considerable tradeoffs are going to be made. They're not going to offer the same level of flexibility you get out of a general purpose CPU, but may not have same the restrictions of most fixed pipeline chips - their product sits somewhere in the middle. Also, P4 seems to sit in a space complex enough to make it unreasonable for most network shops - it's not for your average enterprise or service provider network.
- tw04 7y ago
  That's not entirely accurate - Cisco open sourced VPP which they use in many products: https://blogs.cisco.com/sp/a-bigger-helping-of-internet-please https://blogs.cisco.com/sp/a-bigger-helping-of-internet-plea... https://docs.fd.io/vpp/16.09/ https://docs.fd.io/vpp/16.09/ While they still obviously make custom ASICs, they've moved towards more software where possible.
  - ra1n85 7y ago
    Sorry, what's not accurate here?
  - Hikikomori 7y ago
    Pretty much everything above their small medium business ISR routers use ASIC/NPU for all traffic possible. Which routers are you referring to?
- bogomipz 7y ago
  >"In the last 15 years there has been a hard move away from these architectures." It's been longer than that for router vendors. The move to hardware based forwarding was in the 90s.
- jes 7y ago
  I was intrigued by your comment about the Tomahawk3. Turns out there is a Tomahawk4 implemented in 7nm which has twice the performance. Thank you for tipping me off to these devices: https://www.broadcom.com/company/news/product-releases/52756 https://www.broadcom.com/company/news/product-releases/52756
  - ra1n85 7y ago
    Yep, though it’s not commercially available quite yet. Likely only shipping samples to switch manufacturers at this time.
- rstuart4133 7y ago
  > I don't know how things will shake out for the Linux world and x86 packet forwarding given the trend and lack of real performance in the kernel. That's pretty much settled isn't it? It was settled in the same way the "it got too much for the general purpose CPU" problem always gets settled - you do it in custom silicon, and define a standardised interface. That's pretty much what happened with 3D graphics where the standardised interface was OpenGL (but now seems to be up in the air). It's also pretty much what's happened in AI with standardised libraries interfacing to custom hardware, and it's what happened with networking. For networking, the interface standard is OpenFlow. So, if you think it's possible you will need to handle links of about 1Gbs or over in the future, you do your networking using an OpenFlow implementation like Faucet. If it's not much above 100Mb/s the Linux kernel module that implements OpenFlow, called openvswitch, will be fine. Otherwise use some custom hardware. Openvswitch has been around since 2009 - so it's not exactly a new thing.
- gnufx 7y ago
  Are those kernel bypasses better than the Linux-based RDMA we've had for ages (on appropriate fabrics)?
  - wmf 7y ago
    One problem with RDMA is that it's not compatible with existing protocols.
    - gnufx 6y ago
      It's existed for a fair while as Infiniband doing Linux by-pass; the comment didn't say TCP/IP. (The general technique pre-dates Linux, of course.)
dahfizz 7y ago
This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU just to keep things usable, but it still feels like a big step backwards. A typical k8s deployment runs in containers that are in VMs. So each packet you want to send to or from a container needs to touch a cpu and traverse a networking stack six times. That's dumb.
- heinrichhartman 7y ago
  Interesting. I always suspected something like this was going on. Virtual networking / "SDN" just sounded slow to me. Do you know a good source, to learn about the lower-level Linux/container/docker/k8s networking?
  - zamadatix 7y ago
    I got the most value poking around with network namespaces manually via the "ip" command. There are also true VRFs in Linux these days which are much lighter but not nearly as common, most everything is a namespace with either bridging or NAT and some fancy configuration interface to drive that state.
  - cheeseprocedure 7y ago
    packagecloud's in-depth writeups on the Linux networking stack are a great read: https://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/ https://blog.packagecloud.io/eng/2016/06/22/monitoring-tunin... https://blog.packagecloud.io/eng/2017/02/06/monitoring-tuning-linux-networking-stack-sending-data/ https://blog.packagecloud.io/eng/2017/02/06/monitoring-tunin... The illustrated guide to receiving data is also solid: https://blog.packagecloud.io/eng/2016/10/11/monitoring-tuning-linux-networking-stack-receiving-data-illustrated/ https://blog.packagecloud.io/eng/2016/10/11/monitoring-tunin...
- dan_quixote 7y ago
z3t4 7y ago
What happens when you "netperf TCP_STREAM from 2 sources"?
marsdepinski 7y ago
Good short article. CPU power saving has an effect on interrupt handling and becomes a factor at high packet rates. Turning off p-states will improve performance.

192 points by sohkamyung 7y ago

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architectures. Almost no packets are forwarded by the same processors running management and control plane functions anymore. This is mainly because the required traffic rates today need dedicated silicon purpose built for the task (the Broadcom Tomahawk3 can do 12.8 Terabits/sec above a relatively small packet size). I don't know how things will shake out for the Linux world and x86 packet forwarding given the trend and lack of real performance in the kernel. Right now, your best bet when it comes to Linux and high network throughput/packet processing requirements is to just bypass the kernel entirely with DPDK, a "smart" NIC, or XDP.

diegocg 7y ago
These is the most up-to-date article I've seen on these matters https://lwn.net/Articles/787754/ https://lwn.net/Articles/787754/ https://lwn.net/Articles/629155/ https://lwn.net/Articles/629155/
EvanAnderson 7y ago
I wonder if there's an economic argument, at useful scales, to using FPGA's in general purpose servers to accelerate network performance. The purpose-built ASICs would win on cost-per-unit every time, but the FPGA would have some adaptability to new protocols or algorithms that the ASIC wouldn't.
- wmf 7y ago
  Yes, Azure uses FPGAs for networking. Programmable NPUs like Netronome or Pensando are a middle ground that are more cost-efficient than FPGAs for most needs.
- yummypaint 7y ago
  Ive been waiting for mainstream workstation motherboards with this capability for years. Presumably a pci card is how this would be handled in practice for now. My naive take is that toolchains are still too convoluted and bogged down with licensing schemes to deliver the kind of real-time highly integrated adaptability this would require for individual users. Would be great if the barrier to entry has in fact dropped enough that university-sized networks could consider them.
- karatekidd32v 7y ago
  There's some cool tech going on at barefoot (https://barefootnetworks.com/ https://barefootnetworks.com/) I don't know much about the technical details, but the pitch I've heard is that it gives you ASIC level performance with more flexibility to reprogram the chip (not full FPGA).
  - ra1n85 7y ago
    Yeah, it's an interesting approach. They're basically allowing you to define packet processing with P4 on their Tofino family of chipsets: https://p4.org/ https://p4.org/ That said, there's only so much you can do in a chip before considerable tradeoffs are going to be made. They're not going to offer the same level of flexibility you get out of a general purpose CPU, but may not have same the restrictions of most fixed pipeline chips - their product sits somewhere in the middle. Also, P4 seems to sit in a space complex enough to make it unreasonable for most network shops - it's not for your average enterprise or service provider network.
tw04 7y ago
That's not entirely accurate - Cisco open sourced VPP which they use in many products: https://blogs.cisco.com/sp/a-bigger-helping-of-internet-please https://blogs.cisco.com/sp/a-bigger-helping-of-internet-plea... https://docs.fd.io/vpp/16.09/ https://docs.fd.io/vpp/16.09/ While they still obviously make custom ASICs, they've moved towards more software where possible.
- ra1n85 7y ago
  Sorry, what's not accurate here?
- Hikikomori 7y ago
  Pretty much everything above their small medium business ISR routers use ASIC/NPU for all traffic possible. Which routers are you referring to?
bogomipz 7y ago
>"In the last 15 years there has been a hard move away from these architectures." It's been longer than that for router vendors. The move to hardware based forwarding was in the 90s.
jes 7y ago
I was intrigued by your comment about the Tomahawk3. Turns out there is a Tomahawk4 implemented in 7nm which has twice the performance. Thank you for tipping me off to these devices: https://www.broadcom.com/company/news/product-releases/52756 https://www.broadcom.com/company/news/product-releases/52756
- ra1n85 7y ago
  Yep, though it’s not commercially available quite yet. Likely only shipping samples to switch manufacturers at this time.
rstuart4133 7y ago
> I don't know how things will shake out for the Linux world and x86 packet forwarding given the trend and lack of real performance in the kernel. That's pretty much settled isn't it? It was settled in the same way the "it got too much for the general purpose CPU" problem always gets settled - you do it in custom silicon, and define a standardised interface. That's pretty much what happened with 3D graphics where the standardised interface was OpenGL (but now seems to be up in the air). It's also pretty much what's happened in AI with standardised libraries interfacing to custom hardware, and it's what happened with networking. For networking, the interface standard is OpenFlow. So, if you think it's possible you will need to handle links of about 1Gbs or over in the future, you do your networking using an OpenFlow implementation like Faucet. If it's not much above 100Mb/s the Linux kernel module that implements OpenFlow, called openvswitch, will be fine. Otherwise use some custom hardware. Openvswitch has been around since 2009 - so it's not exactly a new thing.
gnufx 7y ago
Are those kernel bypasses better than the Linux-based RDMA we've had for ages (on appropriate fabrics)?
- wmf 7y ago
  One problem with RDMA is that it's not compatible with existing protocols.
  - gnufx 6y ago
    It's existed for a fair while as Infiniband doing Linux by-pass; the comment didn't say TCP/IP. (The general technique pre-dates Linux, of course.)

dahfizz 7y ago

This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU just to keep things usable, but it still feels like a big step backwards. A typical k8s deployment runs in containers that are in VMs. So each packet you want to send to or from a container needs to touch a cpu and traverse a networking stack six times. That's dumb.

heinrichhartman 7y ago
Interesting. I always suspected something like this was going on. Virtual networking / "SDN" just sounded slow to me. Do you know a good source, to learn about the lower-level Linux/container/docker/k8s networking?
- zamadatix 7y ago
  I got the most value poking around with network namespaces manually via the "ip" command. There are also true VRFs in Linux these days which are much lighter but not nearly as common, most everything is a namespace with either bridging or NAT and some fancy configuration interface to drive that state.
- cheeseprocedure 7y ago
  packagecloud's in-depth writeups on the Linux networking stack are a great read: https://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/ https://blog.packagecloud.io/eng/2016/06/22/monitoring-tunin... https://blog.packagecloud.io/eng/2017/02/06/monitoring-tuning-linux-networking-stack-sending-data/ https://blog.packagecloud.io/eng/2017/02/06/monitoring-tunin... The illustrated guide to receiving data is also solid: https://blog.packagecloud.io/eng/2016/10/11/monitoring-tuning-linux-networking-stack-receiving-data-illustrated/ https://blog.packagecloud.io/eng/2016/10/11/monitoring-tunin...
dan_quixote 7y ago

z3t4 7y ago

What happens when you "netperf TCP_STREAM from 2 sources"?

marsdepinski 7y ago

Good short article. CPU power saving has an effect on interrupt handling and becomes a factor at high packet rates. Turning off p-states will improve performance.