4 ms·

You briefly mentioned Receive Side Steering (RSS) with multiqueue network cards, but didn't mention Receive Flow Steering. From the kernel docs: https://www.ke

by SEJeff 7y ago

You briefly mentioned Receive Side Steering (RSS) with multiqueue network cards, but didn't mention Receive Flow Steering. From the kernel docs: https://www.kernel.org/doc/Documentation/networking/scaling.txt https://www.kernel.org/doc/Documentation/networking/scaling.... """ While RPS steers packets solely based on hash, and thus generally provides good load distribution, it does not take into account application locality. This is accomplished by Receive Flow Steering (RFS). The goal of RFS is to increase datacache hitrate by steering kernel processing of packets to the CPU where the application thread consuming the packet is running. RFS relies on the same RPS mechanisms to enqueue packets onto the backlog of another CPU and to wake up that CPU. """ I have zero problem doing 25G and 40G ethernet at the Linux kernel in RHEL7 (I run Kubernetes clusters on them in fact). Non-ethernet (Infiniband) 100G+ line rate is also totally doable, but IB is an entirely different beast. I agree with you it isn't going to work in the long term, but it should be fine in the medium term. The long term Linux change is likely to just rewrite more and more of the existing filtering code ontop of the eBPF vm, which is fast as holy hell and is replacing large swaths of existing filtering mechanisms for a reason.

ra1n85 7y ago
What if you have no application and you're purely forwarding traffic? That's what I'm measuring here (and that's actually typically faster than passing packets up to user space apps to consume through syscalls). 25G line rate is 37MPPS - are you saying you have zero problem forwarding at that line rate? I'd be very surprised if that's being done in the kernel. I'd be more surprised if you said that you're consuming bytes from the network at that speed with user space apps and no kernel bypass. XDP (the forwarding approach built on top of eBPF) is limited to ~20MPPS, as well: https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise/ https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-wha...
- wmf 7y ago
  For routing, most people prefer to use... routers. Linux isn't great for forwarding and IMO that's fine. As the article notes, there's always VPP when you need to do software routing for some reason.
  - StillBored 7y ago
    Yes, and the current user space approaches are spinloop based so the machine isn't really general purpose at that point. You might as well just call it a router if all your cores are spinning on RSS/etc queues.

by SEJeff 7y ago

ra1n85 7y ago

What if you have no application and you're purely forwarding traffic? That's what I'm measuring here (and that's actually typically faster than passing packets up to user space apps to consume through syscalls). 25G line rate is 37MPPS - are you saying you have zero problem forwarding at that line rate? I'd be very surprised if that's being done in the kernel. I'd be more surprised if you said that you're consuming bytes from the network at that speed with user space apps and no kernel bypass. XDP (the forwarding approach built on top of eBPF) is limited to ~20MPPS, as well: https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise/ https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-wha...

wmf 7y ago
For routing, most people prefer to use... routers. Linux isn't great for forwarding and IMO that's fine. As the article notes, there's always VPP when you need to do software routing for some reason.
- StillBored 7y ago
  Yes, and the current user space approaches are spinloop based so the machine isn't really general purpose at that point. You might as well just call it a router if all your cores are spinning on RSS/etc queues.