5 ms·

The current approach is fundamentally not going to work in the long term. 100Gbps at line rate means single digits[1] of nanoseconds between frames. At that f

by ra1n85 7y ago

The current approach is fundamentally not going to work in the long term. 100Gbps at line rate means single digits[1] of nanoseconds between frames. At that frequency, a cache miss is pretty bad. This is all not to mention locks, or that there are competing functions running in most distros (turn off irqbalance completely and watch your forwarding rate increase). The low hanging fruit seems to have been picked as well - NAPI polling, interrupt coalescing, RSS + multique NICs + SMP, etc, are already out there, and we're still struggling to do 10G line rate in the Kernel...and data centers are moving quickly to 25/100G. [1] Edited for terrible math - 10Gbps at line rate is 67ns per packet, 100Gbps is 6.7ns

SEJeff 7y ago
You briefly mentioned Receive Side Steering (RSS) with multiqueue network cards, but didn't mention Receive Flow Steering. From the kernel docs: https://www.kernel.org/doc/Documentation/networking/scaling.txt https://www.kernel.org/doc/Documentation/networking/scaling.... """ While RPS steers packets solely based on hash, and thus generally provides good load distribution, it does not take into account application locality. This is accomplished by Receive Flow Steering (RFS). The goal of RFS is to increase datacache hitrate by steering kernel processing of packets to the CPU where the application thread consuming the packet is running. RFS relies on the same RPS mechanisms to enqueue packets onto the backlog of another CPU and to wake up that CPU. """ I have zero problem doing 25G and 40G ethernet at the Linux kernel in RHEL7 (I run Kubernetes clusters on them in fact). Non-ethernet (Infiniband) 100G+ line rate is also totally doable, but IB is an entirely different beast. I agree with you it isn't going to work in the long term, but it should be fine in the medium term. The long term Linux change is likely to just rewrite more and more of the existing filtering code ontop of the eBPF vm, which is fast as holy hell and is replacing large swaths of existing filtering mechanisms for a reason.
- ra1n85 7y ago
  What if you have no application and you're purely forwarding traffic? That's what I'm measuring here (and that's actually typically faster than passing packets up to user space apps to consume through syscalls). 25G line rate is 37MPPS - are you saying you have zero problem forwarding at that line rate? I'd be very surprised if that's being done in the kernel. I'd be more surprised if you said that you're consuming bytes from the network at that speed with user space apps and no kernel bypass. XDP (the forwarding approach built on top of eBPF) is limited to ~20MPPS, as well: https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise/ https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-wha...
  - wmf 7y ago
    For routing, most people prefer to use... routers. Linux isn't great for forwarding and IMO that's fine. As the article notes, there's always VPP when you need to do software routing for some reason.
    - StillBored 7y ago
      Yes, and the current user space approaches are spinloop based so the machine isn't really general purpose at that point. You might as well just call it a router if all your cores are spinning on RSS/etc queues.
danceparty 7y ago
We are not struggling to do line rate 10G in the kernel. Modern 100Gbe nics (mellanox, solarflare) will happily do line rate with stock upstream kernel for a while now (definitely since 4.x) you only need to tune your irq balancing, and you can probably get away with not even doing that. If you are buying 100gbe nics you are also buying server class (xeon, rome) processors that can keep up. Source: I operate a CDN with thousands of 100Gbe nics with a stock upstream LTS kernel, and minimal kernel tuning.
- ra1n85 7y ago
  You're saying you can forward 100Gbps at line rate (148MPPS) through a stock kernel?
  - danceparty 7y ago
    You can get within a few percentage points, yes I just tested this with two hosts with 4.14.127 upstream kernel and upstream mlx5 driver, and mellanox connectx-5 card. Using 16 iperf threads [SUM] 0.0-10.0 sec 85.1 Gbits/sec That's pretty close with no tuning, and well beyond 10gb/s we mentioned earlier
    - ra1n85 7y ago
      16 iperf threads...sending at what packet size? Do you understand the notion of line rate? 85Gbps at 1500B is only 7MPPS, which is half of 10Gbps at line rate.
      - toast0 7y ago
        Does line rate imply smallest packets? For CDN style use cases, you want to use the whole pipe, and it's going to be mostly larger packets.
        ra1n85 7y ago
        Yes. The point of discussion here is that the Linux kernel struggles to do line rate 10Gbps. This was misinterpreted as "the Linux kernel struggles to do 10Gbps".
joemag 7y ago
With one thread, maybe. But using multiple threads it’s not even that hard. I’ve hit 100Gbps using stock TCP stack and ~10 threads in Rusts Hyper without much trouble. Another example, you can saturate 100Gbps with just 4 iPerf3 processes.

by ra1n85 7y ago

SEJeff 7y ago

You briefly mentioned Receive Side Steering (RSS) with multiqueue network cards, but didn't mention Receive Flow Steering. From the kernel docs: https://www.kernel.org/doc/Documentation/networking/scaling.txt https://www.kernel.org/doc/Documentation/networking/scaling.... """ While RPS steers packets solely based on hash, and thus generally provides good load distribution, it does not take into account application locality. This is accomplished by Receive Flow Steering (RFS). The goal of RFS is to increase datacache hitrate by steering kernel processing of packets to the CPU where the application thread consuming the packet is running. RFS relies on the same RPS mechanisms to enqueue packets onto the backlog of another CPU and to wake up that CPU. """ I have zero problem doing 25G and 40G ethernet at the Linux kernel in RHEL7 (I run Kubernetes clusters on them in fact). Non-ethernet (Infiniband) 100G+ line rate is also totally doable, but IB is an entirely different beast. I agree with you it isn't going to work in the long term, but it should be fine in the medium term. The long term Linux change is likely to just rewrite more and more of the existing filtering code ontop of the eBPF vm, which is fast as holy hell and is replacing large swaths of existing filtering mechanisms for a reason.

ra1n85 7y ago
What if you have no application and you're purely forwarding traffic? That's what I'm measuring here (and that's actually typically faster than passing packets up to user space apps to consume through syscalls). 25G line rate is 37MPPS - are you saying you have zero problem forwarding at that line rate? I'd be very surprised if that's being done in the kernel. I'd be more surprised if you said that you're consuming bytes from the network at that speed with user space apps and no kernel bypass. XDP (the forwarding approach built on top of eBPF) is limited to ~20MPPS, as well: https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise/ https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-wha...
- wmf 7y ago
  For routing, most people prefer to use... routers. Linux isn't great for forwarding and IMO that's fine. As the article notes, there's always VPP when you need to do software routing for some reason.
  - StillBored 7y ago
    Yes, and the current user space approaches are spinloop based so the machine isn't really general purpose at that point. You might as well just call it a router if all your cores are spinning on RSS/etc queues.

danceparty 7y ago

We are not struggling to do line rate 10G in the kernel. Modern 100Gbe nics (mellanox, solarflare) will happily do line rate with stock upstream kernel for a while now (definitely since 4.x) you only need to tune your irq balancing, and you can probably get away with not even doing that. If you are buying 100gbe nics you are also buying server class (xeon, rome) processors that can keep up. Source: I operate a CDN with thousands of 100Gbe nics with a stock upstream LTS kernel, and minimal kernel tuning.

ra1n85 7y ago
You're saying you can forward 100Gbps at line rate (148MPPS) through a stock kernel?
- danceparty 7y ago
  You can get within a few percentage points, yes I just tested this with two hosts with 4.14.127 upstream kernel and upstream mlx5 driver, and mellanox connectx-5 card. Using 16 iperf threads [SUM] 0.0-10.0 sec 85.1 Gbits/sec That's pretty close with no tuning, and well beyond 10gb/s we mentioned earlier
  - ra1n85 7y ago
    16 iperf threads...sending at what packet size? Do you understand the notion of line rate? 85Gbps at 1500B is only 7MPPS, which is half of 10Gbps at line rate.
    - toast0 7y ago
      Does line rate imply smallest packets? For CDN style use cases, you want to use the whole pipe, and it's going to be mostly larger packets.
      - ra1n85 7y ago
        Yes. The point of discussion here is that the Linux kernel struggles to do line rate 10Gbps. This was misinterpreted as "the Linux kernel struggles to do 10Gbps".

joemag 7y ago

With one thread, maybe. But using multiple threads it’s not even that hard. I’ve hit 100Gbps using stock TCP stack and ~10 threads in Rusts Hyper without much trouble. Another example, you can saturate 100Gbps with just 4 iPerf3 processes.