7 ms·

We are not struggling to do line rate 10G in the kernel. Modern 100Gbe nics (mellanox, solarflare) will happily do line rate with stock upstream kernel for a wh

We are not struggling to do line rate 10G in the kernel. Modern 100Gbe nics (mellanox, solarflare) will happily do line rate with stock upstream kernel for a while now (definitely since 4.x) you only need to tune your irq balancing, and you can probably get away with not even doing that. If you are buying 100gbe nics you are also buying server class (xeon, rome) processors that can keep up. Source: I operate a CDN with thousands of 100Gbe nics with a stock upstream LTS kernel, and minimal kernel tuning.

ra1n85 7y ago
You're saying you can forward 100Gbps at line rate (148MPPS) through a stock kernel?
- danceparty 7y ago
  You can get within a few percentage points, yes I just tested this with two hosts with 4.14.127 upstream kernel and upstream mlx5 driver, and mellanox connectx-5 card. Using 16 iperf threads [SUM] 0.0-10.0 sec 85.1 Gbits/sec That's pretty close with no tuning, and well beyond 10gb/s we mentioned earlier
  - ra1n85 7y ago
    16 iperf threads...sending at what packet size? Do you understand the notion of line rate? 85Gbps at 1500B is only 7MPPS, which is half of 10Gbps at line rate.
    - toast0 7y ago
      Does line rate imply smallest packets? For CDN style use cases, you want to use the whole pipe, and it's going to be mostly larger packets.
      - ra1n85 7y ago
        Yes. The point of discussion here is that the Linux kernel struggles to do line rate 10Gbps. This was misinterpreted as "the Linux kernel struggles to do 10Gbps".
    - Dylan16807 7y ago
      Where are you getting your definitions? I have never seen "line rate" used to refer to packets per second.
      - ra1n85 7y ago
        It implies it. Ethernet at 84B per frame is the smallest you can go, thus the line rate - some examples: https://events19.linuxfoundation.org/wp-content/uploads/2017/12/jim-Thompson.pdf https://events19.linuxfoundation.org/wp-content/uploads/2017... https://www.redhat.com/en/blog/pushing-limits-kernel-networking https://www.redhat.com/en/blog/pushing-limits-kernel-network... https://kernel-recipes.org/en/2014/ndiv-a-low-overhead-network-traffic-diverter/ https://kernel-recipes.org/en/2014/ndiv-a-low-overhead-netwo...
        Dylan16807 7y ago
        "How do you fill a 100GBps pipe with small packets?" "achieve 10 Gbps line rate at 60B frames" "reaching line rate on all packet sizes" Line rate is just bits per second. You have to add in a qualifier about packet size before you're talking about packets per second.
        ra1n85 7y ago
        Nope, I'm sorry you're not quite getting it here. Minimum Ethernet frame is 84B on the wire - it's simple enough from there.
        big_chungus 7y ago
        I've never heard this weird qualification for the definition of "line rate" that it somehow requires minimum packet size, so I looked it up. The first three sources for a quoted big-g search all imply or directly state that it's the same as bandwidth: https://blog.ipspace.net/2009/03/line-rate-and-bit-rate.html https://blog.ipspace.net/2009/03/line-rate-and-bit-rate.html https://www.reddit.com/r/networking/comments/4tk2to/bandwidth_vs_line_rate_vs_throughput_vs/ https://www.reddit.com/r/networking/comments/4tk2to/bandwidt... https://www.fmad.io/blog-what-is-10g-line-rate.html https://www.fmad.io/blog-what-is-10g-line-rate.html Also, for gigabit networks, ethernet packets are padded to at least 512 bytes because of a bigger slot size: https://www.cse.wustl.edu/~jain/cis788-97/ftp/gigabit_ethernet/index.html https://www.cse.wustl.edu/~jain/cis788-97/ftp/gigabit_ethern...
  - GhettoMaestro 7y ago
    Wrong. You’re like an order of magnitude wrong - rofl there is no fucking way the stock Linux kernel will even do 40Mpps at 64 byte packets. It chokes way before that. This is partly why things like DPDK exist.
  - gonzo 7y ago
    You’re sending 1500 byte MTU (1538 bytes on the wire) or maybe larger (9000 byte MTU) packets. 1538 bytes is 12,304 bits. 10,000,000,000 bits/sec / 12,304 bits/packet is 812,744 packets per second. Now try it with 64 byte packets, which are 84 bytes on the wire. 14,880,952 packets per second. And this is 10gbps.

ra1n85 7y ago

You're saying you can forward 100Gbps at line rate (148MPPS) through a stock kernel?

danceparty 7y ago
You can get within a few percentage points, yes I just tested this with two hosts with 4.14.127 upstream kernel and upstream mlx5 driver, and mellanox connectx-5 card. Using 16 iperf threads [SUM] 0.0-10.0 sec 85.1 Gbits/sec That's pretty close with no tuning, and well beyond 10gb/s we mentioned earlier
- ra1n85 7y ago
  16 iperf threads...sending at what packet size? Do you understand the notion of line rate? 85Gbps at 1500B is only 7MPPS, which is half of 10Gbps at line rate.
  - toast0 7y ago
    Does line rate imply smallest packets? For CDN style use cases, you want to use the whole pipe, and it's going to be mostly larger packets.
    - ra1n85 7y ago
      Yes. The point of discussion here is that the Linux kernel struggles to do line rate 10Gbps. This was misinterpreted as "the Linux kernel struggles to do 10Gbps".
  - Dylan16807 7y ago
    Where are you getting your definitions? I have never seen "line rate" used to refer to packets per second.
    - ra1n85 7y ago
      It implies it. Ethernet at 84B per frame is the smallest you can go, thus the line rate - some examples: https://events19.linuxfoundation.org/wp-content/uploads/2017/12/jim-Thompson.pdf https://events19.linuxfoundation.org/wp-content/uploads/2017... https://www.redhat.com/en/blog/pushing-limits-kernel-networking https://www.redhat.com/en/blog/pushing-limits-kernel-network... https://kernel-recipes.org/en/2014/ndiv-a-low-overhead-network-traffic-diverter/ https://kernel-recipes.org/en/2014/ndiv-a-low-overhead-netwo...
      - Dylan16807 7y ago
        "How do you fill a 100GBps pipe with small packets?" "achieve 10 Gbps line rate at 60B frames" "reaching line rate on all packet sizes" Line rate is just bits per second. You have to add in a qualifier about packet size before you're talking about packets per second.
        ra1n85 7y ago
        Nope, I'm sorry you're not quite getting it here. Minimum Ethernet frame is 84B on the wire - it's simple enough from there.
        big_chungus 7y ago
        I've never heard this weird qualification for the definition of "line rate" that it somehow requires minimum packet size, so I looked it up. The first three sources for a quoted big-g search all imply or directly state that it's the same as bandwidth: https://blog.ipspace.net/2009/03/line-rate-and-bit-rate.html https://blog.ipspace.net/2009/03/line-rate-and-bit-rate.html https://www.reddit.com/r/networking/comments/4tk2to/bandwidth_vs_line_rate_vs_throughput_vs/ https://www.reddit.com/r/networking/comments/4tk2to/bandwidt... https://www.fmad.io/blog-what-is-10g-line-rate.html https://www.fmad.io/blog-what-is-10g-line-rate.html Also, for gigabit networks, ethernet packets are padded to at least 512 bytes because of a bigger slot size: https://www.cse.wustl.edu/~jain/cis788-97/ftp/gigabit_ethernet/index.html https://www.cse.wustl.edu/~jain/cis788-97/ftp/gigabit_ethern...
- GhettoMaestro 7y ago
  Wrong. You’re like an order of magnitude wrong - rofl there is no fucking way the stock Linux kernel will even do 40Mpps at 64 byte packets. It chokes way before that. This is partly why things like DPDK exist.
- gonzo 7y ago
  You’re sending 1500 byte MTU (1538 bytes on the wire) or maybe larger (9000 byte MTU) packets. 1538 bytes is 12,304 bits. 10,000,000,000 bits/sec / 12,304 bits/packet is 812,744 packets per second. Now try it with 64 byte packets, which are 84 bytes on the wire. 14,880,952 packets per second. And this is 10gbps.