9 ms·

How to receive a million packets per second (2015)

96 points by warpech 6y ago

tayo42 6y ago
> Don't expect performance like that for any practical application without a lot more work. 5 years later I have servers that can do 2 million (multi process) without messing with Numa or nic settings.
- osrec 6y ago
  Out of interest, what spec are your servers?
  - tayo42 6y ago
    I dont remember to much detail off the top of my head but the cpu was a newish 128 core amd and a mellanox nic, i had to get my work stuff out to get more detail
    - osrec 6y ago
      Ah, so it's an absolute beast of a machine! I thought your response might have been pointing to additional efficiency in new hardware, but it's more that you have more powerful hardware.
      - Sebb767 6y ago
        Also, given that this was on 2014-ish hardware, 2Ghz/8Core sounds a lot like a Xeon L. A high-powered Xeon or something like an i7 could've probably already done a lot better simply because of raw single core performance. So these numbers definitely sound reasonable.
- dpe82 6y ago
  At least for TX it's much easier to get that performance today due to relatively recent kernel changes. Unsure about RX though. A few years ago I worked on the upload service within the YouTube Android app. Shortly after YouTube started using QUIC for video streaming I decided it would be interesting to experiment using QUIC for mobile uploads; my thesis being that its BBR implementation might lead to better throughput on lossy wireless connections and therefore faster mobile uploads. I discovered though when tethered to my dev workstation I wasn't able to break ~100mbit while the native TCP stack was able to saturate the USB-C gigabit ethernet adapter I was using. This problem got the attention of the QUIC folks, which lead to a more detailed investigation than I was capable of and eventually to kernel patches described here: http://vger.kernel.org/lpc_net2018_talks/willemdebruijn-lpc2018-udpgso-paper-DRAFT-1.pdf http://vger.kernel.org/lpc_net2018_talks/willemdebruijn-lpc2... I'd like to think my little project was the major motivator for that work but I'm pretty sure the server efficiency gains described in that paper were a much stronger motivation. :)
  - londons_explore 6y ago
    That paper is a real disappointment with the places it doesnt go. Specifically, if QUIC is to have high performance and high power efficiency, just batching the packets isn't enough. Think of a usecase - a CDN wants to be able to send a block of data from disk to a client. It should be able to direct that data from the SSD to the network card without the CPU touching the data at all. On the receive side, a mobile device should be able to download a big file to ram or flash with the main CPU asleep most of the time. That means the network device should be encrypting the packets, doing the pacing, doing the retries, etc. It may be that a quic specific set of options or syscalls is best for that, or it might be that the application should provide a "this is how to send my data" set of bytecode which the network hardware executes repeatedly. But just batching UDP packets is a half way measure which isn't in the direction of the final goal.
    - Klinky 6y ago
      Wouldn't this mean the networking interface is now Layer 7? It has to understand how QUIC works. Sounds like Smart NIC, though that seems mainly enterprise focused.
    - dpe82 6y ago
      One step at a time. Right, that paper doesn't introduce life-shattering innovation but rather just fixes a long-standing issue with UDP performance in the Linux kernel. That ain't nothing. A large motivator to develop and deploy QUIC is that by implementing much of the stack as a userspace library the protocol can be iterated on at a pace impossible with TCP. BBR congestion control experiments are a great example of that; there's enormous value in quick iteration, even at the cost of compute efficiency. I agree with your sentiment though; I have to imagine eventually the protocol will settle down and hardware acceleration will become a thing - but imho not to the degree you propose.
mehrdadn 6y ago
Curious, does anyone know how things look on Windows?
- abafazi 6y ago
  Not good ~200K pp/s max
ncmncm 6y ago
A thousand nanoseconds per packet? Luxury! At 10Gbps, and hundred-odd byte packets as are common on finance multicast feeds, I get 73ns per packet. The amount that can be done in that time is limited. Typically, you just dump them in a big-ass ring buffer and let processes on a bunch of other cores pick out interesting ones to spend more time on. The way to get the packets is usually with some proprietary kernel-bypass library (some have success with DPDK), and a very carefully isolated core that the kernel is kept from interrupting, and just spins moving packets. NIC hardware-level ring buffers are usually pretty small, so pauses make for drops. Many NICs can be persuaded to filter packets out to multiple ring buffers, and then you can get multiple cores picking them off; that is necessary when you get to 40 and 100 Gbps. Multiple rings even with one core multiplies headroom, making occasional housekeeping without drops possible.
- londons_explore 6y ago
  Sounds to me like you should tell the hardware manufacturers to make bigger ring buffers, or even to provide an indexed ring buffer which indexes by a user-defined part of the packet...
  - ncmncm 6y ago
    I have found limited success dictating to hardware manufacturers. I would be satisfied if the kernel could just be persuaded to leave my cores the hell alone without an endless list of boot flags.
    - LunaSea 6y ago
      Do you have links about the flags that you mention or were you referring to process pining? Have you tried out the new eBPF libraries and kTLS?
      - ncmncm 6y ago
        Search isolcpus, nohz_full, and rcu_nocbs. Process pinning & isolation from other processes is the first step; keeping kernel interruptions out is the hard part. Kernel people generally think anything they do is implicitly more important than every single thing you do. eBPF and kTLS tend to involve the kernel in your business. But, eBPF can put code directly on certain NICs (e.g. Netronome), writing directly to user process memory. Then there is no need to pin or isolate, because the action is on the NIC, out of reach.
frumiousirc 6y ago
1 Gbps, unoptimized, commodity home network between an ancient thinkpad t520 and gen 4 i7 workstation using ZeroMQ throughput perf test gives more than 1M messages per second: $ ./local_thr tcp://192.168.1.123:5678 100 1000000 message size: 100 [B] message count: 1000000 mean throughput: 1143708 [msg/s] mean throughput: 914.966 [Mb/s]
- 7532yahoogmail 6y ago
  The author specifically pointed out metric was packets per second not bytes per second because getting rather higher numbers with other units was easy.
- davidcuddeback 6y ago
  Assuming ZeroMQ batches messages like other message queue technologies, you're comparing apples to oranges. The article is about receiving 1M network packets per second.
7532yahoogmail 6y ago
OP has Solarflare NICs. So surely > 1MM pps is possible on that HW? But as you opened with 1 million per second on the Linux stack, I gather that kernel by-pass and such were by definition out of scope. The goal here was to always pass through the kernel. But if so, why use Solarflare? Why not see if one can do + 1MM pps on a nice, even beefy but not Solarflare/Mellonx NICs whose motivation includes kernel by-pass.

96 points by warpech 6y ago

> Don't expect performance like that for any practical application without a lot more work. 5 years later I have servers that can do 2 million (multi process) without messing with Numa or nic settings.

osrec 6y ago
Out of interest, what spec are your servers?
- tayo42 6y ago
  I dont remember to much detail off the top of my head but the cpu was a newish 128 core amd and a mellanox nic, i had to get my work stuff out to get more detail
  - osrec 6y ago
    Ah, so it's an absolute beast of a machine! I thought your response might have been pointing to additional efficiency in new hardware, but it's more that you have more powerful hardware.
    - Sebb767 6y ago
      Also, given that this was on 2014-ish hardware, 2Ghz/8Core sounds a lot like a Xeon L. A high-powered Xeon or something like an i7 could've probably already done a lot better simply because of raw single core performance. So these numbers definitely sound reasonable.
dpe82 6y ago
At least for TX it's much easier to get that performance today due to relatively recent kernel changes. Unsure about RX though. A few years ago I worked on the upload service within the YouTube Android app. Shortly after YouTube started using QUIC for video streaming I decided it would be interesting to experiment using QUIC for mobile uploads; my thesis being that its BBR implementation might lead to better throughput on lossy wireless connections and therefore faster mobile uploads. I discovered though when tethered to my dev workstation I wasn't able to break ~100mbit while the native TCP stack was able to saturate the USB-C gigabit ethernet adapter I was using. This problem got the attention of the QUIC folks, which lead to a more detailed investigation than I was capable of and eventually to kernel patches described here: http://vger.kernel.org/lpc_net2018_talks/willemdebruijn-lpc2018-udpgso-paper-DRAFT-1.pdf http://vger.kernel.org/lpc_net2018_talks/willemdebruijn-lpc2... I'd like to think my little project was the major motivator for that work but I'm pretty sure the server efficiency gains described in that paper were a much stronger motivation. :)
- londons_explore 6y ago
  That paper is a real disappointment with the places it doesnt go. Specifically, if QUIC is to have high performance and high power efficiency, just batching the packets isn't enough. Think of a usecase - a CDN wants to be able to send a block of data from disk to a client. It should be able to direct that data from the SSD to the network card without the CPU touching the data at all. On the receive side, a mobile device should be able to download a big file to ram or flash with the main CPU asleep most of the time. That means the network device should be encrypting the packets, doing the pacing, doing the retries, etc. It may be that a quic specific set of options or syscalls is best for that, or it might be that the application should provide a "this is how to send my data" set of bytecode which the network hardware executes repeatedly. But just batching UDP packets is a half way measure which isn't in the direction of the final goal.
  - Klinky 6y ago
    Wouldn't this mean the networking interface is now Layer 7? It has to understand how QUIC works. Sounds like Smart NIC, though that seems mainly enterprise focused.
  - dpe82 6y ago
    One step at a time. Right, that paper doesn't introduce life-shattering innovation but rather just fixes a long-standing issue with UDP performance in the Linux kernel. That ain't nothing. A large motivator to develop and deploy QUIC is that by implementing much of the stack as a userspace library the protocol can be iterated on at a pace impossible with TCP. BBR congestion control experiments are a great example of that; there's enormous value in quick iteration, even at the cost of compute efficiency. I agree with your sentiment though; I have to imagine eventually the protocol will settle down and hardware acceleration will become a thing - but imho not to the degree you propose.

mehrdadn 6y ago

Curious, does anyone know how things look on Windows?

abafazi 6y ago
Not good ~200K pp/s max

ncmncm 6y ago

A thousand nanoseconds per packet? Luxury! At 10Gbps, and hundred-odd byte packets as are common on finance multicast feeds, I get 73ns per packet. The amount that can be done in that time is limited. Typically, you just dump them in a big-ass ring buffer and let processes on a bunch of other cores pick out interesting ones to spend more time on. The way to get the packets is usually with some proprietary kernel-bypass library (some have success with DPDK), and a very carefully isolated core that the kernel is kept from interrupting, and just spins moving packets. NIC hardware-level ring buffers are usually pretty small, so pauses make for drops. Many NICs can be persuaded to filter packets out to multiple ring buffers, and then you can get multiple cores picking them off; that is necessary when you get to 40 and 100 Gbps. Multiple rings even with one core multiplies headroom, making occasional housekeeping without drops possible.

londons_explore 6y ago
Sounds to me like you should tell the hardware manufacturers to make bigger ring buffers, or even to provide an indexed ring buffer which indexes by a user-defined part of the packet...
- ncmncm 6y ago
  I have found limited success dictating to hardware manufacturers. I would be satisfied if the kernel could just be persuaded to leave my cores the hell alone without an endless list of boot flags.
  - LunaSea 6y ago
    Do you have links about the flags that you mention or were you referring to process pining? Have you tried out the new eBPF libraries and kTLS?
    - ncmncm 6y ago
      Search isolcpus, nohz_full, and rcu_nocbs. Process pinning & isolation from other processes is the first step; keeping kernel interruptions out is the hard part. Kernel people generally think anything they do is implicitly more important than every single thing you do. eBPF and kTLS tend to involve the kernel in your business. But, eBPF can put code directly on certain NICs (e.g. Netronome), writing directly to user process memory. Then there is no need to pin or isolate, because the action is on the NIC, out of reach.

frumiousirc 6y ago

1 Gbps, unoptimized, commodity home network between an ancient thinkpad t520 and gen 4 i7 workstation using ZeroMQ throughput perf test gives more than 1M messages per second: $ ./local_thr tcp://192.168.1.123:5678 100 1000000 message size: 100 [B] message count: 1000000 mean throughput: 1143708 [msg/s] mean throughput: 914.966 [Mb/s]

7532yahoogmail 6y ago
The author specifically pointed out metric was packets per second not bytes per second because getting rather higher numbers with other units was easy.
davidcuddeback 6y ago
Assuming ZeroMQ batches messages like other message queue technologies, you're comparing apples to oranges. The article is about receiving 1M network packets per second.

7532yahoogmail 6y ago

OP has Solarflare NICs. So surely > 1MM pps is possible on that HW? But as you opened with 1 million per second on the Linux stack, I gather that kernel by-pass and such were by definition out of scope. The goal here was to always pass through the kernel. But if so, why use Solarflare? Why not see if one can do + 1MM pps on a nice, even beefy but not Solarflare/Mellonx NICs whose motivation includes kernel by-pass.