3 ms·

Wireblast a 100 Gbps packet generator in Go using AF_XDP

53 points by bgpdude 2mo ago

mjevans 2mo ago
https://toonk.io/wireblast-building-a-line-rate-packet-generator-in-go-with-af_xdp/index.html https://toonk.io/wireblast-building-a-line-rate-packet-gener...
- bgpdude 2mo ago
  Ah thanks!!
spaceywilly 2mo ago
That’s very interesting, thank you for sharing. Looks like it could be a very useful tool for testing high performance networking. I wonder if something similar could be done using TC BPF instead of AF_XDP? My only reservation about AF XDP is that it requires a special NIC to support it, so it may not be useful for a “regular Joe” user. I wonder if TC BPF would also work since it similarly bypasses the Kernel networking stack, I believe you can put packets directly into the NIC TX queue for transmission
- bgpdude 2mo ago
  you can use generic af_xdp which sits at the TC layer. Just get a bit less performance.
  - bgpdude 2mo ago
    That's what the veth examples are using
- tptacek 2mo ago
  You can, but the interesting thing about AF_XDP is that you've got a userland path to writing directly to the card's DMA buffers; TC BPF still allocates an skbuff for every packet you send.
  - adrian_b 2mo ago
    Also with liburing you have zero-copy send and receive operations for normal protocols like UDP or TCP, but this requires a NIC that supports scatter/gather DMA (so that the packet headers go to/from kernel buffers, while the data goes to/from userland buffers). AF_XDP is also available in older kernels, but with recent enough kernels (zero-copy receive is a recent addition) and with a good NIC, liburing should provide a similar performance.
- Palomides 2mo ago
  it seems like every NIC on the market that can do 100Gb has support in its linux kernel driver, so probably not a big deal in practice
bgpdude 2mo ago
The video demo is pretty neat as well https://www.youtube.com/watch?v=8EmxvaOR5uA https://www.youtube.com/watch?v=8EmxvaOR5uA
barryvand 2mo ago
ooh no more DPDK, this will make it a lot easier. I just started working with trex but it's all really complicated. Going to give this a try
- tptacek 2mo ago
  I think, and I'm saying this in part to get someone to correct me, that post-XDP (so 5 years or so now) DPDK is basically obsolete. Is there a circumstance where it would make sense to start from DPDK rather than XDP?
  - bgpdude 2mo ago
    I think you're correct. Not really aware of any real limitations, other than it's slightly slower than dpdk (it's not a complete bypass), but at a much easier ease of use.
    - tptacek 2mo ago
      AF_XDP kind of is a complete bypass, right? RX get scooped right off the DMA buffer for the card, and TX get shoved right back in.
      - bgpdude 2mo ago
        Yeah fair point. I should have been more precise. AF_XDP with ZC bypasses the kernel networking stack and the NIC DMA's directly into UMEM and so in that sense it absolutely is a kernel bypass. The difference I was getting at vs DPDK is that the kernel NIC driver/NAPI/XDP path is still involvd. With DPDK the userspace PMD is effectively driving the NIC and accessing the queues directly. either way, it's great and everyone should use it :) that is assuming they have a use case for it. The use-cases are perhaps somewhat limited as it also bypasses the kernel tcp-ip stack, so you gotta do a lot yourself.
  - pstavirs 2mo ago
    At this point there's a much bigger ecosystem for DPDK than AF_XDP I think. Also more people know about DPDK than AF_XDP right now e.g. the Ostinato traffic generator's line-rate Turbo functionality uses AF_XDP but most customers assume it uses DPDK. Disclosure: Ostinato creator here.
m4r1k 2mo ago
As someone who has worked in the Telco/NFV industry for well over 10 years now, I can confidently say this is a breath of fresh air. Don’t get me wrong, DPDK-based tools are great, but setting up Cisco TRex or even basic TestPMD requires a lot of specialized knowledge. Indeed, in the era of LLMs, the barrier to entry is much lower, but hardware requirements remain regardless of whether an AI agent handles the implementation for you. For anybody looking for XDP-supported drivers/NICs -> https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md#xdp https://github.com/iovisor/bcc/blob/master/docs/kernel-versi...

53 points by bgpdude 2mo ago

https://toonk.io/wireblast-building-a-line-rate-packet-generator-in-go-with-af_xdp/index.html https://toonk.io/wireblast-building-a-line-rate-packet-gener...

bgpdude 2mo ago
Ah thanks!!

spaceywilly 2mo ago

That’s very interesting, thank you for sharing. Looks like it could be a very useful tool for testing high performance networking. I wonder if something similar could be done using TC BPF instead of AF_XDP? My only reservation about AF XDP is that it requires a special NIC to support it, so it may not be useful for a “regular Joe” user. I wonder if TC BPF would also work since it similarly bypasses the Kernel networking stack, I believe you can put packets directly into the NIC TX queue for transmission

bgpdude 2mo ago
you can use generic af_xdp which sits at the TC layer. Just get a bit less performance.
- bgpdude 2mo ago
  That's what the veth examples are using
tptacek 2mo ago
You can, but the interesting thing about AF_XDP is that you've got a userland path to writing directly to the card's DMA buffers; TC BPF still allocates an skbuff for every packet you send.
- adrian_b 2mo ago
  Also with liburing you have zero-copy send and receive operations for normal protocols like UDP or TCP, but this requires a NIC that supports scatter/gather DMA (so that the packet headers go to/from kernel buffers, while the data goes to/from userland buffers). AF_XDP is also available in older kernels, but with recent enough kernels (zero-copy receive is a recent addition) and with a good NIC, liburing should provide a similar performance.
Palomides 2mo ago
it seems like every NIC on the market that can do 100Gb has support in its linux kernel driver, so probably not a big deal in practice

bgpdude 2mo ago

The video demo is pretty neat as well https://www.youtube.com/watch?v=8EmxvaOR5uA https://www.youtube.com/watch?v=8EmxvaOR5uA

barryvand 2mo ago

ooh no more DPDK, this will make it a lot easier. I just started working with trex but it's all really complicated. Going to give this a try

tptacek 2mo ago
I think, and I'm saying this in part to get someone to correct me, that post-XDP (so 5 years or so now) DPDK is basically obsolete. Is there a circumstance where it would make sense to start from DPDK rather than XDP?
- bgpdude 2mo ago
  I think you're correct. Not really aware of any real limitations, other than it's slightly slower than dpdk (it's not a complete bypass), but at a much easier ease of use.
  - tptacek 2mo ago
    AF_XDP kind of is a complete bypass, right? RX get scooped right off the DMA buffer for the card, and TX get shoved right back in.
    - bgpdude 2mo ago
      Yeah fair point. I should have been more precise. AF_XDP with ZC bypasses the kernel networking stack and the NIC DMA's directly into UMEM and so in that sense it absolutely is a kernel bypass. The difference I was getting at vs DPDK is that the kernel NIC driver/NAPI/XDP path is still involvd. With DPDK the userspace PMD is effectively driving the NIC and accessing the queues directly. either way, it's great and everyone should use it :) that is assuming they have a use case for it. The use-cases are perhaps somewhat limited as it also bypasses the kernel tcp-ip stack, so you gotta do a lot yourself.
- pstavirs 2mo ago
  At this point there's a much bigger ecosystem for DPDK than AF_XDP I think. Also more people know about DPDK than AF_XDP right now e.g. the Ostinato traffic generator's line-rate Turbo functionality uses AF_XDP but most customers assume it uses DPDK. Disclosure: Ostinato creator here.

m4r1k 2mo ago

As someone who has worked in the Telco/NFV industry for well over 10 years now, I can confidently say this is a breath of fresh air. Don’t get me wrong, DPDK-based tools are great, but setting up Cisco TRex or even basic TestPMD requires a lot of specialized knowledge. Indeed, in the era of LLMs, the barrier to entry is much lower, but hardware requirements remain regardless of whether an AI agent handles the implementation for you. For anybody looking for XDP-supported drivers/NICs -> https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md#xdp https://github.com/iovisor/bcc/blob/master/docs/kernel-versi...