5 ms·

"With XDP we can drop 10 million packets per second on a single CPU." is not really correct, the packets are being dropped on the network card which has its own

"With XDP we can drop 10 million packets per second on a single CPU." is not really correct, the packets are being dropped on the network card which has its own processor. The packets never get transferred to the main CPU.

bhouston 8y ago
Exactly. And thus this approach does not scale with more cores at all. Let me show how to render 100B polygons per second on a single cpu by using a gpu to do all the work... Argh.
deleted 8y ago
[deleted]
rayiner 8y ago
That’s not true. While there are NICS that can drop packets in hardware (e.g. Chelsio) that is not what XDP does. XDP runs eBPF programs in a driver-implemented recieve callback. The recieve callback happens at the lowest level of the Linux network stack, when an interrupt is delivered to a processor and a packet descriptor is queued by the NIC, but before the generic packet handling metadata (SKB) is allocated. So XDP operates on the packet in place, but the code runs on the host CPU.
- iajrz 8y ago
  When you say "in place" you mean the data is still on the NIC? Or is the data 'elsewhere in RAM'? I'm curious because this would be the difference between "scale with more cores" or "scale with more NICs".
  - penagwin 8y ago
    If I read that correctly the data is still on the NIC. The CPU receives an interrupt - notifying it of a new packet, which it then decides to drop? Thus the CPU is the one dropping packets, but the data (most of it at least) is still on the NIC. Please correct me if I'm wrong.
    - emmericp 8y ago
      This all happens after the packet is transferred over PCIe to the CPU/memory (even the interrupt happens after DMA transfer). Otherwise the XDP code wouldn't have access to the packet to filter it; the example doesn't just drop all packets, it drops all packets matching a /24 subnet which happens to be all packets here ;)
      - penagwin 8y ago
        Ah ok! I was kinda confused by that xD
  - rayiner 8y ago
    Generally, how it works is that there will be multiple hardware-managed ring buffers allocated in host memory, which will point to packet data which also is allocated in host memory. When a packet is received, the NIC will DMA the packet into host memory, update the descriptors in the ring buffers, and trigger an interrupt.[1] That is the point at which the kernel driver can access the packet (in host memory). See: https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xl710-10-40-controller-datasheet.pdf https://www.intel.com/content/dam/www/public/us/en/documents... (section 8.3.3). By "in place" I mean it can be accessed from the location where the NIC put it. In terms of scaling with multiple CPUs, generally these days the NIC will be able to manage multiple separate ring buffers. Ring buffers can be dedicated to specific CPUs, and the NIC can distribute packets among the ring buffers (in hardware) by hashing on the packet's IP 5-tuple.[2] So long as you have packets that are distributed evenly by that mechanism (not the case when, e.g. the packets all belong to the same connection), you can scale with multiple CPUs. (Presumably the NIC can perform the distribution process in hardware at line rate.) [1] At high packet rates, there might be an interrupt less frequently than for every packet. [2] See section 7.1.8 of the linked manual.
- barbegal 8y ago
  XDP can be offloaded to the NIC [1]. This is what I took the quote "the userspace loader, which loads the code onto the right network card and manages it" to mean. But I think you might be right in this case, it seems more likely that the XDP processing is being done by the network card driver. [1] https://cilium.readthedocs.io/en/latest/bpf/#iproute2 https://cilium.readthedocs.io/en/latest/bpf/#iproute2
  - fdvfvf 8y ago
    XDP can run in the driver in the "normal" mode, so it's on host CPU. But for some NICs like Netronome nfp, the kernel can JIT the BPF program into the instruction set for nfp and then run 'offloaded' on the card directly, so not even affecting host CPU in any way. See also here for a recent blog post: https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise/ https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-wha... So both is possible, but even with XDP running on host CPU like in Cloudflare case above you get excellent performance numbers.

bhouston 8y ago

Exactly. And thus this approach does not scale with more cores at all. Let me show how to render 100B polygons per second on a single cpu by using a gpu to do all the work... Argh.

deleted 8y ago

[deleted]

rayiner 8y ago

That’s not true. While there are NICS that can drop packets in hardware (e.g. Chelsio) that is not what XDP does. XDP runs eBPF programs in a driver-implemented recieve callback. The recieve callback happens at the lowest level of the Linux network stack, when an interrupt is delivered to a processor and a packet descriptor is queued by the NIC, but before the generic packet handling metadata (SKB) is allocated. So XDP operates on the packet in place, but the code runs on the host CPU.

iajrz 8y ago
When you say "in place" you mean the data is still on the NIC? Or is the data 'elsewhere in RAM'? I'm curious because this would be the difference between "scale with more cores" or "scale with more NICs".
- penagwin 8y ago
  If I read that correctly the data is still on the NIC. The CPU receives an interrupt - notifying it of a new packet, which it then decides to drop? Thus the CPU is the one dropping packets, but the data (most of it at least) is still on the NIC. Please correct me if I'm wrong.
  - emmericp 8y ago
    This all happens after the packet is transferred over PCIe to the CPU/memory (even the interrupt happens after DMA transfer). Otherwise the XDP code wouldn't have access to the packet to filter it; the example doesn't just drop all packets, it drops all packets matching a /24 subnet which happens to be all packets here ;)
    - penagwin 8y ago
      Ah ok! I was kinda confused by that xD
- rayiner 8y ago
  Generally, how it works is that there will be multiple hardware-managed ring buffers allocated in host memory, which will point to packet data which also is allocated in host memory. When a packet is received, the NIC will DMA the packet into host memory, update the descriptors in the ring buffers, and trigger an interrupt.[1] That is the point at which the kernel driver can access the packet (in host memory). See: https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xl710-10-40-controller-datasheet.pdf https://www.intel.com/content/dam/www/public/us/en/documents... (section 8.3.3). By "in place" I mean it can be accessed from the location where the NIC put it. In terms of scaling with multiple CPUs, generally these days the NIC will be able to manage multiple separate ring buffers. Ring buffers can be dedicated to specific CPUs, and the NIC can distribute packets among the ring buffers (in hardware) by hashing on the packet's IP 5-tuple.[2] So long as you have packets that are distributed evenly by that mechanism (not the case when, e.g. the packets all belong to the same connection), you can scale with multiple CPUs. (Presumably the NIC can perform the distribution process in hardware at line rate.) [1] At high packet rates, there might be an interrupt less frequently than for every packet. [2] See section 7.1.8 of the linked manual.
barbegal 8y ago
XDP can be offloaded to the NIC [1]. This is what I took the quote "the userspace loader, which loads the code onto the right network card and manages it" to mean. But I think you might be right in this case, it seems more likely that the XDP processing is being done by the network card driver. [1] https://cilium.readthedocs.io/en/latest/bpf/#iproute2 https://cilium.readthedocs.io/en/latest/bpf/#iproute2
- fdvfvf 8y ago
  XDP can run in the driver in the "normal" mode, so it's on host CPU. But for some NICs like Netronome nfp, the kernel can JIT the BPF program into the instruction set for nfp and then run 'offloaded' on the card directly, so not even affecting host CPU in any way. See also here for a recent blog post: https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise/ https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-wha... So both is possible, but even with XDP running on host CPU like in Cloudflare case above you get excellent performance numbers.