4 ms·

can you give an ELI5 of the benefits of XDP over current technologies?

ilovecaching 8y ago
eBPF programs are bytecode that runs on a virtual machine in the kernel. The programs are limited by number of instructions and computational complexity (they must halt). These programs can transfer data back to user space using maps, which are simple data structures for passing things like lists of ips, for example. These eBPF programs are attached to probes in the kernel where they can gather information at the probe point. XDP is essentially a probe before any of the networking stack has happened. An XDP eBPF program can read and manipulate raw packet data, allowing you to process the packet as fast as your network card will allow, before that data is ever seen by the slow kernel networking code. Unlike other options like DPDK, eBPF programs are tightly integrated in the kernel. They allow you to write kernel code that the kernel can safely run. They are simple to write, easy to install (it's just a syscall) and there are tools in Rust and other languages (check out BCC) that allow you to compile to eBPF bytecode from various languages. You're also not just limited to networking. eBPF programs can hook in to many places in the kernel, and there are already efforts to build new system performance tools using eBPF.
- buttersbrian 8y ago
  What are the downsides of eBPF(XDP)programs compared to the previous options (DPDK etc.)? Do you consider eBPF(XDP) to be the clear next step in this space?
  - ilovecaching 8y ago
    https://t.co/UjQE8Z6HlD https://t.co/UjQE8Z6HlD Check this out, it spells out why DPDK isn't a great solution. The downsides to XDP is that it requires newer kernel releases, also XDP eBPF programs do not provide chaining and are limited in size (they can be chained, you just have to do it yourself).
    - baolongtrann 8y ago
      Maybe I missed something, but the argument that was stated in that talk, "DPDK is not linux," is not very clear to me. How is "not linux" a bad thing? I'm not saying DPDK is perfect. As an user, here are some of the most annoying things about DPDK: - 100% CPU, even in down time. There has been works done recently in power management for DPDK but it still is quite limited. - Debugging is difficult. Valgrind doesn't even work ootb. - Very limited tool set compared to linux. For example no tcpdump. - Setting up is cumbersome. Allocating hugepages must be done soon after reboot, NICs must be bound to uio,... - Ad-hoc Layer 4-7. That said, some of the packet processing libraries that come with DPDK is awesome. Once you get through the first few hurdles the dev experience is actually quite nice. I think combining DPDK with XDP is very promising.
      - cscscsc 8y ago
        There is actually a patch to enable AF_XDP for DPDK: http://mails.dpdk.org/archives/dev/2018-August/109792.html http://mails.dpdk.org/archives/dev/2018-August/109792.html And also a recent talk on how AF_XDP can be optimised further to get closer to DPDK speed (http://vger.kernel.org/lpc-networking2018.html#session-11 http://vger.kernel.org/lpc-networking2018.html#session-11) where it concludes "DPDK still faster for 'notouch', but AF XDP on par when data is touched". And latter is what matters. I think combining the two definitely has huge potential: - All the Linux tooling can be used again as drivers in kernel are used. - No 100% busy polling needed, it's definitely not a must for workloads. - Easy setup as simply kernel drivers are used as is. - Vendors only have to maintain their kernel drivers, but not DPDK ones, so less cost - Users can simply switch from one NIC to another without any hassle - DPDK library for application development can be fully reused. Sounds like a big win to me for both worlds.

eBPF programs are bytecode that runs on a virtual machine in the kernel. The programs are limited by number of instructions and computational complexity (they must halt). These programs can transfer data back to user space using maps, which are simple data structures for passing things like lists of ips, for example. These eBPF programs are attached to probes in the kernel where they can gather information at the probe point. XDP is essentially a probe before any of the networking stack has happened. An XDP eBPF program can read and manipulate raw packet data, allowing you to process the packet as fast as your network card will allow, before that data is ever seen by the slow kernel networking code. Unlike other options like DPDK, eBPF programs are tightly integrated in the kernel. They allow you to write kernel code that the kernel can safely run. They are simple to write, easy to install (it's just a syscall) and there are tools in Rust and other languages (check out BCC) that allow you to compile to eBPF bytecode from various languages. You're also not just limited to networking. eBPF programs can hook in to many places in the kernel, and there are already efforts to build new system performance tools using eBPF.

buttersbrian 8y ago
What are the downsides of eBPF(XDP)programs compared to the previous options (DPDK etc.)? Do you consider eBPF(XDP) to be the clear next step in this space?
- ilovecaching 8y ago
  https://t.co/UjQE8Z6HlD https://t.co/UjQE8Z6HlD Check this out, it spells out why DPDK isn't a great solution. The downsides to XDP is that it requires newer kernel releases, also XDP eBPF programs do not provide chaining and are limited in size (they can be chained, you just have to do it yourself).
  - baolongtrann 8y ago
    Maybe I missed something, but the argument that was stated in that talk, "DPDK is not linux," is not very clear to me. How is "not linux" a bad thing? I'm not saying DPDK is perfect. As an user, here are some of the most annoying things about DPDK: - 100% CPU, even in down time. There has been works done recently in power management for DPDK but it still is quite limited. - Debugging is difficult. Valgrind doesn't even work ootb. - Very limited tool set compared to linux. For example no tcpdump. - Setting up is cumbersome. Allocating hugepages must be done soon after reboot, NICs must be bound to uio,... - Ad-hoc Layer 4-7. That said, some of the packet processing libraries that come with DPDK is awesome. Once you get through the first few hurdles the dev experience is actually quite nice. I think combining DPDK with XDP is very promising.
    - cscscsc 8y ago
      There is actually a patch to enable AF_XDP for DPDK: http://mails.dpdk.org/archives/dev/2018-August/109792.html http://mails.dpdk.org/archives/dev/2018-August/109792.html And also a recent talk on how AF_XDP can be optimised further to get closer to DPDK speed (http://vger.kernel.org/lpc-networking2018.html#session-11 http://vger.kernel.org/lpc-networking2018.html#session-11) where it concludes "DPDK still faster for 'notouch', but AF XDP on par when data is touched". And latter is what matters. I think combining the two definitely has huge potential: - All the Linux tooling can be used again as drivers in kernel are used. - No 100% busy polling needed, it's definitely not a must for workloads. - Easy setup as simply kernel drivers are used as is. - Vendors only have to maintain their kernel drivers, but not DPDK ones, so less cost - Users can simply switch from one NIC to another without any hassle - DPDK library for application development can be fully reused. Sounds like a big win to me for both worlds.