3 ms·

Actually, userspace packet switching can be fast (10GbE linespeed-fast) thanks to approaches like DPDK [1], where a userspace process has zero-copy, direct acce

by q3k 12y ago

Actually, userspace packet switching can be fast (10GbE linespeed-fast) thanks to approaches like DPDK [1], where a userspace process has zero-copy, direct access to NIC ring buffers. 1 - http://dpdk.org/ http://dpdk.org/

lclarkmichalek 12y ago
The problem with DPDK is (as I understand it) that it doesn't handle multiplexing the connection between multiple cores/processes/containers (i.e. lack of polling makes scheduling hard/impossible). The nice thing about using the Linux kernel as your data plane is that you can still have all your bog standard routing, in addition to this fun VXLAN/GRE/etc stuff. That said, I haven't ever implemented anything using DPDK, so I may be talking out my arse. You are correct however that userspace networking is awesome, and I look forward to it becoming more and more prevalent in applications that can benefit from it.
- swansonc 12y ago
  It depends on how you use DPDK. If I use it from the container directly to the NIC, you certainly do loose all of the kernel capabilities. However, we believe (but have not tested) that you can use a DPDK virtual interface in the container/vm (memnic or virtio) that connects to the DPDK driver in the kernel, so the path from the container/VM is 0 copy. The kernel then does it's processing, and then, another DPDK path could (potentially) be used to 0 copy the traffic to the NIC (really uncertain about that last stage). Basically, you are just using DPDK to save on the copy cost. This is all academic until tested, btw. As of yet, we (on Calico) haven't had anyone stand up and say that they need more performance than what the native data-path we use today is capable of delivering.
  - lclarkmichalek 12y ago
    Now that sounds interesting. I'd love to read about that if you ever do move the idea from paper to production :)
    - swansonc 12y ago
      We'll let you know.
justincormack 12y ago
The problem is Docker containers expect a system network endpoint, while dpdk gives you a userspace network endpoint.
- wmf 12y ago
  Yep, DPDK is great for NIC-to-NIC traffic and maybe for VM-to-NIC now that vhost-user exists. But that's leading people to suggest DPDK as the answer for everything, which it isn't.

by q3k 12y ago

lclarkmichalek 12y ago

The problem with DPDK is (as I understand it) that it doesn't handle multiplexing the connection between multiple cores/processes/containers (i.e. lack of polling makes scheduling hard/impossible). The nice thing about using the Linux kernel as your data plane is that you can still have all your bog standard routing, in addition to this fun VXLAN/GRE/etc stuff. That said, I haven't ever implemented anything using DPDK, so I may be talking out my arse. You are correct however that userspace networking is awesome, and I look forward to it becoming more and more prevalent in applications that can benefit from it.

swansonc 12y ago
It depends on how you use DPDK. If I use it from the container directly to the NIC, you certainly do loose all of the kernel capabilities. However, we believe (but have not tested) that you can use a DPDK virtual interface in the container/vm (memnic or virtio) that connects to the DPDK driver in the kernel, so the path from the container/VM is 0 copy. The kernel then does it's processing, and then, another DPDK path could (potentially) be used to 0 copy the traffic to the NIC (really uncertain about that last stage). Basically, you are just using DPDK to save on the copy cost. This is all academic until tested, btw. As of yet, we (on Calico) haven't had anyone stand up and say that they need more performance than what the native data-path we use today is capable of delivering.
- lclarkmichalek 12y ago
  Now that sounds interesting. I'd love to read about that if you ever do move the idea from paper to production :)
  - swansonc 12y ago
    We'll let you know.

justincormack 12y ago

The problem is Docker containers expect a system network endpoint, while dpdk gives you a userspace network endpoint.

wmf 12y ago
Yep, DPDK is great for NIC-to-NIC traffic and maybe for VM-to-NIC now that vhost-user exists. But that's leading people to suggest DPDK as the answer for everything, which it isn't.