5 ms·

User-space overlays are slow, Weave more so than flannel. The really interesting data-point for me is that flannel with VXLAN has negligible overhead.

hb42 11y ago
Is there any solution based on openvswitch ? If using vxlan to encapsulate containers L2 frames, the openvswitch kernel module should be able to route packets very well with much lower overhead than any userspace solution that do not access the nic memory directly.
- wmf 11y ago
  SocketPlane or OVN.
  - weavenetwork 11y ago
    I haven't seen those used in the wild yet, but fwiw, weave is working on a 'fast data path' that uses the same kernel path as ovs
  - techdragon 11y ago
    OVN is 0.1 fairy dust and unicorn horns at the moment, looks very promising but you're most likely going to be waiting another 12 months before its solving your problems. SocketPlane was bought by Docker and is now DEAD, to become whatever Docker decides is good enough to silence their critics. "See we bought these network guys, we swear we're taking networking and plugins seriously!"
lclarkmichalek 11y ago
Well, the latency is far from negligible. However, the bandwidth really is impressive. Having userspace as the control plane while keeping the data plane firmly in the kernel has long been the way of things, and this blog post's not-so-subtle message is "your problem is not novel enough to justify the cost of your solution". To be clear, there is nothing fundamental (to the best of my knowledge) that would prevent Weave implementing a VXLAN backend. It is simply a poor implementation at the moment, and even then, only in this regard. Who knows, maybe Weave's use of a gossip protocol vs flannel's CP etcd could make it more suited to some deployments. Though, that does need to be tested.
- justinsb 11y ago
  Sorry, yes, you're right - the latency overhead is not negligible (as a percentage, though in absolute terms it seems pretty tolerable). Is this added latency just VXLAN overhead, or is flannel userspace still involved in per-packet-processing? I'd love to see a test comparing flannel/VXLAN to a manually configured VXLAN tunnel. (I'm off to read the flannel code...)
  - deleted 11y ago
    [deleted]
q3k 11y ago
Actually, userspace packet switching can be fast (10GbE linespeed-fast) thanks to approaches like DPDK [1], where a userspace process has zero-copy, direct access to NIC ring buffers. 1 - http://dpdk.org/ http://dpdk.org/
- lclarkmichalek 11y ago
  The problem with DPDK is (as I understand it) that it doesn't handle multiplexing the connection between multiple cores/processes/containers (i.e. lack of polling makes scheduling hard/impossible). The nice thing about using the Linux kernel as your data plane is that you can still have all your bog standard routing, in addition to this fun VXLAN/GRE/etc stuff. That said, I haven't ever implemented anything using DPDK, so I may be talking out my arse. You are correct however that userspace networking is awesome, and I look forward to it becoming more and more prevalent in applications that can benefit from it.
  - swansonc 11y ago
    It depends on how you use DPDK. If I use it from the container directly to the NIC, you certainly do loose all of the kernel capabilities. However, we believe (but have not tested) that you can use a DPDK virtual interface in the container/vm (memnic or virtio) that connects to the DPDK driver in the kernel, so the path from the container/VM is 0 copy. The kernel then does it's processing, and then, another DPDK path could (potentially) be used to 0 copy the traffic to the NIC (really uncertain about that last stage). Basically, you are just using DPDK to save on the copy cost. This is all academic until tested, btw. As of yet, we (on Calico) haven't had anyone stand up and say that they need more performance than what the native data-path we use today is capable of delivering.
    - lclarkmichalek 11y ago
      Now that sounds interesting. I'd love to read about that if you ever do move the idea from paper to production :)
      - 11y ago
nine_k 11y ago
Using a VLAN (VXLAN), as the post shows?

hb42 11y ago

Is there any solution based on openvswitch ? If using vxlan to encapsulate containers L2 frames, the openvswitch kernel module should be able to route packets very well with much lower overhead than any userspace solution that do not access the nic memory directly.

wmf 11y ago
SocketPlane or OVN.
- weavenetwork 11y ago
  I haven't seen those used in the wild yet, but fwiw, weave is working on a 'fast data path' that uses the same kernel path as ovs
- techdragon 11y ago
  OVN is 0.1 fairy dust and unicorn horns at the moment, looks very promising but you're most likely going to be waiting another 12 months before its solving your problems. SocketPlane was bought by Docker and is now DEAD, to become whatever Docker decides is good enough to silence their critics. "See we bought these network guys, we swear we're taking networking and plugins seriously!"

lclarkmichalek 11y ago

Well, the latency is far from negligible. However, the bandwidth really is impressive. Having userspace as the control plane while keeping the data plane firmly in the kernel has long been the way of things, and this blog post's not-so-subtle message is "your problem is not novel enough to justify the cost of your solution". To be clear, there is nothing fundamental (to the best of my knowledge) that would prevent Weave implementing a VXLAN backend. It is simply a poor implementation at the moment, and even then, only in this regard. Who knows, maybe Weave's use of a gossip protocol vs flannel's CP etcd could make it more suited to some deployments. Though, that does need to be tested.

justinsb 11y ago
Sorry, yes, you're right - the latency overhead is not negligible (as a percentage, though in absolute terms it seems pretty tolerable). Is this added latency just VXLAN overhead, or is flannel userspace still involved in per-packet-processing? I'd love to see a test comparing flannel/VXLAN to a manually configured VXLAN tunnel. (I'm off to read the flannel code...)
- deleted 11y ago
  [deleted]

q3k 11y ago

Actually, userspace packet switching can be fast (10GbE linespeed-fast) thanks to approaches like DPDK [1], where a userspace process has zero-copy, direct access to NIC ring buffers. 1 - http://dpdk.org/ http://dpdk.org/

lclarkmichalek 11y ago
The problem with DPDK is (as I understand it) that it doesn't handle multiplexing the connection between multiple cores/processes/containers (i.e. lack of polling makes scheduling hard/impossible). The nice thing about using the Linux kernel as your data plane is that you can still have all your bog standard routing, in addition to this fun VXLAN/GRE/etc stuff. That said, I haven't ever implemented anything using DPDK, so I may be talking out my arse. You are correct however that userspace networking is awesome, and I look forward to it becoming more and more prevalent in applications that can benefit from it.
- swansonc 11y ago
  It depends on how you use DPDK. If I use it from the container directly to the NIC, you certainly do loose all of the kernel capabilities. However, we believe (but have not tested) that you can use a DPDK virtual interface in the container/vm (memnic or virtio) that connects to the DPDK driver in the kernel, so the path from the container/VM is 0 copy. The kernel then does it's processing, and then, another DPDK path could (potentially) be used to 0 copy the traffic to the NIC (really uncertain about that last stage). Basically, you are just using DPDK to save on the copy cost. This is all academic until tested, btw. As of yet, we (on Calico) haven't had anyone stand up and say that they need more performance than what the native data-path we use today is capable of delivering.
  - lclarkmichalek 11y ago
    Now that sounds interesting. I'd love to read about that if you ever do move the idea from paper to production :)
    - 11y ago

nine_k 11y ago

Using a VLAN (VXLAN), as the post shows?