6 ms·

Is the poor performance of Weave really surprising? Userland networking will always be lesser performing than that of native kernel networking (the show is some

Is the poor performance of Weave really surprising? Userland networking will always be lesser performing than that of native kernel networking (the show is somewhat better with technologies such netmap/PF_RING/DPDK). Weave itself would probably improve in performance if it utilised a TUN/TAP device + static routes as opposed to its current mangling of packets with pcap and BPF.

dpw 11y ago
We did try TUN/TAP. It seemed to be slower than pcap. Our answer to weave performance concerns is on its way: http://blog.weave.works/2015/06/12/weave-fast-datapath/ http://blog.weave.works/2015/06/12/weave-fast-datapath/ We're really interested to hear how weave fast datapath works out in all kinds of environments, which is why we put the preview out. It's a shame that Arjan's post did not include numbers for FDP, but maybe he'll be able to include them in an update.
- justincormack 11y ago
  Indeed tun/tap performance is terrible, it seems to take about a whole CPU at gigabit.
  - jsolson 11y ago
    That, in general, is not the behavior I've observed on native hardware. Without getting into hard numbers, I routinely see multiple gigabits/second/core.
    - justincormack 11y ago
      With which options? Standard tap (or tun?), or macvtap? The last test I did (macvtap to an external interface but none of the offload options), a gigabit used a lot of CPU, like around a whole CPU just copying data.
- slintes 11y ago
  regarding fast data path: Arjan's colleague Paul posted some new numbers: http://paulbakker.io/docker/docker-cloud-network-performance/ http://paulbakker.io/docker/docker-cloud-network-performance...
api 11y ago
User space networking overlays impose a penalty just like and for much the same reason as dynamic language, garbage collection, etc. -- it's a trade-off between versatility and performance. The best networking performance is achieved by binding directly to the metal using SOCK_RAW, but that also gives you the least flexibility.
pyvpx 11y ago
oddly, there is a growing trend of going userland and skipping native kernel because, well...the kernel is slow. userland can be made fast. I think seeing Weave implemented in something like say, Snabb Switch, would at the very least be interesting -- from a performance standpoint.
- bboreham 11y ago
  A major point of weave is that you don't have to change your code, so it has to work with the kernel network APIs.
  - dpw 11y ago
    Indeed. Kernel-bypass networking is great when you are building something that just receives and send packets (and maybe does other things that can be moved to other threads within the same process using lightweight synchronization mechanisms). It's not great when you want to interact with other processes, because then you have to go through the kernel anyway. And if you have to go through the kernel anyway, the fastest approach is to do as much as you reasonably can in the kernel.

dpw 11y ago

We did try TUN/TAP. It seemed to be slower than pcap. Our answer to weave performance concerns is on its way: http://blog.weave.works/2015/06/12/weave-fast-datapath/ http://blog.weave.works/2015/06/12/weave-fast-datapath/ We're really interested to hear how weave fast datapath works out in all kinds of environments, which is why we put the preview out. It's a shame that Arjan's post did not include numbers for FDP, but maybe he'll be able to include them in an update.

justincormack 11y ago
Indeed tun/tap performance is terrible, it seems to take about a whole CPU at gigabit.
- jsolson 11y ago
  That, in general, is not the behavior I've observed on native hardware. Without getting into hard numbers, I routinely see multiple gigabits/second/core.
  - justincormack 11y ago
    With which options? Standard tap (or tun?), or macvtap? The last test I did (macvtap to an external interface but none of the offload options), a gigabit used a lot of CPU, like around a whole CPU just copying data.
slintes 11y ago
regarding fast data path: Arjan's colleague Paul posted some new numbers: http://paulbakker.io/docker/docker-cloud-network-performance/ http://paulbakker.io/docker/docker-cloud-network-performance...

api 11y ago

User space networking overlays impose a penalty just like and for much the same reason as dynamic language, garbage collection, etc. -- it's a trade-off between versatility and performance. The best networking performance is achieved by binding directly to the metal using SOCK_RAW, but that also gives you the least flexibility.

pyvpx 11y ago

oddly, there is a growing trend of going userland and skipping native kernel because, well...the kernel is slow. userland can be made fast. I think seeing Weave implemented in something like say, Snabb Switch, would at the very least be interesting -- from a performance standpoint.

bboreham 11y ago
A major point of weave is that you don't have to change your code, so it has to work with the kernel network APIs.
- dpw 11y ago
  Indeed. Kernel-bypass networking is great when you are building something that just receives and send packets (and maybe does other things that can be moved to other threads within the same process using lightweight synchronization mechanisms). It's not great when you want to interact with other processes, because then you have to go through the kernel anyway. And if you have to go through the kernel anyway, the fastest approach is to do as much as you reasonably can in the kernel.