5 ms·

This looks like a great idea. For me this was a missing piece two months ago when playing with Docker. However I have strong doubts about the network performan

by t0mas88 12y ago

This looks like a great idea. For me this was a missing piece two months ago when playing with Docker. However I have strong doubts about the network performance, not only the overhead of the UDP encapsulation (that should be quite small), but mostly the capturing of packets with pcap and then handling them in user-mode. Looks like a lot of context-switches, copying and parsing with non-optimal code paths. Are there any benchmarks available? My feeling is that this will consume large amounts of CPU for moderate network loads and thus be unusable with most NoSQL kind of systems that benefit from clustering across hosts?

weavenetwork 12y ago
re benchmarks...publishing some is on the TODO list. See https://github.com/zettio/weave/issues/37 https://github.com/zettio/weave/issues/37. Informally, weave is pretty fast but it's not saturating Gbit Ethernet. As you say, capturing with pcap and handling packets in user space carries an appreciable overhead. We've got some issues filed to look at pcap alternatives and also generally aim to improve performance. re suitability for NoSQL clustering... depends on where the bottlenecks are; if you want to cluster for HA rather than scale, i.e. there aren't any real bottlenecks, then weave will work well. Same if you want to cluster because of CPU or memory bottlenecks. If, otoh, networking is the bottleneck then adding weave into the mix isn't going to improve matters.
- t0mas88 12y ago
  Ok, good to know. I think the challenge in taking another route than pcap is that you would need to do complex tricks with the existing network stack. Because if I understand the way Weave works you would really only need to do processing at the beginning of a connection and for some ARP requests etc while you don't need to do anything to existing TCP streams apart from encapsulating and forwarding?
  - weavenetwork 12y ago
    > complex tricks with the existing network stack To retain the essence of how weave operates, this would likely not just be complex but impossible, short of kernel hackery. > you would really only need to do processing at the beginning of a connection and for some ARP request Weave needs to look at every Ethernet packet. Well, the headers at least. It's a virtual Ethernet switch. It doesn't even really know about IP, let alone TCP streams. See https://github.com/zettio/weave#how-does-it-work https://github.com/zettio/weave#how-does-it-work

by t0mas88 12y ago

weavenetwork 12y ago

re benchmarks...publishing some is on the TODO list. See https://github.com/zettio/weave/issues/37 https://github.com/zettio/weave/issues/37. Informally, weave is pretty fast but it's not saturating Gbit Ethernet. As you say, capturing with pcap and handling packets in user space carries an appreciable overhead. We've got some issues filed to look at pcap alternatives and also generally aim to improve performance. re suitability for NoSQL clustering... depends on where the bottlenecks are; if you want to cluster for HA rather than scale, i.e. there aren't any real bottlenecks, then weave will work well. Same if you want to cluster because of CPU or memory bottlenecks. If, otoh, networking is the bottleneck then adding weave into the mix isn't going to improve matters.

t0mas88 12y ago
Ok, good to know. I think the challenge in taking another route than pcap is that you would need to do complex tricks with the existing network stack. Because if I understand the way Weave works you would really only need to do processing at the beginning of a connection and for some ARP requests etc while you don't need to do anything to existing TCP streams apart from encapsulating and forwarding?
- weavenetwork 12y ago
  > complex tricks with the existing network stack To retain the essence of how weave operates, this would likely not just be complex but impossible, short of kernel hackery. > you would really only need to do processing at the beginning of a connection and for some ARP request Weave needs to look at every Ethernet packet. Well, the headers at least. It's a virtual Ethernet switch. It doesn't even really know about IP, let alone TCP streams. See https://github.com/zettio/weave#how-does-it-work https://github.com/zettio/weave#how-does-it-work