31 ms·
Weave is kinda slow
- justinsb 11y agoUser-space overlays are slow, Weave more so than flannel. The really interesting data-point for me is that flannel with VXLAN has negligible overhead.
- hb42 11y agoIs there any solution based on openvswitch ? If using vxlan to encapsulate containers L2 frames, the openvswitch kernel module should be able to route packets very well with much lower overhead than any userspace solution that do not access the nic memory directly.
- wmf 11y agoSocketPlane or OVN.
- weavenetwork 11y agoI haven't seen those used in the wild yet, but fwiw, weave is working on a 'fast data path' that uses the same kernel path as ovs
- techdragon 11y agoOVN is 0.1 fairy dust and unicorn horns at the moment, looks very promising but you're most likely going to be waiting another 12 months before its solving your problems. SocketPlane was bought by Docker and is now DEAD, to become whatever Docker decides is good enough to silence their critics. "See we bought these network guys, we swear we're taking networking and plugins seriously!"
- lclarkmichalek 11y agoWell, the latency is far from negligible. However, the bandwidth really is impressive. Having userspace as the control plane while keeping the data plane firmly in the kernel has long been the way of things, and this blog post's not-so-subtle message is "your problem is not novel enough to justify the cost of your solution". To be clear, there is nothing fundamental (to the best of my knowledge) that would prevent Weave implementing a VXLAN backend. It is simply a poor implementation at the moment, and even then, only in this regard. Who knows, maybe Weave's use of a gossip protocol vs flannel's CP etcd could make it more suited to some deployments. Though, that does need to be tested.
- justinsb 11y agoSorry, yes, you're right - the latency overhead is not negligible (as a percentage, though in absolute terms it seems pretty tolerable). Is this added latency just VXLAN overhead, or is flannel userspace still involved in per-packet-processing? I'd love to see a test comparing flannel/VXLAN to a manually configured VXLAN tunnel. (I'm off to read the flannel code...)
- deleted 11y ago[deleted]
- q3k 11y agoActually, userspace packet switching can be fast (10GbE linespeed-fast) thanks to approaches like DPDK [1], where a userspace process has zero-copy, direct access to NIC ring buffers. 1 - http://dpdk.org/ http://dpdk.org/
- lclarkmichalek 11y agoThe problem with DPDK is (as I understand it) that it doesn't handle multiplexing the connection between multiple cores/processes/containers (i.e. lack of polling makes scheduling hard/impossible). The nice thing about using the Linux kernel as your data plane is that you can still have all your bog standard routing, in addition to this fun VXLAN/GRE/etc stuff. That said, I haven't ever implemented anything using DPDK, so I may be talking out my arse. You are correct however that userspace networking is awesome, and I look forward to it becoming more and more prevalent in applications that can benefit from it.
- swansonc 11y agoIt depends on how you use DPDK. If I use it from the container directly to the NIC, you certainly do loose all of the kernel capabilities. However, we believe (but have not tested) that you can use a DPDK virtual interface in the container/vm (memnic or virtio) that connects to the DPDK driver in the kernel, so the path from the container/VM is 0 copy. The kernel then does it's processing, and then, another DPDK path could (potentially) be used to 0 copy the traffic to the NIC (really uncertain about that last stage). Basically, you are just using DPDK to save on the copy cost. This is all academic until tested, btw. As of yet, we (on Calico) haven't had anyone stand up and say that they need more performance than what the native data-path we use today is capable of delivering.
- lclarkmichalek 11y agoNow that sounds interesting. I'd love to read about that if you ever do move the idea from paper to production :)
- 11y ago
- nine_k 11y agoUsing a VLAN (VXLAN), as the post shows?
- mdekkers 11y ago"Thankfully, the fact that these problems were solved decades ago has not stopped people from coming up with their own solutions, and we now all get to witness the resulting disasters." Spot on
- deleted 11y ago[deleted]
- Confusion 11y agoSo what are the decades old solutions to these problems?
- otterley 11y agoThankfully, they are all discussed in the linked article.
- Confusion 11y agoI read the article: if they are, that is quite unclear to me. Possible answers that are discussed are only 'some sort of IP encapsulation', which is vague and GRE, which is just a single solution. He doesn't seem to disapprove of VXlan, so probably something was missing in 'IP encapsulation and GRE'. Was 'all problems solved decades ago' merely hyperbole or is there actually something to it?
- otterley 11y agoIs GRE inadequate? A single solution that solves most cases, is codified in an RFC, and has mature, reliable, performant implementations sounds like a winner to me.
- wmf 11y agoGRE (well NVGRE) works, but it's not offloaded by most NICs and it doesn't have port numbers to provide as much ECMP entropy as UDP.
- SEJeff 11y agoflannel appears to be what kubernetes is using as well and I know it is what RedHat is using for their OpenShift platform ontop of k8. It seems like the obvious path forward.
- karmakaze 11y agoMore and more, I'm finding I like what comes from the CoreOS group.
- tracker1 11y agoI have to agree... I do hope to see something close to a drop-in coreos cluster on a number of networks that becomes easier to use in practice and on in-house servers... The reliability is still a concern, and when you are using a PFM (pure f*ing magic) solution, it needs to be rock solid. I think within 1-2 years CoreOS will be a choice solution... unfortunately I can't wait for that, and don't have the time to become intimate with every detail therein. Although etcd had a lot of issues even when I was in the evaluation stage... I setup a dev cluster, having a node running on half a dozen machines (developer workstations included) in the office. The etcd cluster broke 3-4 times before I abandoned the thought of using it. With the 2.x release, my test network (3 virtual servers) was a lot more reliable, but I decided I didn't need it that much, opting to use my cloud provider's table solution for maintaining my minimal configuration... For my deployment, the cluster is relatively small, 3 nodes mainly for redunjdancy... our load could pretty easily be handled by a single larger server. That said, I decided to just use dokku-alt for deployments directly to each server (each service with two instances on each server). To make deployments easier to use, I have internal dns setup with wildcards using v#-#-#.instance-#.service.host.domain.local as a deploy pattern for versioned targets and instance-#.service.host.domain.local for default instances. I have two nginx servers setup for ssl, spdy termination and caching configured to relay requests to the dokku cluster for user-facing services. This is turning out to be simpler than implementing etcd/flannel for that communications layer. Each instance is passed relevant information regarding the host/service information to report it's own health to table storage, internal communication is using ZeroMQ req/res interfaces to minimize communication overhead, and allow for reconnects. Which is working a little better than relying on WebService/REST interfaces internally.
- api 11y agoNetworking is a disaster. Kernel level networking is nice, but that requires access to the kernel which you can't provide in a container. Doing so means containers no longer contain. IPV6 was supposed to solve a lot of this by having an address space so huge you could easily give every vm host a few billion IPs. But nobody uses it.
- ploxiln 11y agoContainers already fully depend on a shared and complicated kernel to do containment (which is why some say "containers don't contain" e.g. http://www.projectatomic.io/blog/2014/09/yet-another-reason-containers-don-t-contain-kernel-keyrings/ http://www.projectatomic.io/blog/2014/09/yet-another-reason-... ). Virtual Machines have a smaller attack interface to the host, so they do have a chance of containing malicious code. Kernels were supposed to contain processes, providing them with a virtual private memory space, virtual storage access so they can't read or write any file you don't want them to, etc... but only openbsd is really trying to make that work.
- thinkingkong 11y agoNetworking isnt a disaster. It's just that the current container ecosystem on linux hasnt yet resulted in any real domain specific improvements yet. For "default" systems, networking is pretty damned reliable and performant. A lot of the problems with regards to containment have already been solved in different systems. I believe solaris or opensolaris had Crossbow [1]. Any system that aims to provide connectivity needs to do the least amount of encapsulation possible and probably be a kernel module. 1. http://en.m.wikipedia.org/wiki/OpenSolaris_Network_Virtualization_and_Resource_Control http://en.m.wikipedia.org/wiki/OpenSolaris_Network_Virtualiz...
- adamc 11y ago"the least amount of encapsulation possible" doesn't sound like much of a solution if what you are looking for is networking with strong encapsulation.
- 11y ago
- d136o 11y agowhat I appreciate about weave is that it solved the cross host container networking problem easily (it's very very easy to use) and now, i.e. no waiting for promises of future solutions or fooling around with more complicated set ups. Here's where I got burned: I set up an elastic search cluster using containers and weave and life was great, but it then grew to need another node. Upon setting up the new host with docker and weave it turns out the new node couldn't talk to the old nodes because they were using different versions of the weave protocol. That was disappointing and a pain, I stopped experimenting with weave at that point.
- bboreham 11y agoHi, I work on Weave; it may well have been one of my commits that broke the protocol compatibility for you. We've changed things over time to improve performance and resilience. It should be fairly straightforward to deploy the same version on every host, but maybe that wasn't explained well enough, or didn't work for you. We'd welcome more feedback. Lastly, I appreciate the positive comments. "Very very easy to use" is exactly what we aimed for.
- garthk 11y agoThanks for using your own account.
- aajtechmail 11y agoWHAT
- ceequof 11y agoIn the terminal snippets, latency is specified in "us", which I'm guessing is µs, microseconds. In the table, those latency numbers are specified in "ms". Also microseconds? Couldn't possibly be milliseconds, for two VMs that should only be a couple dozen metres away from each other, right?
- lclarkmichalek 11y agoWow, huge mea culpa on that one. I would assume that qperf is using us to mean microseconds, and judging by the source line char *tab[] ={ "ns", "us", "ms", "sec" }; it would certainly seem that way. I'll update the post, thanks. Now I'll just be waiting for someone to complain about inconsistent precision. I am not a good scientist...
- hueving 11y agoI think they are supposed to be 'us' based on the qperf output.
- weavenetwork 11y agoyes, microseconds surely - we estimate that the cost of switching from kernel to user space, and back again, several times, adds up to 2-300 microseconds. in return for this, you gain a lot of flexibility and ease of use. for apps that need lower latency, weave is currently working on implementing a fast data path option..
- tehbeard 11y agoOut of interest, how does rancher's network offering compare to weave/flannel?
- weavenetwork 11y agohello, weave here. a few very quick comments! weave has lots of very happy users who find that weave is plenty fast enough for their purposes, see eg http://blog.weave.works/2015/02/24/get-your-kicks-on-cloud66/ http://blog.weave.works/2015/02/24/get-your-kicks-on-cloud66... the strong points of weave network, as it is right now, are ease of use (not to be sniffed at), and enormous flexibility. it is really quite easy to create an application involving containers, that runs anywhere and does not commit you to specific architectural choices... typically though, one weave network might be used by one app, or just a few. but you might run a lot of weave networks weave works very nicely with kubernetes - later I shall dig out a few links for this in our own tests, throughput varies by payload size; we tend to think of weave-as-is is best compared with using, for example, amazon cloud networking directly for users with higher perf needs, we have a fast data path in the works, that uses the same data path that ovs implementations use... the hard problem to solve here is making that stuff incredibly easy and robust w.r.t deployment choices -- see above :-)
- LukeHoersten 11y agoThanks for taking the time to write a thoughtful and non-defensive comment.
- errordeveloper 11y agoFor example, here is the guide on using Weave for Kubernetes on Azure cloud: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/coreos/azure/README.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...
- kordless 11y agoI'm thinking replies like this should come from an individual, not an account representing a company. Having an individual behind the words (instead of a loose consensus mechanism for a company) goes a long way to establishing trust. Hoping to see a direct address of the observations raised in the post!
- weavenetwork 11y ago
- erni1234 11y agoYes
- justinholmes 11y agoWould have been interesting to compare http://www.projectcalico.org/ http://www.projectcalico.org/.
- grkvlt 11y agoAgree. I may try some benchmarks, as I have recently added support for Calico (as well as Weave, which was my original choice due to its simplicity) as another swappable SDN providers for Clocker [1]. [1] http://clocker.io/ http://clocker.io/
- bascule 11y agoWeave has other issues... like they homebrewed their own ECDHE-PSK-based transport encryption protocol on top of NaCl. Homebrewing your own crypto, especially transport encryption which has to solve problems like key exchange, replay attacks, etc is generally the wrong answer. Also, even if they were using a standard transport encryption like SSL/TLS or IPSEC, PSKs are generally frowned upon for anything other than point-to-point connections. They describe the PSK as a "password", so what they really want is a PAKE algorithm, however they do not use a password hashing function, so weak "passwords" are susceptible to brute force attacks. Anyway, all these things are why you should just stick to standard protocols like SSL/TLS or IPSEC.
- api 11y agoYou speak as if SSL/TLS hasn't been a rat's nest of problems. If they did it competently, there is no reason they couldn't implement their own crypto encapsulation. As far as why they rolled their own... have you ever actually tried to use IPSec? It's a usability nightmare. It's also problematic in containers due to container permission issues. I suppose they could have used DSSL (datagram SSL) but that'd probably add more overhead than what they did. I see little real world evidence that this "let the pros handle it" attitude toward crypto is helping.
- weavenetwork 11y agoTrue. We did try ipsec, and couldn't find an implementation that was oss, demonstrably safe, and easy enough to pull into a first release. As weave matures, we'd love to work with experts to implement standard solutions, even if they are costly to put in place.
- api 11y ago"demonstrably safe" -- this is another issue I have with this crypto conservative FUD. "Use SSL, don't roll your own!" Then we get BEAST, CRIME, Heartbleed, etc., and we discover that the dominant SSL/TLS implementation is a rat's nest of comically awful code: http://opensslrampage.org http://opensslrampage.org Look at the older posts for LuLz like: http://opensslrampage.org/post/83007010531/well-even-if-time-isnt-random-your-rsa http://opensslrampage.org/post/83007010531/well-even-if-time... I wonder just how much scrutiny IPSec implementations have gotten, especially since it's such a usability nightmare that nobody uses it.
- cbsmith 11y agoIn other news, we believe that there isn't much oxygen on the moon...
- aristotle 11y agoThere are ways to make networking easy and performant at the same time without resorting to these user space hacks. Wait for much better products to show up (including one from us).
- otterley 11y agoI'm not sure I even understand the problem that weave and Docker bridging/NAT solves for real world cases. IP allocation for containers isn't a problem for most networks, is it? Certainly AWS can give you up to 8 IPs per instance, and every datacenter I've ever worked in can give you even more, if you ask. All you have to do is spin up additional virtual NICs with virtual MACs and use DHCP to assign IP addresses to them. Or is there something fundamental that I don't understand? Please edify me.
- wmf 11y agoDocker was designed to be very easy to get started on your laptop with one IP address and it looks like some people are getting stuck in that model. I agree that if you are running on AWS VPC or some other overlay you should just use VPC for container networking. You shouldn't overlay your overlay. But there isn't any tooling that I know of to do that.
- otterley 11y agoEveryone I know who runs Docker runs it in a virtual machine manager that has a built-in DHCP server and provide multiple virtual interfaces to the virtual machine. Certainly both VirtualBox and VMWare do. Even if one runs a Docker bridge in his development VM, that doesn't one must do so in production as well. Are we in this mess because production engineers don't understand networking?
- bmurphy1976 11y agoWe're in this mess because cloud providers don't give you all the functionality you need to solve things properly. It's only a matter of time before they do, until then...
- tobbyb 11y agoContainer networking is not special or different from VM networking. There are tons of proven and widely used technologies available in the open source ecosystem. We have been building out a series of networking tutorials at flockport from basics; static, public, private IPs, NAT, bridging etc to multi-host container networking with GRE, L2TP, VxLAN, IPSEC focussed on LXC but these will work with VMs and containers in general. They don't need any special tools, just IP tools and the kernel and deliver performance and security. A lot of the Docker centric networking projects use these under the hood but they are easy enough to use on their own. http://www.flockport.com/news http://www.flockport.com/news
- hobarrera 11y agoLooks like flannel only supports IPv4, while weave would work on IPv4 or IPv6. That's a huge difference IMHO, since one will work on any scenario.
- ploxiln 11y agoIf you're using private address space, you don't need IPv6.