3 ms·

Support for multi-cluster routing is huge. One of the main pain points managing Kubernetes in the large-deployment or cross-cloud use case is how to bridge mult

Support for multi-cluster routing is huge. One of the main pain points managing Kubernetes in the large-deployment or cross-cloud use case is how to bridge multiple clusters. SIG multi-cluster works on related architecture issues/designs full time, and there are lots of other solutions out there, but they're all kind of large and bulky. If one "thing" (Cilium) can give me local-node breakneck speed (via eBPF magic) and cross-cluster (v1.4) and node <-> node routing I would absolutely drop everything and switch to it (my deployment is very very small). All that said, I'm currently very happy with kube-router[0], because it gave me all-in-one node & service/pod routing (as in, you can actually get rid of kube-proxy. Cilium is now way higher on my short list of things to check out now as well. [0]: https://www.kube-router.io/ https://www.kube-router.io/

dexcs 8y ago
Another killer feature is that they can do networkpolicies on fqdns and not only on ips. That's not a new feature in 1.4 but outstanding though.
- hardwaresofton 8y ago
  Whoa I didn't know that, that's huge as well -- There are some overlays that don't even come with NetworkPolicy support at all -- didn't know they supported FQDNs as well
colemickens 8y ago
Cilium also does Service IP routing, ala the kube-router feature you're referring to.
- hardwaresofton 8y ago
  Unless I'm misunderstanding you I'd consider that table stakes -- you don't have much of a overlay network if it can't do at the very least pod<>pod and pod<>svc routing. Thinking aloud, by "Service IP" routing do you mean that actual services can have external facing IPs? I had no idea kube-router supported that, maybe I'm on a version that's too old? [EDIT] Looking back at the kube-router docs, I see no mention of what I was discussing above so I'm going to assume "Service IP" means the usual IPVS/LVS stuff that kube-router does. I find that feature doesn't really affect me directly as a cluster operator though... Though it's probably saving me some latency (I assume you can skip a DNS resolution or two?)
  - colemickens 8y ago
    Sorry, I meant "Service ClusterIP", which a common name of the feature you're talking about that provides VIPs and load balancing for internal services. Anyway, I was just pointing out it is also available via Cilium. To put it another way, Cilium can replace kube-proxy, just as kube-router does. As well as doing all of the other mesh networking functionality that k8s needs. Put a final way, I still don't know what kube-router does or roles it can take, that Cilium can't also do or provide. But I'm not deep on them and am mostly replying here in the hopes that you will fill in a missing gap in my knowledge. :). Thanks!
lima 8y ago
How does kube-router compare to Project Calico?
- dexcs 8y ago
  Calico or Cilium? ;) Compared to Cilium I would spontaneously say it's bpf and fqdn based network policies.
- hardwaresofton 8y ago
  On what axis? kube-router is closer to Canal (Calico + flannel), and adds NetworkPolicy support, and the ability to remove kube-proxy completely from your setup.

dexcs 8y ago

Another killer feature is that they can do networkpolicies on fqdns and not only on ips. That's not a new feature in 1.4 but outstanding though.

hardwaresofton 8y ago
Whoa I didn't know that, that's huge as well -- There are some overlays that don't even come with NetworkPolicy support at all -- didn't know they supported FQDNs as well

colemickens 8y ago

Cilium also does Service IP routing, ala the kube-router feature you're referring to.

hardwaresofton 8y ago
Unless I'm misunderstanding you I'd consider that table stakes -- you don't have much of a overlay network if it can't do at the very least pod<>pod and pod<>svc routing. Thinking aloud, by "Service IP" routing do you mean that actual services can have external facing IPs? I had no idea kube-router supported that, maybe I'm on a version that's too old? [EDIT] Looking back at the kube-router docs, I see no mention of what I was discussing above so I'm going to assume "Service IP" means the usual IPVS/LVS stuff that kube-router does. I find that feature doesn't really affect me directly as a cluster operator though... Though it's probably saving me some latency (I assume you can skip a DNS resolution or two?)
- colemickens 8y ago
  Sorry, I meant "Service ClusterIP", which a common name of the feature you're talking about that provides VIPs and load balancing for internal services. Anyway, I was just pointing out it is also available via Cilium. To put it another way, Cilium can replace kube-proxy, just as kube-router does. As well as doing all of the other mesh networking functionality that k8s needs. Put a final way, I still don't know what kube-router does or roles it can take, that Cilium can't also do or provide. But I'm not deep on them and am mostly replying here in the hopes that you will fill in a missing gap in my knowledge. :). Thanks!

lima 8y ago

How does kube-router compare to Project Calico?

dexcs 8y ago
Calico or Cilium? ;) Compared to Cilium I would spontaneously say it's bpf and fqdn based network policies.
hardwaresofton 8y ago
On what axis? kube-router is closer to Canal (Calico + flannel), and adds NetworkPolicy support, and the ability to remove kube-proxy completely from your setup.