4 ms·

A Service Mesh for Kubernetes, Part I: Top-Line Service Metrics

71 points by aberoham 10y ago

whalesalad 10y ago
Still trying to get the laymans explanation of linkerd. Is it a drop-in replacement for making service-to-service HTTP calls? AFAIK it's a standalone proxy-like version of Finagle.
- olix0r 10y ago
  That's basically right. linkerd is a proxy that deals with a bunch of operational things like service discovery, metrics/observability, tracing, load balancing, etc. It's a proxy so that you can deploy one solution across all languages/frameworks without having to modify your code to deal with these things.
- skMed 10y ago
  Right, if I want to rely on Finagle's RPC capabilities, e.g. load balancing, routing, tracing, etc, then my microservice must be JVM based and use the Finagle library. This is a standalone proxy deployed as a 'sidecar' app alongside my service. This allows me to implement the microservice in any language I want and get some of the features listed previously. Of course you now have another moving part in your system :)
justinsaccount 10y ago
See also: Envoy https://lyft.github.io/envoy/ https://lyft.github.io/envoy/ https://news.ycombinator.com/item?id=12497926 https://news.ycombinator.com/item?id=12497926
tamalsaha001 10y ago
I thought Kubernetes itself does service discovery and load balancing. Why would I want another layer/proxy to the involved?
- williamallthing 10y ago
  Request-level versus connection-level load balancing. See e.g. https://blog.buoyant.io/2016/03/16/beyond-round-robin-load-balancing-for-latency/ https://blog.buoyant.io/2016/03/16/beyond-round-robin-load-b...
- deleted 10y ago
  [deleted]
- theptip 10y ago
  kube-proxy (the k8s per-node service agent) is a good generic solution, but has some issues. The primary issue is that it doesn't support advanced use-cases like circuit breaking or retry budgets (it's just a set of iptables rules). Also, I believe linkerd uses IPC, which wil be quicker for intra-node communication between processes[1] (i.e. east-west communications). Not sure how linkerd does it, but it's possible to bypass the kube-proxy and directly watch the set of pod IPs that a service is selecting, so linkerd can avoid hitting the default kube-proxy if it wants to. [1]: http://bhavin.directi.com/unix-domain-sockets-vs-tcp-sockets/ http://bhavin.directi.com/unix-domain-sockets-vs-tcp-sockets...
lobster_johnson 10y ago
I'm curious how managable this in practice. On the one hand, Linkerd brings some real technical solutions to the table that Kubernetes can't solve right now — K8s' L4 load-balancing can only load-balance entire connections, not requests. For that reason, K8s can't do anything to assist you with timeouts, retries, rate-limiting, tracing or metrics collection. I like the idea of a semi-transparent middleman that handles these things for you and lets your applications be small and simple. On the other, this setup replaces a core K8s concept with its own, less proven components. As I understand it, Linkerd's daemon replaces service lookup entirely; it will talk to the API server to get service endpoints, which means it needs to invent its own logic to keep up to date with changing pod configurations. It had better be solid stuff! To reduce surface area, I wonder if running one Linkerd per pod in a sidecar container is a viable option, or would that be crazy? As an aside, can anyone explain why the example needs to start "kubectl proxy" in a sidecar container? Kubernetes already has a perfectly good way for apps to talk to the API server through a service account [1]. Seems iffy to me. [1] http://kubernetes.io/docs/user-guide/accessing-the-cluster/#accessing-the-api-from-a-pod http://kubernetes.io/docs/user-guide/accessing-the-cluster/#...
- adlleong 10y ago
  Running one linkerd per pod as a sidecar is definitely not crazy. You can see an example of this configuration here: https://linkerd.io/getting-started/k8s/ https://linkerd.io/getting-started/k8s/ Using kubectl proxy takes care of locating and sending the service account token and certificates. This is described a bit farther down in the Kubernetes doc that you linked.
- williamallthing 10y ago
  Great point about the need for production worthiness. FWIW the load balancing pool management, etc., in linkerd is straight from Finagle, which is probably more proven than Kubernetes, if anything (powers Twitter, Pinterest, SoundCloud, etc.) Some K8s-specific bits are fresher code, but even those have been tested at non-trivial scale, e.g. https://monzo.com/blog/2016/09/19/building-a-modern-bank-backend/ https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac...
- 10y ago

71 points by aberoham 10y ago

Still trying to get the laymans explanation of linkerd. Is it a drop-in replacement for making service-to-service HTTP calls? AFAIK it's a standalone proxy-like version of Finagle.

olix0r 10y ago
That's basically right. linkerd is a proxy that deals with a bunch of operational things like service discovery, metrics/observability, tracing, load balancing, etc. It's a proxy so that you can deploy one solution across all languages/frameworks without having to modify your code to deal with these things.
skMed 10y ago
Right, if I want to rely on Finagle's RPC capabilities, e.g. load balancing, routing, tracing, etc, then my microservice must be JVM based and use the Finagle library. This is a standalone proxy deployed as a 'sidecar' app alongside my service. This allows me to implement the microservice in any language I want and get some of the features listed previously. Of course you now have another moving part in your system :)

justinsaccount 10y ago

See also: Envoy https://lyft.github.io/envoy/ https://lyft.github.io/envoy/ https://news.ycombinator.com/item?id=12497926 https://news.ycombinator.com/item?id=12497926

tamalsaha001 10y ago

I thought Kubernetes itself does service discovery and load balancing. Why would I want another layer/proxy to the involved?

williamallthing 10y ago
Request-level versus connection-level load balancing. See e.g. https://blog.buoyant.io/2016/03/16/beyond-round-robin-load-balancing-for-latency/ https://blog.buoyant.io/2016/03/16/beyond-round-robin-load-b...
deleted 10y ago
[deleted]
theptip 10y ago
kube-proxy (the k8s per-node service agent) is a good generic solution, but has some issues. The primary issue is that it doesn't support advanced use-cases like circuit breaking or retry budgets (it's just a set of iptables rules). Also, I believe linkerd uses IPC, which wil be quicker for intra-node communication between processes[1] (i.e. east-west communications). Not sure how linkerd does it, but it's possible to bypass the kube-proxy and directly watch the set of pod IPs that a service is selecting, so linkerd can avoid hitting the default kube-proxy if it wants to. [1]: http://bhavin.directi.com/unix-domain-sockets-vs-tcp-sockets/ http://bhavin.directi.com/unix-domain-sockets-vs-tcp-sockets...

lobster_johnson 10y ago

I'm curious how managable this in practice. On the one hand, Linkerd brings some real technical solutions to the table that Kubernetes can't solve right now — K8s' L4 load-balancing can only load-balance entire connections, not requests. For that reason, K8s can't do anything to assist you with timeouts, retries, rate-limiting, tracing or metrics collection. I like the idea of a semi-transparent middleman that handles these things for you and lets your applications be small and simple. On the other, this setup replaces a core K8s concept with its own, less proven components. As I understand it, Linkerd's daemon replaces service lookup entirely; it will talk to the API server to get service endpoints, which means it needs to invent its own logic to keep up to date with changing pod configurations. It had better be solid stuff! To reduce surface area, I wonder if running one Linkerd per pod in a sidecar container is a viable option, or would that be crazy? As an aside, can anyone explain why the example needs to start "kubectl proxy" in a sidecar container? Kubernetes already has a perfectly good way for apps to talk to the API server through a service account [1]. Seems iffy to me. [1] http://kubernetes.io/docs/user-guide/accessing-the-cluster/#accessing-the-api-from-a-pod http://kubernetes.io/docs/user-guide/accessing-the-cluster/#...

adlleong 10y ago
Running one linkerd per pod as a sidecar is definitely not crazy. You can see an example of this configuration here: https://linkerd.io/getting-started/k8s/ https://linkerd.io/getting-started/k8s/ Using kubectl proxy takes care of locating and sending the service account token and certificates. This is described a bit farther down in the Kubernetes doc that you linked.
williamallthing 10y ago
Great point about the need for production worthiness. FWIW the load balancing pool management, etc., in linkerd is straight from Finagle, which is probably more proven than Kubernetes, if anything (powers Twitter, Pinterest, SoundCloud, etc.) Some K8s-specific bits are fresher code, but even those have been tested at non-trivial scale, e.g. https://monzo.com/blog/2016/09/19/building-a-modern-bank-backend/ https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac...
10y ago