4 ms·

> When you start using Kubernetes one of the first suggestions you'll get is to install a service mesh. Who on earth is giving you this advice? Service-message

> When you start using Kubernetes one of the first suggestions you'll get is to install a service mesh. Who on earth is giving you this advice? Service-messages are squarely in the “you’ll know when you need it, and it’s not day 1” bucket. Is this the experience people are having with K8s? Welcome to Kubernetes, here’s a service mesh gl;hf. If this advice is common, no wonder some people think K8s is over complicated. To anyone not aware: it _does not_ have to be this complicated. Default ingress controller and a normal Deployment will carry you really, really far.

politelemon 3y ago
To answer your question, yes this is the majority of the experience and is almost a given decision in the initial setup, in part because of the security association. Especially in enterprises it's easier to just say service mesh than have to think the decision through. Regardless of service mesh though, although it's a part of it, the complicated reputation is not undeserved and not solely due to meshes.
sofixa 3y ago
> If this advice is common, no wonder some people think K8s is over complicated I mean, if even Google who were behind it in the first place, are saying it's complicated and have 3 levels of managed Kubernetes services, I'd say it's pretty clear to everyone it is indeed complicated. Some of the complexity is simply due to the complex problems it solves, some is footguns, some is layers upon layers of abstractions to glue around design deficiencies. > To anyone not aware: it _does not_ have to be this complicated. Default ingress controller and a normal Deployment will carry you really, really far You're absolutely right. However, Kubernetes is rarely chosen after a careful evaluation of requirements; it's more often than not because it's considered necessary or for resume driven development. In that case, service mesh is easy to tack on, regardless of need.
vbezhenar 3y ago
I have small cluster and the only reason I’d want service mesh is to inspect req/resp between services for better debug. And I’m not even sure they can do that. Retries for free are good I guess, but not something essential. MTLS is something I don’t want at all.
- nullify88 3y ago
  Couldn't you accomplish that with tracing? You may not even need any changes in your application as eBPF maybe able to automatically instrument the application. Alternatively, If you use Cilium as a CNI, this functionality can come out the box. https://docs.cilium.io/en/latest/observability/visibility/ https://docs.cilium.io/en/latest/observability/visibility/
- jakupovic 3y ago
  Exactly if you don't need the functionality provided don't use a mesh, but consequently if you then use it. Honestly not very complicated, but mTLS is very complicated done correctly.
moondev 3y ago
I'm sad this is the top comment on an incredibly detailed and well researched article. The context of the content is obviously for Kubernetes enterprise platform teams and not deploying your first nginx pod. The author also gives specific examples of why a service mesh is useful and in what scenarios it shines.
- YetAnotherNick 3y ago
  > The author also gives specific examples of why a service mesh is useful and in what scenarios it shines. The author's entire point was money which is weird as it is peanuts compared to the DevOps salary of people needed to work with this complex setup. Unless you are in a startup and good in tech, in which case service mesh doesn't make a lot of sense. Why do you need encryption or metrics collection above ingress. Nginx does a good enough job for metrics.
  - darkwater 3y ago
    It might come as a shocker for you but many medium-sized orgs that prefer to pay salaries for people who can do N things and just augment their capabilities with some Salas/paid service. But if the service is too costly, they will not allocate budget for it. Also, unlike many Salas/software vendors would try to make you believe, those solutions usually need tweaking & knowledge to be operated. And if any issues happen, swift support comes at a hefty premium. Also, not all salaries are SV-level, while these softwares usually have the same cost wherever you are in the world.
    - YetAnotherNick 3y ago
      > It might come as a shocker for you Thanks for the unneeded tone. But the solution that author likely uses is linkerd which is free for 50 users. And even then very likely it would be less than $1000/month. If it is too much for the org to pay, I don't think they need it. Service mesh is not something a small or medium sized company needs. > those solutions usually need tweaking & knowledge to be operated. And if any issues happen, swift support comes at a hefty premium. Exactly my point. Base price is not something that is gonna cost them the most when using advanced tools like this. Even if it was free it would costed pretty much the same for any medium sized org.
out-of-ideas 3y ago
its yet another tradeoff, it adds more security for the apps too, now your apps dont need to have tls stuff, you let the sidecar envoy-proxy handle that ect
- FridgeSeal 3y ago
  Yeah I’m aware of how useful they can be, my comment was about for someone first introduction to K8s, they’re probably overkill.
  - out-of-ideas 3y ago
    yes a bit overkill for first use; deff want to tinker around with it without the extra moving parts at first; but i'd say if somebody is venturing into that space they should also consider security as well and should certainly try to get the added layers. a lot of companies miss out on security and implement it as an afterthought and run into more pain for everybody involved
ixaxaar 3y ago
K8s, service meshes, all this infrastructure complexity to me seems like more of a band-aid to bad backend practices and lack of standardization. This kind of world arises if an org has, say 1000 engineers, or ~100-200 teams, and every team wants to do their own thing with accountability improperly mapped etc. And infrastructure engineers turn into police with an obvious overemphasis on observability. I guess a large amount of complexity in infra and wastage is done to hide human inefficiencies.
- FridgeSeal 3y ago
  The flip side is that some of this standardisation lets smaller teams very easily get this functionality without the man-hours required to implement it in your application layer. < Usual caveats apply, don’t bother until you need it, etc etc. Personally though, I’ve found dropping in linkerd, or Cilium CNI about an afternoon’s work, with no application code changes necessary>
- jakupovic 3y ago
  Or there is a company mandate to encrypt everything due to the customer and legal needs. Then instead of rolling your own solutions you use an established mesh and move to other problems. The complexity is not there but your lack of understanding why someone may need a mesh. The last sentence is because you pre-judge the need and the platform it needs. Yes I do this for living snd know what I'm talking about.
Already__Taken 3y ago
and default gateway API is going to let your ops people help you go far without adding a bunch of coordination over ingress.
dvfjsdhgfv 3y ago
If you don't hear this advice, you are simply not in the audience that routinely uses service meshes. They are quite useful in enterprise environments.
deleted 3y ago
[deleted]

politelemon 3y ago

To answer your question, yes this is the majority of the experience and is almost a given decision in the initial setup, in part because of the security association. Especially in enterprises it's easier to just say service mesh than have to think the decision through. Regardless of service mesh though, although it's a part of it, the complicated reputation is not undeserved and not solely due to meshes.

sofixa 3y ago

> If this advice is common, no wonder some people think K8s is over complicated I mean, if even Google who were behind it in the first place, are saying it's complicated and have 3 levels of managed Kubernetes services, I'd say it's pretty clear to everyone it is indeed complicated. Some of the complexity is simply due to the complex problems it solves, some is footguns, some is layers upon layers of abstractions to glue around design deficiencies. > To anyone not aware: it _does not_ have to be this complicated. Default ingress controller and a normal Deployment will carry you really, really far You're absolutely right. However, Kubernetes is rarely chosen after a careful evaluation of requirements; it's more often than not because it's considered necessary or for resume driven development. In that case, service mesh is easy to tack on, regardless of need.

vbezhenar 3y ago

I have small cluster and the only reason I’d want service mesh is to inspect req/resp between services for better debug. And I’m not even sure they can do that. Retries for free are good I guess, but not something essential. MTLS is something I don’t want at all.

nullify88 3y ago
Couldn't you accomplish that with tracing? You may not even need any changes in your application as eBPF maybe able to automatically instrument the application. Alternatively, If you use Cilium as a CNI, this functionality can come out the box. https://docs.cilium.io/en/latest/observability/visibility/ https://docs.cilium.io/en/latest/observability/visibility/
jakupovic 3y ago
Exactly if you don't need the functionality provided don't use a mesh, but consequently if you then use it. Honestly not very complicated, but mTLS is very complicated done correctly.

moondev 3y ago

I'm sad this is the top comment on an incredibly detailed and well researched article. The context of the content is obviously for Kubernetes enterprise platform teams and not deploying your first nginx pod. The author also gives specific examples of why a service mesh is useful and in what scenarios it shines.

YetAnotherNick 3y ago
> The author also gives specific examples of why a service mesh is useful and in what scenarios it shines. The author's entire point was money which is weird as it is peanuts compared to the DevOps salary of people needed to work with this complex setup. Unless you are in a startup and good in tech, in which case service mesh doesn't make a lot of sense. Why do you need encryption or metrics collection above ingress. Nginx does a good enough job for metrics.
- darkwater 3y ago
  It might come as a shocker for you but many medium-sized orgs that prefer to pay salaries for people who can do N things and just augment their capabilities with some Salas/paid service. But if the service is too costly, they will not allocate budget for it. Also, unlike many Salas/software vendors would try to make you believe, those solutions usually need tweaking & knowledge to be operated. And if any issues happen, swift support comes at a hefty premium. Also, not all salaries are SV-level, while these softwares usually have the same cost wherever you are in the world.
  - YetAnotherNick 3y ago
    > It might come as a shocker for you Thanks for the unneeded tone. But the solution that author likely uses is linkerd which is free for 50 users. And even then very likely it would be less than $1000/month. If it is too much for the org to pay, I don't think they need it. Service mesh is not something a small or medium sized company needs. > those solutions usually need tweaking & knowledge to be operated. And if any issues happen, swift support comes at a hefty premium. Exactly my point. Base price is not something that is gonna cost them the most when using advanced tools like this. Even if it was free it would costed pretty much the same for any medium sized org.

out-of-ideas 3y ago

its yet another tradeoff, it adds more security for the apps too, now your apps dont need to have tls stuff, you let the sidecar envoy-proxy handle that ect

FridgeSeal 3y ago
Yeah I’m aware of how useful they can be, my comment was about for someone first introduction to K8s, they’re probably overkill.
- out-of-ideas 3y ago
  yes a bit overkill for first use; deff want to tinker around with it without the extra moving parts at first; but i'd say if somebody is venturing into that space they should also consider security as well and should certainly try to get the added layers. a lot of companies miss out on security and implement it as an afterthought and run into more pain for everybody involved

ixaxaar 3y ago

K8s, service meshes, all this infrastructure complexity to me seems like more of a band-aid to bad backend practices and lack of standardization. This kind of world arises if an org has, say 1000 engineers, or ~100-200 teams, and every team wants to do their own thing with accountability improperly mapped etc. And infrastructure engineers turn into police with an obvious overemphasis on observability. I guess a large amount of complexity in infra and wastage is done to hide human inefficiencies.

FridgeSeal 3y ago
The flip side is that some of this standardisation lets smaller teams very easily get this functionality without the man-hours required to implement it in your application layer. < Usual caveats apply, don’t bother until you need it, etc etc. Personally though, I’ve found dropping in linkerd, or Cilium CNI about an afternoon’s work, with no application code changes necessary>
jakupovic 3y ago
Or there is a company mandate to encrypt everything due to the customer and legal needs. Then instead of rolling your own solutions you use an established mesh and move to other problems. The complexity is not there but your lack of understanding why someone may need a mesh. The last sentence is because you pre-judge the need and the platform it needs. Yes I do this for living snd know what I'm talking about.

Already__Taken 3y ago

and default gateway API is going to let your ops people help you go far without adding a bunch of coordination over ingress.

dvfjsdhgfv 3y ago

If you don't hear this advice, you are simply not in the audience that routinely uses service meshes. They are quite useful in enterprise environments.

deleted 3y ago

[deleted]