5 ms·

There are a lot of good and bad reasons to adopt a mesh. Some of which might relate to your concerns. The things I like most about them, working in infrastructu

There are a lot of good and bad reasons to adopt a mesh. Some of which might relate to your concerns. The things I like most about them, working in infrastructur: 1. I can have a unified set of metrics for all services, regardless of language/platform or how diligent the team is at instrumenting their apps. 2. I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. 3. I can implement automation around canary releases without much lift from dev teams. Other projects leverage these capabilities and do it for you as well. 4. I can get the equivalent of tcpdump for a pod pretty easily which I can use to help app teams debug issues. 5. I can improve app reliability with automatic retries and timeouts. Probably some other things as well... That said, it can be a big increase in complexity to your system the pains of which aren't always distributed to the folks getting the benefits.

deathanatos 3y ago
> 1. I can have a unified set of metrics for all services, regardless of language/platform or how diligent the team is at instrumenting their apps. But they're only going to be coarse metrics, like what requests/second. You're still going to be needing application-specific metrics. > 2. I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. I do like the idea of this feature of service meshes. It is a slog to get teams to do this responsibly. But, like I said: fighting Istio to understand why it was RST-ing a connection, for no apparent reason. Not logging errors is worse. Perhaps the idea is sound, but the implementation leaves one desiring more. I should mention the same Istio service mesh above is a SPoF in the cluster it runs in, on account of being a single pod. I can't tell if the people who set it up were clueless, or if that's the default. I suspect probably the latter. > 3., 4., and 5., as well as actually using mTLS in 2. TBH, these are just benefits I've never been able to realize. I'm stuck slogging through the swamp of service mesh marketing and the people who want to bring the light of their savior the service mesh but without actually getting their hands dirty doing the work of deploying it.
- gotbeans 3y ago
  Just gonna touch some wood on (1); The fact that you need other metrics does not substract from OPs original point. It's still good and much better overall to handle a set of comprehensive metrics at infra level than to orchestrate every app. About the coarsness, i think it's not really true. Proxies are freaking powerful and they do a lot of stuff at l7, too much in fact (look at envoy, jesus christ). That's one of the reasons why despite the insane complexity of service meshes, they are paramount for observability.
KaiserPro 3y ago
> I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. ok, but what's your threat model for this? great you can tie service versions to each other, but they are just proxies.
ongy 3y ago
1. I agree with the sibling comment, that's usually not the depth I need for application level info. So the entire skill and infrastructure cost for metrics still exists. But it's nice to have as basic data for triage. 2. I always wonder whether that's a timing thing vs. NetworkPolicies and encrypted inter-node traffic. Are the realistic attack scenarios where it's possible to read out intra-cluster traffic but not mess with the cluster, or even read the intra-pod traffic? 3. I've been quite disappointed with how little k8s provides here. I wish it was easier to move traffic off of an old version and only shut the pod down once the last connection was done :/ Maybe I need to look into a service mesh for that? 4. What's the difference to e.g. kubeshark, or just attaching a tcpdump debugcontainer to the pod? Another instance of first to market / potentially nicer ecosystem? 5. I get squeemish with infra-level activities like this. Yes, technically the http method and some headers should make it obvious whether that's save or might break at-least/at-most once or similar semantics. But that requires well behaved applications. While the premise here is infra imposing behaviour to allow applications to be looser around these kind of things.
- campbel 3y ago
  For #5, I like mesh level retries for a few reasons, but perhaps the biggest is avoiding retry storms by using budgets https://linkerd.io/2.15/tasks/configuring-retries/#retry-budgets https://linkerd.io/2.15/tasks/configuring-retries/#retry-bud...

deathanatos 3y ago

> 1. I can have a unified set of metrics for all services, regardless of language/platform or how diligent the team is at instrumenting their apps. But they're only going to be coarse metrics, like what requests/second. You're still going to be needing application-specific metrics. > 2. I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. I do like the idea of this feature of service meshes. It is a slog to get teams to do this responsibly. But, like I said: fighting Istio to understand why it was RST-ing a connection, for no apparent reason. Not logging errors is worse. Perhaps the idea is sound, but the implementation leaves one desiring more. I should mention the same Istio service mesh above is a SPoF in the cluster it runs in, on account of being a single pod. I can't tell if the people who set it up were clueless, or if that's the default. I suspect probably the latter. > 3., 4., and 5., as well as actually using mTLS in 2. TBH, these are just benefits I've never been able to realize. I'm stuck slogging through the swamp of service mesh marketing and the people who want to bring the light of their savior the service mesh but without actually getting their hands dirty doing the work of deploying it.

gotbeans 3y ago
Just gonna touch some wood on (1); The fact that you need other metrics does not substract from OPs original point. It's still good and much better overall to handle a set of comprehensive metrics at infra level than to orchestrate every app. About the coarsness, i think it's not really true. Proxies are freaking powerful and they do a lot of stuff at l7, too much in fact (look at envoy, jesus christ). That's one of the reasons why despite the insane complexity of service meshes, they are paramount for observability.

KaiserPro 3y ago

> I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. ok, but what's your threat model for this? great you can tie service versions to each other, but they are just proxies.

ongy 3y ago

1. I agree with the sibling comment, that's usually not the depth I need for application level info. So the entire skill and infrastructure cost for metrics still exists. But it's nice to have as basic data for triage. 2. I always wonder whether that's a timing thing vs. NetworkPolicies and encrypted inter-node traffic. Are the realistic attack scenarios where it's possible to read out intra-cluster traffic but not mess with the cluster, or even read the intra-pod traffic? 3. I've been quite disappointed with how little k8s provides here. I wish it was easier to move traffic off of an old version and only shut the pod down once the last connection was done :/ Maybe I need to look into a service mesh for that? 4. What's the difference to e.g. kubeshark, or just attaching a tcpdump debugcontainer to the pod? Another instance of first to market / potentially nicer ecosystem? 5. I get squeemish with infra-level activities like this. Yes, technically the http method and some headers should make it obvious whether that's save or might break at-least/at-most once or similar semantics. But that requires well behaved applications. While the premise here is infra imposing behaviour to allow applications to be looser around these kind of things.

campbel 3y ago
For #5, I like mesh level retries for a few reasons, but perhaps the biggest is avoiding retry storms by using budgets https://linkerd.io/2.15/tasks/configuring-retries/#retry-budgets https://linkerd.io/2.15/tasks/configuring-retries/#retry-bud...