4 ms·

> How is it better have service A request to a proxy, which requests to another proxy, which requests to service B? I get the security benefits of that, but the

by pm90 3y ago

> How is it better have service A request to a proxy, which requests to another proxy, which requests to service B? I get the security benefits of that, but the network architecture is boggling. How many PBs of data are sent each day for what could be a monolithic service? The tradeoff here is to decouple services in order to allow them to be developed somewhat independently of each other. Monoliths remove the overhead of network requests but they present their own challenges. You have a lot of implicit dependencies and feature development becomes complicated with changes having unexpected effects very far from the source. Ultimately engineering organizations need to decide the model that works best for them. Neither is inherently better, they’re solving different problems.

deathanatos 3y ago
> The tradeoff here is to decouple services in order to allow them to be developed somewhat independently of each other. You don't need a service mesh for that, though? Heck, you don't even need an ingress for service to service traffic. ingress-nginx does the job well, without being overly complex, and most importantly to me, logs when something is wrong, which I cannot say the same for Istio which I was fighting earlier this week where it was just happily RST'ing a connection and saying nothing about why it was deciding to do that.
- campbel 3y ago
  There are a lot of good and bad reasons to adopt a mesh. Some of which might relate to your concerns. The things I like most about them, working in infrastructur: 1. I can have a unified set of metrics for all services, regardless of language/platform or how diligent the team is at instrumenting their apps. 2. I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. 3. I can implement automation around canary releases without much lift from dev teams. Other projects leverage these capabilities and do it for you as well. 4. I can get the equivalent of tcpdump for a pod pretty easily which I can use to help app teams debug issues. 5. I can improve app reliability with automatic retries and timeouts. Probably some other things as well... That said, it can be a big increase in complexity to your system the pains of which aren't always distributed to the folks getting the benefits.
  - deathanatos 3y ago
    > 1. I can have a unified set of metrics for all services, regardless of language/platform or how diligent the team is at instrumenting their apps. But they're only going to be coarse metrics, like what requests/second. You're still going to be needing application-specific metrics. > 2. I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. I do like the idea of this feature of service meshes. It is a slog to get teams to do this responsibly. But, like I said: fighting Istio to understand why it was RST-ing a connection, for no apparent reason. Not logging errors is worse. Perhaps the idea is sound, but the implementation leaves one desiring more. I should mention the same Istio service mesh above is a SPoF in the cluster it runs in, on account of being a single pod. I can't tell if the people who set it up were clueless, or if that's the default. I suspect probably the latter. > 3., 4., and 5., as well as actually using mTLS in 2. TBH, these are just benefits I've never been able to realize. I'm stuck slogging through the swamp of service mesh marketing and the people who want to bring the light of their savior the service mesh but without actually getting their hands dirty doing the work of deploying it.
    - gotbeans 3y ago
      Just gonna touch some wood on (1); The fact that you need other metrics does not substract from OPs original point. It's still good and much better overall to handle a set of comprehensive metrics at infra level than to orchestrate every app. About the coarsness, i think it's not really true. Proxies are freaking powerful and they do a lot of stuff at l7, too much in fact (look at envoy, jesus christ). That's one of the reasons why despite the insane complexity of service meshes, they are paramount for observability.
  - KaiserPro 3y ago
    > I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. ok, but what's your threat model for this? great you can tie service versions to each other, but they are just proxies.
  - ongy 3y ago
    1. I agree with the sibling comment, that's usually not the depth I need for application level info. So the entire skill and infrastructure cost for metrics still exists. But it's nice to have as basic data for triage. 2. I always wonder whether that's a timing thing vs. NetworkPolicies and encrypted inter-node traffic. Are the realistic attack scenarios where it's possible to read out intra-cluster traffic but not mess with the cluster, or even read the intra-pod traffic? 3. I've been quite disappointed with how little k8s provides here. I wish it was easier to move traffic off of an old version and only shut the pod down once the last connection was done :/ Maybe I need to look into a service mesh for that? 4. What's the difference to e.g. kubeshark, or just attaching a tcpdump debugcontainer to the pod? Another instance of first to market / potentially nicer ecosystem? 5. I get squeemish with infra-level activities like this. Yes, technically the http method and some headers should make it obvious whether that's save or might break at-least/at-most once or similar semantics. But that requires well behaved applications. While the premise here is infra imposing behaviour to allow applications to be looser around these kind of things.
mdekkers 3y ago
> The tradeoff here is to decouple services in order to allow them to be developed somewhat independently of each other. Monoliths remove the overhead of network requests but they present their own challenges. You have a lot of implicit dependencies and feature development becomes complicated with changes having unexpected effects very far from the source. I have seen, developed, designed, managed, deployed, operated, fondled, and otherwise been around thousands of large systems that are anything between trivial importance to “must always be running, in the national interest”. I was around when SOAs were a hot new thing, and SOAP was being rumoured as the thing that was going to save us from everything. A fondly recall an overpaid Compaq consultant talking about “token passing systems” when they were describing message queues. I have seen exactly two systems that really had to be designed and built along a microservice architecture. Both of these had requirements that introduces a scale, scope, and complexity you simply don’t see very often. All other microservice architectures didn’t solve for requirements, they solved for organisational inefficiencies, misalignments, mismanagement, and - in no small part - ego. When discussing this topic, proponents of microservice proponents talk about many of the advantages these architectures bring, and they are often not completely wrong. What is lacking from these discussions is often a sense of perspective. “Is this solving real problems we have?”, “What is the compound lifecycle cost of this approach?”, and, of course, “How much work is involved in displaying the users’ birthday date in the settings page?”[1], and “when will Omega Star get their fucking shit together?!”[2]. Don’t start with microservices as the default. I will typically work out the monolithic approach as a point of departure. Want microservices? I’m open to that, just demonstrate how that will be better. [1][2] https://m.youtube.com/watch?v=y8OnoxKotPQ https://m.youtube.com/watch?v=y8OnoxKotPQ
- dragonwriter 3y ago
  > All other microservice architectures didn’t solve for requirements, they solved for organisational inefficiencies, misalignments, mismanagement, and - in no small part - ego. Be developed, maintained, and operated by the real organization that exists and not an ideal organization which doesn't is, in fact, usually a practical if not a theoretical requirement, and its usually easier to adapt architecture than to adapt organization.
never_inline 3y ago
Can someone enlighten me on this: if Authorization policies (which pods can communicate with which services) was built in kube-proxy, wouldn't it solve the use case for a large percent of service mesh deployments?
- gotbeans 3y ago
  Kube proxy works configuring ip tables (oversimplified) and its scope is limitted to l4. It doesn't operate at L7.
- adhamsalama 3y ago
  Isn't that what k8s network policies do?
  - never_inline 3y ago
    The support varies by CNI so I wouldn't call it "built-in". But yeah I forgot this existed.
- adhamsalama 3y ago
  Isn't that what k8s network policies do

by pm90 3y ago

deathanatos 3y ago

> The tradeoff here is to decouple services in order to allow them to be developed somewhat independently of each other. You don't need a service mesh for that, though? Heck, you don't even need an ingress for service to service traffic. ingress-nginx does the job well, without being overly complex, and most importantly to me, logs when something is wrong, which I cannot say the same for Istio which I was fighting earlier this week where it was just happily RST'ing a connection and saying nothing about why it was deciding to do that.

campbel 3y ago
There are a lot of good and bad reasons to adopt a mesh. Some of which might relate to your concerns. The things I like most about them, working in infrastructur: 1. I can have a unified set of metrics for all services, regardless of language/platform or how diligent the team is at instrumenting their apps. 2. I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. 3. I can implement automation around canary releases without much lift from dev teams. Other projects leverage these capabilities and do it for you as well. 4. I can get the equivalent of tcpdump for a pod pretty easily which I can use to help app teams debug issues. 5. I can improve app reliability with automatic retries and timeouts. Probably some other things as well... That said, it can be a big increase in complexity to your system the pains of which aren't always distributed to the folks getting the benefits.
- deathanatos 3y ago
  > 1. I can have a unified set of metrics for all services, regardless of language/platform or how diligent the team is at instrumenting their apps. But they're only going to be coarse metrics, like what requests/second. You're still going to be needing application-specific metrics. > 2. I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. I do like the idea of this feature of service meshes. It is a slog to get teams to do this responsibly. But, like I said: fighting Istio to understand why it was RST-ing a connection, for no apparent reason. Not logging errors is worse. Perhaps the idea is sound, but the implementation leaves one desiring more. I should mention the same Istio service mesh above is a SPoF in the cluster it runs in, on account of being a single pod. I can't tell if the people who set it up were clueless, or if that's the default. I suspect probably the latter. > 3., 4., and 5., as well as actually using mTLS in 2. TBH, these are just benefits I've never been able to realize. I'm stuck slogging through the swamp of service mesh marketing and the people who want to bring the light of their savior the service mesh but without actually getting their hands dirty doing the work of deploying it.
  - gotbeans 3y ago
    Just gonna touch some wood on (1); The fact that you need other metrics does not substract from OPs original point. It's still good and much better overall to handle a set of comprehensive metrics at infra level than to orchestrate every app. About the coarsness, i think it's not really true. Proxies are freaking powerful and they do a lot of stuff at l7, too much in fact (look at envoy, jesus christ). That's one of the reasons why despite the insane complexity of service meshes, they are paramount for observability.
- KaiserPro 3y ago
  > I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. ok, but what's your threat model for this? great you can tie service versions to each other, but they are just proxies.
- ongy 3y ago
  1. I agree with the sibling comment, that's usually not the depth I need for application level info. So the entire skill and infrastructure cost for metrics still exists. But it's nice to have as basic data for triage. 2. I always wonder whether that's a timing thing vs. NetworkPolicies and encrypted inter-node traffic. Are the realistic attack scenarios where it's possible to read out intra-cluster traffic but not mess with the cluster, or even read the intra-pod traffic? 3. I've been quite disappointed with how little k8s provides here. I wish it was easier to move traffic off of an old version and only shut the pod down once the last connection was done :/ Maybe I need to look into a service mesh for that? 4. What's the difference to e.g. kubeshark, or just attaching a tcpdump debugcontainer to the pod? Another instance of first to market / potentially nicer ecosystem? 5. I get squeemish with infra-level activities like this. Yes, technically the http method and some headers should make it obvious whether that's save or might break at-least/at-most once or similar semantics. But that requires well behaved applications. While the premise here is infra imposing behaviour to allow applications to be looser around these kind of things.

mdekkers 3y ago

> The tradeoff here is to decouple services in order to allow them to be developed somewhat independently of each other. Monoliths remove the overhead of network requests but they present their own challenges. You have a lot of implicit dependencies and feature development becomes complicated with changes having unexpected effects very far from the source. I have seen, developed, designed, managed, deployed, operated, fondled, and otherwise been around thousands of large systems that are anything between trivial importance to “must always be running, in the national interest”. I was around when SOAs were a hot new thing, and SOAP was being rumoured as the thing that was going to save us from everything. A fondly recall an overpaid Compaq consultant talking about “token passing systems” when they were describing message queues. I have seen exactly two systems that really had to be designed and built along a microservice architecture. Both of these had requirements that introduces a scale, scope, and complexity you simply don’t see very often. All other microservice architectures didn’t solve for requirements, they solved for organisational inefficiencies, misalignments, mismanagement, and - in no small part - ego. When discussing this topic, proponents of microservice proponents talk about many of the advantages these architectures bring, and they are often not completely wrong. What is lacking from these discussions is often a sense of perspective. “Is this solving real problems we have?”, “What is the compound lifecycle cost of this approach?”, and, of course, “How much work is involved in displaying the users’ birthday date in the settings page?”[1], and “when will Omega Star get their fucking shit together?!”[2]. Don’t start with microservices as the default. I will typically work out the monolithic approach as a point of departure. Want microservices? I’m open to that, just demonstrate how that will be better. [1][2] https://m.youtube.com/watch?v=y8OnoxKotPQ https://m.youtube.com/watch?v=y8OnoxKotPQ

dragonwriter 3y ago
> All other microservice architectures didn’t solve for requirements, they solved for organisational inefficiencies, misalignments, mismanagement, and - in no small part - ego. Be developed, maintained, and operated by the real organization that exists and not an ideal organization which doesn't is, in fact, usually a practical if not a theoretical requirement, and its usually easier to adapt architecture than to adapt organization.

never_inline 3y ago

Can someone enlighten me on this: if Authorization policies (which pods can communicate with which services) was built in kube-proxy, wouldn't it solve the use case for a large percent of service mesh deployments?

gotbeans 3y ago
Kube proxy works configuring ip tables (oversimplified) and its scope is limitted to l4. It doesn't operate at L7.
adhamsalama 3y ago
Isn't that what k8s network policies do?
- never_inline 3y ago
  The support varies by CNI so I wouldn't call it "built-in". But yeah I forgot this existed.
adhamsalama 3y ago
Isn't that what k8s network policies do