42 ms·

K8s Service Meshes: The Bill Comes Due

233 points by zdw 3y ago

phildenhoff 3y ago
good lord is this what modern microservices are like? How is it better have service A request to a proxy, which requests to another proxy, which requests to service B? I get the security benefits of that, but the network architecture is boggling. How many PBs of data are sent each day for what could be a monolithic service? Actually, to the end — companies that embrace microservices, which see the value in them. How do they manage network traffic for these kinds of K8s setups at scale? Surely they’re not using REST and HTTP. Is it as simple as protobufs over HTTP? Quic? Something else? Edit: lots of great discussion below but I really meant how do they manage network TRAFFIC, not microservices in general :)
- throwup238 3y ago
  > How do they manage these kinds of K8s setups at scale? Our DevOps team starts off the monthly all hands meeting by leading a ritual during which they ceremoniously sacrifice an animal from the Fish and Wildlife Service's Threatened & Endangered Species list while the rest of the company chants: Exorcizamus te, omnis immundus spiritus omnis satanica potestas, omnis incursio infernalis adversarii, omnis legio, omnis congregatio et secta diabolica.
  - zubairq 3y ago
    haha, IT staff doing mystic chants reminds me of this video I made over a decade ago: https://www.youtube.com/watch?v=8Sj3_NfDYeU https://www.youtube.com/watch?v=8Sj3_NfDYeU
  - jacinda 3y ago
    I needed this laugh so much. Thank you.
  - vaporary 3y ago
    "Getting a SCSI chain working is perfectly simple if you remember that there must be exactly three terminations: one on one end of the cable, one on the far end, and the goat, terminated over the SCSI chain with a silver-handled knife whilst burning *black* candles." -- Anthony DeBoer "SCSI is *not* magic. There are *fundamental* *technical* *reasons* why you have to sacrifice a young goat to your SCSI chain every now and then." -- John F. Woods
    - jakupovic 3y ago
      I LOL-ed at this because I used to cry when no goats were around.
- spockz 3y ago
  The data over the wire is the effect of having micro services and has little to do with service meshes. If anything, a service mesh could help by transparently enabling compression or upgrading to h3 without having to bake that in every app. Also, from a networking perspective, the proxies are hosted next to the instances so there is no difference there. We run 1800 services in our mesh. Most of them rest/http, a select few gRPC and graphql. We even have some soap/http services in it.
  - slimsag 3y ago
    with 1800 services, do you feel like you are programming/architecting code in the same way a complex monolithic codebase might, working across them all? Or are the services just cogs in the machine, managed by individual devs / cogs in the machine? I imagine the latter and presume that's the main benefit of so many services, but genuinely curious as I've never experience that many services in an architecture
    - bostik 3y ago
      I would say it's right about the middle of those two extremes. A well functioning service mesh (or even just a well maintained and discoverable ingress controller) is essentially invisible to the individual dev teams. Just think how modern stacks work from a frontend dev's perspective: team wants to use an additional feature, so they find the budgeted credit card, sign up to a random third-party provider, get their access token, and go. From the codebase standpoint, they merely added a new roundtrip to a random service and process the responses in their code. From the dev team's point of view, having the same feature available internally, behind "just another URL", makes no big difference. Maybe less politics around vendor spend and, with luck, easier integration with the remote service auth. Almost certainly less wrangling with compliance and legal. Whether that URL is provided by an ingress with a proper FQDN, or a service mesh entry with otherwise unresolvable name, is (and should be) irrelevant. Modern distributed systems have long since become too large for any single person to fully comprehend them through and through. There is no Grand Design[tm], they are all results of organic changes and evolution. Service discovery and routing can be architected. Individual services within the system can be architected. The complete system where hundreds or even thousands of services interact can not.
- de6u99er 3y ago
  The trick is to know which code you want to maintain yourself and which stuff you want to get off the shelf. Going the off the shelf way, also means being stateless and using distributed transactions with all their pitfalls (eventual consistency, fire and forget, ...). Good architects will know how and when to do what E.g. start with a modulith instead of a monolith, to ease refactoring into micro-services once user count goes through the roof and vertical scaling won't do it any more.
- __turbobrew__ 3y ago
  > How do they manage these kinds of K8s setups at scale? You check in configs into the monorepo and there is tooling to continuously sync the configs with the actual state of the infrastructure. The advantage of microservices at scale is that team X breaking the build doesn’t affect team Y. This scale is probably not until you have 1000+ engineers however.
  - pclmulqdq 3y ago
    To be clear, that is the advantage of services. Microservices are a culture of taking that isolation as far as you can.
  - jakupovic 3y ago
    You do deployments as one used to do with big C++ programs also, you can delete everything and deploy again to make sure everything is new. The latter is preferred. Doing continuous deployments is very hard and not sure possible, at least I haven't seen it work well even though it's touted as fundamental k8s feature. K8s groups resources and presents a platform, how you use it it's up to you, and the management.
- pm90 3y ago
  > How is it better have service A request to a proxy, which requests to another proxy, which requests to service B? I get the security benefits of that, but the network architecture is boggling. How many PBs of data are sent each day for what could be a monolithic service? The tradeoff here is to decouple services in order to allow them to be developed somewhat independently of each other. Monoliths remove the overhead of network requests but they present their own challenges. You have a lot of implicit dependencies and feature development becomes complicated with changes having unexpected effects very far from the source. Ultimately engineering organizations need to decide the model that works best for them. Neither is inherently better, they’re solving different problems.
  - deathanatos 3y ago
    > The tradeoff here is to decouple services in order to allow them to be developed somewhat independently of each other. You don't need a service mesh for that, though? Heck, you don't even need an ingress for service to service traffic. ingress-nginx does the job well, without being overly complex, and most importantly to me, logs when something is wrong, which I cannot say the same for Istio which I was fighting earlier this week where it was just happily RST'ing a connection and saying nothing about why it was deciding to do that.
    - campbel 3y ago
      There are a lot of good and bad reasons to adopt a mesh. Some of which might relate to your concerns. The things I like most about them, working in infrastructur: 1. I can have a unified set of metrics for all services, regardless of language/platform or how diligent the team is at instrumenting their apps. 2. I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. 3. I can implement automation around canary releases without much lift from dev teams. Other projects leverage these capabilities and do it for you as well. 4. I can get the equivalent of tcpdump for a pod pretty easily which I can use to help app teams debug issues. 5. I can improve app reliability with automatic retries and timeouts. Probably some other things as well... That said, it can be a big increase in complexity to your system the pains of which aren't always distributed to the folks getting the benefits.
- paulddraper 3y ago
  > A request to a proxy, which requests to another proxy, which requests to service B? I hesitate to tell you how many layers are between me an HN's servers right now.
  - phildenhoff 3y ago
    Right but there are all those layers so that HNs one server can service your request and send a response. How much traffic would be generated if, for every request to HN, six other requests fire? And every time you comment a cascade of requests fire in HNs imaginary K8s cluster? It’s not that there is anything wrong with this, or that the tradeoff isn’t worth it… it’s just so much data flying back and forth over the wire.
    - aetimmes 3y ago
      Ultimately, in the age of massive cloud compute, the constraining resource in an organization is engineering hours, not CPU/memory/bandwidth. And even then, I've yet to encounter a system (outside of massively parallel MPI-based HPC) where saturating network pipes became the bottleneck for a system before CPU/memory utilization. Microservice architecture means there's lot of data flying around, but it keeps local resource utilization predictable.
      - com 3y ago
        I thought that the constraining resource was money in all but the top 1000 or so businesses in the world with effectively 99%+ margins, mostly because they control their markets absolutely and can raise prices or change demand like the big three algorithmic advertising companies. Everybody else in the cloud goes to the wall without very careful cost control, which is by no means automated or low cost, either.
    - eptcyka 3y ago
      Most often, the wire is virtual, its just copying bytes between processes.
    - paulddraper 3y ago
- patcon 3y ago
  Heh wait until you hear how the cell signalling and any biological process in your body works :) It's an absolute clusterfuck of dependencies and side-effects. It's like space bar heating functionality[1] all the way down...! In my thinking, we might as well get used to the levels of indirection and complexity that AIs will be comfortable with. I suspect it will be more akin to what biological computation is "comfortable" with, and less like what our minds happen to prefer But I digress :) yes, microservices strike me as wiiild [1]: https://xkcd.com/1172/ https://xkcd.com/1172/
- 15457345234 3y ago
  You're asking some really good questions here I find that microservices and this type of architecture have become a religion - you do it this way because you do it this way. You add another layer of complication because that's what you do now. You add this product because that's what you do now. Now you do it this way. Now you stop doing this thing and do this thing instead. It's all proclamations and a truly insane level of complexity and often a truly stunningly low level of performance achieved from some very powerful hardware because everything is behind at least twenty layers of abstraction and you're like, encrypting traffic which is just being passed between VMs which are on the same hardware, but because you can't guarantee that they're always on the same hardware you have to encrypt and use a proxy and... oh wow Watching it from the outside is a bit exhausting, it just seems to be so much churn and overhead.
- deleted 3y ago
  [deleted]
- threeseed 3y ago
  > for what could be a monolithic service On larger projects: a) It is sometimes impossible to run the entire application on a laptop. And so having a micro-service means you can quickly iterate and test before embedding it into the wider system. b) You will commonly run into conflicting transitive dependencies which you simply can't work around. Classic example being Spark on the JVM which brings in years old Hadoop libraries. c) The inter-relationships between component can become so complex that you really want to be able to use canary or green/blue deployment techniques to reduce risk.
- deathanatos 3y ago
  > Edit: lots of great discussion below but I really meant how do they manage network TRAFFIC, not microservices in general :) I don't really understand the question you're asking, but I think maybe the answer is that network pipes are just bigger than the scale most people are operating at? I don't think anything I've ever done has really had that many qps, and if it has, it is more likely to raise an eyebrow that says "who's spamming requests" more that "I guess we've made it to the big time". REST & protobufs are orthogonal. Empirically, literally nobody is doing REST, and most things are just ad hoc, poorly to not-at-all defined JSON/HTTP with a few HTTP verbs sprinkled in to make everyone feel good. It could be protobuf, too, if you like, but unless you have some truly gargantuan JSON, it really won't matter in the end. Compression will make up enough of the difference in size on the network. Some languages don't have to allocate the keys a billion times, too, though even in Python, it's a while before it starts to hurt. What I see more of is processes just inexplicably using gigabytes upon gigabytes of RAM, burning through whole years of CPU time for no particular reason before just dropping back to nominal levels like nothing happened, and dev teams that can't coherently understand the disconnect between just how much power a modern machine has, and what their design doc says their process is supposed to do (hint: something that shouldn't take that many resources). I like microservices, but there should be strong areas of responsibility to them. For most companies, I think that's ~2–3 services. At my current company, it's ~2 services + a database, with the rest being things like cron jobs or really small services that are just various glue or infra tooling.
- foota 3y ago
  Traffic within a data center is relatively cheap, is it not? Of course, I think managed network proxies tend to be expensive, but I think that mostly comes from the machine costs, not the network itself. You pay a bit of a tax for network serialization, but it's unlikely to be the bottleneck in most things ime. Plus, applications built on an external database (e.g., postgres not sqlite) will already have 1 hop. Going from 1 to 2 hops is less dramatic than going from no hops to 1. And I guess implicitly any service sending lots of data to the client already has 1 hop. Imo the only case where a network proxy would be egregious is an in memory database kind of workload where the response size is small relative to the size of the data accessed (maybe something like a custom analytics engine), but that's pretty niche.
  - ozr 3y ago
    > Traffic within a data center is relatively cheap, is it not? Complexity is very, very expensive.
  - manfre 3y ago
    > Traffic within a data center is relatively cheap, Most organizations will run their services spread across multiple data centers / AZs.
    - foota 3y ago
      Hm, wouldn't you want to set it up to route traffic preferentially to a nearby location?
- lukeschlather 3y ago
  I really hate this pattern in principle, but it isn't as bad as you're making it out to be. Most of the service meshes operate as sidecars, which is to say that if service A is calling service B through a mesh, there are two proxies in between the services, but proxy A is on the same machine as A, and proxy B is on the same machine as proxy B. So it is kind of offensive to have 3 network hops involved, but actually only one of those is over the wire, and in most cases I don't think the proxies are actually causing any measurable latency. (At least, whatever latency it's causing is smaller than the latency caused by the TLS encryption, which is necessary and the whole point.)
  - deleted 3y ago
    [deleted]
  - phildenhoff 3y ago
    That makes sense! I didn't realise the proxies were sidecars, but that seems obvious in retrospect.
- dilyevsky 3y ago
  They manage traffic with “service meshes” which includes anything from relatively” basic” distributed iptables/ipvs VIPs (kube-proxy) + dns to full-blown intercepting proxies a la istio/linkerd
- geraldwhen 3y ago
  It’s all http with hundreds of milliseconds of latency per request.
- solatic 3y ago
  > How many PBs of data are sent each day for what could be a monolithic service? The companies that need service meshes couldn't possibly run everything in a single monolith. They already have several if not dozens of monoliths, each one typically coming into the architecture when a large enterprise acquires another company and its monolith, or simply different business units / product lines that don't talk to each other because of sheer organizational scale. It's called a service mesh, not a microservice mesh. First you get the benefits wrapping, securing, and monitoring each of your monoliths, then you reap more benefit when you start to break up each of those monoliths so common concerns can be addressed by common services and provide a unified experience to the enterprise's customers/users.
- deleted 3y ago
  [deleted]
pm90 3y ago
> Istio has become infamous among k8s infrastructure staff as being the cause of more problems than any other part of the stack True in my opinion. Its very complicated and the docs are confusing af.
- throwitaway222 3y ago
  At my last company the devops guy was installing istio for 2 years before he gave up. K8s by itself was just fine.
- jq-r 3y ago
  I've attended a talk on Kubecon last year on how one company adopted Istio service mesh. I've lost the guy in the first 10 minutes of the talk as it was so complicated, and decided that service mesh is 100% not going into our k8s clusters. Recently an overly confident security engineer came to us and demanded that we get service mesh because thats a SOX requirement. I have no idea from where these people get pipe dreams like that.
- AlecBG 3y ago
  I think Istio docs are great! I do agree that it's complicated, and I think their API is more confusing that it needs to be. The ontology of DestinationRules, VirtualServices, ServiceEntrys, Gateways (as in the K8s resource), gateways (as in the istio gateway Helm chart) is not the best.
- rexarex 3y ago
  The docs are TERRIBLE once you need to actually use them in prod.
pclmulqdq 3y ago
I am sort of surprised that the solution to encrypting your traffic isn't using HTTPS internally, and people rolled these wild systems of proxies. It makes sense thinking about it now because you have no idea whether you need the encryption or not (two containers may be on the same host), but this seems to be a pyrrhic victory. Once again, the bad implications of a seemingly good idea come to bite us.
- sweetjuly 3y ago
  It does have some advantages in that you don't have key material on every single server. How much this risk actually matters will vary. I suspect most enterprises would actually be relatively safe with their frontend certificates being compromised, though the same cannot be said about the actual compromise of the backend application.
  - pclmulqdq 3y ago
    I assume you would either use self-signed certs or run a private CA to do this. You wouldn't get the attestation benefits, but there's no material of any value in each container if you do this. The encryption is the only benefit. Reading that comment, I sort of understand why people don't do this because it takes some understanding of HTTPS and cryptography to do this properly.
- cyberax 3y ago
  It boils down to the complexity of running an internal DNS and of issuing certificates. If you don't do that, HTTPS is useful only for encryption against passive eavesdroppers.
- flumpcakes 3y ago
  The solution _is_ running HTTPS internally. But as an application developer, do you want to manage certificates? Do you want to make sure you're using the "correct" TLS versions and that your app can talk to other TLS versions? What happens when team Z deploys their legacy Java app that doesn't? Where are you getting your certificates from? All of these go away with a service mesh and sidecar model where the everything is encrypted for you and certificates are completely managed by the service mesh. No need to roll your own PKI. Your app only needs to speak plain HTTP and from it's perspective every other app is also just speaking plain HTTP, no TLS in sight. It makes developers' jobs a lot easier, only concentrating on shipping features vs. wrangling with TLS. Saying all of that, you could just roll your own fine. Depending on the competency of your company.
deleted 3y ago
[deleted]
throwawaaarrgh 3y ago
Just like you probably shouldn't be using K8s, you probably shouldn't be using a service mesh. Only add it when you actually need it. You'll know when that happens.
parhamn 3y ago
I think people often don't realize that depending on the language runtime, micro-services can easily be a must. Most service boundaries at organizations are "I need a different version of a pinned package we can't upgrade.". This is common in languages where there is support for only using one version of a given package, and it's worse if there isn't a culture of preserving function APIs. E.g. any python company with pandas/numpy in the stack will need to split the environments at some point in the future, no ifs ands or buts!
- pclmulqdq 3y ago
  I have heard that the reason Docker (and containers in general) took off was that they solved the problem of Python's awful package management. I didn't believe it until I saw people put Python in production and have to deal with this. At this point, I would rather have a physical snake in my server racks than any Python code.
  - nurettin 3y ago
    I have used python in production for years, multiple servers, multiple racks, and deployment has always been as simple as ./deploy.sh pull sync migrate seed restart pull calls git pull, sync runs pipenv sync, migrate runs django migrate, seed runs django management command seed, restart calls systemctl --user restart
    - toomuchtodo 3y ago
      This was not my experience building infra for a startup heavily leveraging a Python monolith. It was painful AF (both when developing locally and deploying to VMs) and Docker made the deployment story palatable (build, push to hundreds of VMs, run).
      - nurettin 3y ago
        Any details as to why?
jmspring 3y ago
The article is about service meshes and the tradeoffs amongst them. Going back several years, teams at companies ask about feature X - mtls a big one. The discussion goes to - should we use a service mesh, often the answer was no. K8s is a great platform with many options, but many decision makers have little knowledge (or don’t research) the implication of their choices.
- mianos 3y ago
  It is worse than that. Many decision makers are making their decisions based on advice from people who fired up k8s and all its gadgets for their pet project or google.
- gnarbarian 3y ago
  this is a personal attack
  - jmspring 3y ago
    How so?
    - DonHopkins 3y ago
      I take what he said as admitting that he is one of the "many decision makers [who] have little knowledge (or don’t research) the implication of their choices", and that objective fact stings him personally.
      - gnarbarian 3y ago
        yes it was also a joke
- billfor 3y ago
  I read the article as being about service meshes now being a cost item whereas they were free or low cost. I’m not sure debating the technical merits speaks to that.
n3storm 3y ago
To me every of these sounds like "why don't we put 10ys(tm) inside ZFWs(tm) inside Funnylettes(tm) proxy-edge-borderline stuff so stuff is so messhy and uncomprehensible that you need an IA to explain your application deployment"
- lpapez 3y ago
  Same. I've never worked in infrastructure teams because I simply don't care about this stuff, and I always find it funny how the infrastructure details tend to leak out to us application developers. Come on, I just want to deploy my app somewhere. I don't want to know about yamls and kubernetes, I just want a button where I deploy my branch. It often feels like people working on infrastructure and platform development do it for it's own sake, forgetting that they were supposed to be enablers for other teams building on top of it.
  - root_axis 3y ago
    You say you don't want to understand infrastructure and just want a button that magically deploys your app, yet in the same breath you complain that devops folks are "working on infrastructure and platform development for it's own sake". These are contradictory sentiments. "Leaking" infrastructure details to developers is actually less work for them because they are deferring the finer details of the app deployment to the developers who made it. Your request for a magic button that doesn't require you to understand anything about infrastructure is exactly why we have so much complexity.
    - jackpeterfletch 3y ago
      I agree that you never escape the requirement to have knowledge of how your application is deployed. Things like autoscaling, security, performance etc, all intrinsically effect how you actually write your application. I think there's a broader point though, in that very often, infra teams will pursue solutions that solve problems in the perspective of their own lens and interests without good oversight from the broader organisation, for developer experience, and economies of scale. I've seen and worked in environments at both ends of that scale and the gap in dev-ex, and agility as a result can be absolutely staggering. So often its to avoid 'vendor lock-in', only for infra teams to become the 'vendor', its complex, so they grow by necessity to be expensive, but then still lack the resources to be able provide a clean experience that can be easily migrated off of, resulting in lock-in. As a dev, the cost isn't my concern, but whats frustrating is knowing that its possible to deploy a new service, with all the bells and whistles, in an hour, and being unable to.
FridgeSeal 3y ago
> When you start using Kubernetes one of the first suggestions you'll get is to install a service mesh. Who on earth is giving you this advice? Service-messages are squarely in the “you’ll know when you need it, and it’s not day 1” bucket. Is this the experience people are having with K8s? Welcome to Kubernetes, here’s a service mesh gl;hf. If this advice is common, no wonder some people think K8s is over complicated. To anyone not aware: it _does not_ have to be this complicated. Default ingress controller and a normal Deployment will carry you really, really far.
- politelemon 3y ago
  To answer your question, yes this is the majority of the experience and is almost a given decision in the initial setup, in part because of the security association. Especially in enterprises it's easier to just say service mesh than have to think the decision through. Regardless of service mesh though, although it's a part of it, the complicated reputation is not undeserved and not solely due to meshes.
- sofixa 3y ago
  > If this advice is common, no wonder some people think K8s is over complicated I mean, if even Google who were behind it in the first place, are saying it's complicated and have 3 levels of managed Kubernetes services, I'd say it's pretty clear to everyone it is indeed complicated. Some of the complexity is simply due to the complex problems it solves, some is footguns, some is layers upon layers of abstractions to glue around design deficiencies. > To anyone not aware: it _does not_ have to be this complicated. Default ingress controller and a normal Deployment will carry you really, really far You're absolutely right. However, Kubernetes is rarely chosen after a careful evaluation of requirements; it's more often than not because it's considered necessary or for resume driven development. In that case, service mesh is easy to tack on, regardless of need.
- vbezhenar 3y ago
  I have small cluster and the only reason I’d want service mesh is to inspect req/resp between services for better debug. And I’m not even sure they can do that. Retries for free are good I guess, but not something essential. MTLS is something I don’t want at all.
qazxcvbnm 3y ago
Not having worked with K8s, it seems to me a number of things that service meshes are capable of can be done by SDN (e.g. Tailscale, ZeroTier). As far as I'm aware, SDN can do encryption and service discovery (via things like custom DNS) just fine. Can someone explain to me the differences and tradeoffs involved?
- ibotty 3y ago
  Cilium is a service mesh via being a SDN. That's hinted at in the article.
- atombender 3y ago
  That's been my thinking for a while, too. I work extensively on Kubernetes-hosted apps, but our org has (wisely, probably) eschewed service meshes in favour of ingress-based solutions. However, the simplicity of those solutions make things creaky and error-prone. Rather than injecting sidecar containers that set up networking and so on, having pods join an existing SDN that just works with no app-side config would be a much more elegant solution. Other than Cilium, I'm not aware of an SDN like ZeroTier or Wireguard that works seamlessly with Kubernetes this way (and which also works on managed Kubernetes like GKE and EKS).
throwitaway222 3y ago
K8S is quickly turning into BPML
- xyz-x 3y ago
  Seems to be it's an ebb and flow of complexity; sometimes it increases like with sidecar proxies and their issues, like how they might listen on NICs differently, or how they cause grey failures, and sometimes it decreases like when the sidecar moves to the kernel and gets rewritten in eBPF and we can remove another ip filtering-firewall-chain. So right now it seems to be a bit of an ebb, as these things are becoming moduralised.
andsens 3y ago
I never understood the appeal of service meshes. Half of their reason to exist is covered by vanilla kubernetes, the rest is inter-node VPN (e.g. wireguard) and tracing (cilium hubble). Unless I’m missing something encrypting intra-node traffic is pretty silly. K8S has service routing rules, network policies, access policies, and can be extended up the wazoo with whatever CNI you choose. It’s similar to Helm, in that Helm puts a DSL (values.yaml) on top of a DSL (go templates) on top of a DSL (k8s yaml), just that it is routing, authentication, and encryption on top.. well, routing (service route keys), authentication (netpols), and encryption. It boggles the mind!
- fragmede 3y ago
  Look, back in the day, things weren't encrypted, so you could listen in on your neighbor's phone calls, read their email, hack their bank accounts. Wireshark and etherdump and the most fun of all, driftnet. So, since then, everything has to be encrypted, lest someone hack there way to the family jewels. Never mind that the number of breaks to get there means there are usually bigger fish to fry. The important thing is to sprinkle magic encryption dust on everything because then we know it's Very Secure. (That's not to deride the fact that encryption is important, because it is, but sometimes it goes a bit far when there are other gaping holes that should be patched first.)
  - anonzzzies 3y ago
    Usually, unless someone is really doing naive things, you will need to have access to a lot of almost physical things to sniff traffic. You almost need to physically have access to room where either the server or the client is, even with unencrypted traffic. People say; 'but they can sniff it at level3'; they sure can, IF they have actual access to level3 on a higher level than just using them for normal traffic. Hacked switch or router or so. Probably state actors can and do pull that off, but outside that, it's really not so easy to get to unencrypted traffic of just a random target. You still should encrypt things of course when you can, but you don't have to get quite that paranoid about it. All major hacks are 0-days (well, not updated Wordpress is not necessarily 0-day; a lot of 0-days are exploited months or years later), stolen credentials (social engineering usually), brute force password hacks or applications that are left open (root/root for mysql with 3306 open to the world). Those have nothing to do with (un)encrypted traffic.
romantomjak 3y ago
> Infrastructure teams can usually implement a feature faster than every app team in a company, so this tends to get solved by them. Well, that's comparing apples to oranges. Product teams have completely different goals, e.g. adoption/retention/engagement, so naturally internal cluster encryption is so far out of scope that in fact only the platform team can reasonably implement it. I don't see how that statement is relevant. You don't send an electrician to build a brick wall
- flumpcakes 3y ago
  Application security should be everyone's responsibility. Architects, developers, and operations. Too many times have I seen architects and developers completely ignore it to make their jobs easier, leaving it to operations/infrastructure to implement. It's easy to twist the arm of business people with a "I can't ship feature X if you want me to look at security Y". If everyone took this seriously perhaps we would have fewer issues.
  - romantomjak 3y ago
    I agree, I was just making a point that different teams have different priorities and thus different scope. Saying "PodA can only talk to PodB over mTLS" is very different to "Users need to login using oauth". Who is going to build the product if product team is working on the service mesh?
    - arccy 3y ago
      you can implement mtls (and almost all the other service mesh features) without service meshes, and it's usually better because of lower overhead, less total complexity (see for example the fat client libraries in use by google, netflix, etc.). but people don't want to think about this so leave it to infra teams to plaster a service mesh over everything.
      - solatic 3y ago
        > mtls without service meshes You can, but it's absolutely a pain in the neck. Services need to load the certs from the filesystem on boot-up and trust the certs provided by other services. To manage trust, you need a certificate authority. Now you need to load the certificate authority's cert, and you need to manage rotation of certs. You need to help developers set up laptop-local certificate authorities and get them to issue certs so that you have Dev/Prod parity. You need to ensure that developers are enforcing modern ciphersuites, not doing bullshit "insecure-skip-verify" kind of toggles that make their jobs easier (because remember, their job isn't security, it's shipping features), not accepting self-signed certs or other certs not signed by the certificate authority. You need to make sure all this stuff is put in the testing suite to make sure it keeps getting maintained, and you need the files for these tests marked in CODEOWNERS to be under InfoSec control to ensure nobody rips them out just because they're inconvenient. And you need to copy this for every single service you run in production and every single development team. You know what else you can do? Write your own web server (/sarcasm). I mean, who needs nginx? Probably writing your own will have lower overhead and less total complexity, not running a bunch of features that you don't use. And probably it will not be anywhere close to as good as a battle-hardened web server used by millions of engineers that gets regular support. Personally I think it's debatable whether services really need mTLS within a private network. It's mostly a question of what scale you're running at; probably there's higher benefit-to-effort-ratio InfoSec projects to tackle. But if you do decide you need it, unless you can prove that the overhead is unworkable for your requirements, really you need to bite the bullet and put in a service mesh.
kosolam 3y ago
This article gives an overview of the 5 or so most popular services mesh options for Kubernetes for Enterprise installations. So if you are reading it and not the target audience, keep that in mind. Also, people in the comments arguing against service mesh with “it seems”, “i assume”, “i think”, etc. Please read the list of concerns that a service mesh is good at resolving, also in this article. If you don’t have these concerns, it doesn’t mean they dont exist elsewhere, right?
- deleted 3y ago
  [deleted]
secondcoming 3y ago
Assuming that this is yet another DevOps tech I'll be expected to be an expert in, I looked up Istio [0]: As organizations accelerate their moves to the cloud, they are, by necessity, modernizing their applications as well. But shifting from monolithic legacy apps to cloud-native ones can raise challenges for DevOps teams. Developers must learn to assemble apps using loosely coupled microservices to ensure portability in the cloud. This just isn't true. You don't need microservices to use the Cloud. [0] https://cloud.google.com/learn/what-is-istio https://cloud.google.com/learn/what-is-istio
- baq 3y ago
  I think they mean ‘you must be able to run your monolith in a way which only starts a small subset of services’ to leverage the cloud cost model, otherwise just rent a colo.
- geodel 3y ago
  Well it is true in same sense many think they need meal kit delivery service if they want to avoid eating outside.
trailrunner46 3y ago
This was a good read, as someone using K8s a lot in the last two years but not service meshes yet, it gave me a lot to think about. I understand there are various advantages like metrics, etc, but the encrypted traffic between pods and services is the one that I can see many orgs demanding. If not a service mesh what other options are there?
kapilvt 3y ago
[flagged]
- deleted 3y ago
  [deleted]
nyrikki 3y ago
Service meshes as a default pattern is harmful. Ideally you would be using immutable events over a message bus as a default pattern. Just as anti corruption layers have a use case. So do service meshes. But service meshes they are synchronous communication, with indirection typically through a sidecar, they will have the very real costs of synchronous communication in distributed systems. IMHO the problem isn't that they are problematic when applied to the correct use case, it is that as a default pattern they inherently will cause problems with little to no benefit for systems that can use events. It is simply the same as any tight coupling, it should be avoided unless a specific use case justifies the costs. Cargo culting it in because it is superficially 'easier' will typically result in code that doesn't have the scaffolding in place to support events and is very expensive to pivot away from if you don't intentionally write your code to support looser coupling in the future. Istio, Envoy, Linkerd, etc... currently cater to the synchronous request/response communication between microservices. It doesn't matter how decetralized their implementation is, it is still essentially sync operations over a network. We have known about the costs of that decision for a long time irrespective of the implementation complexity. It is a 'sometimes' food.
sporkland 3y ago
As someone that got a lot of value out of client side load balancers as a language neutral network library (e.g. we ran envoy under synapse for a while) From the start the Istio design choices tended to address problems we didn't have / felt like architectural dream states while making problems we did have harder to address. The messaging and overall focus seems to have improved and focused more on meat and potatoes features but there's still a number of things that are hard to tweak via the standard istio apis that we have to backdoor in there.
- BlackjackCF 3y ago
  Istio reminds me of the saying on Sawbones (a medical history podcast) when it comes to “cure-alls”: Cure-alls cure nothing.
whatthesmack 3y ago
As great as reducing complexity is, I just don't see how it's possible to avoid implementing a service mesh in a FedRAMP moderate or high impact level environment. You essentially need to implement mTLS to meet SC-8(1), and to implement mTLS at scale, you need something like a service mesh. Are there other ways of going about this for FedRAMP moderate or high IL?
- remram 3y ago
  > to implement mTLS at scale, you need something like a service mesh What makes you think that?

233 points by zdw 3y ago

good lord is this what modern microservices are like? How is it better have service A request to a proxy, which requests to another proxy, which requests to service B? I get the security benefits of that, but the network architecture is boggling. How many PBs of data are sent each day for what could be a monolithic service? Actually, to the end — companies that embrace microservices, which see the value in them. How do they manage network traffic for these kinds of K8s setups at scale? Surely they’re not using REST and HTTP. Is it as simple as protobufs over HTTP? Quic? Something else? Edit: lots of great discussion below but I really meant how do they manage network TRAFFIC, not microservices in general :)

throwup238 3y ago
> How do they manage these kinds of K8s setups at scale? Our DevOps team starts off the monthly all hands meeting by leading a ritual during which they ceremoniously sacrifice an animal from the Fish and Wildlife Service's Threatened & Endangered Species list while the rest of the company chants: Exorcizamus te, omnis immundus spiritus omnis satanica potestas, omnis incursio infernalis adversarii, omnis legio, omnis congregatio et secta diabolica.
- zubairq 3y ago
  haha, IT staff doing mystic chants reminds me of this video I made over a decade ago: https://www.youtube.com/watch?v=8Sj3_NfDYeU https://www.youtube.com/watch?v=8Sj3_NfDYeU
- jacinda 3y ago
  I needed this laugh so much. Thank you.
- vaporary 3y ago
  "Getting a SCSI chain working is perfectly simple if you remember that there must be exactly three terminations: one on one end of the cable, one on the far end, and the goat, terminated over the SCSI chain with a silver-handled knife whilst burning *black* candles." -- Anthony DeBoer "SCSI is *not* magic. There are *fundamental* *technical* *reasons* why you have to sacrifice a young goat to your SCSI chain every now and then." -- John F. Woods
  - jakupovic 3y ago
    I LOL-ed at this because I used to cry when no goats were around.
spockz 3y ago
The data over the wire is the effect of having micro services and has little to do with service meshes. If anything, a service mesh could help by transparently enabling compression or upgrading to h3 without having to bake that in every app. Also, from a networking perspective, the proxies are hosted next to the instances so there is no difference there. We run 1800 services in our mesh. Most of them rest/http, a select few gRPC and graphql. We even have some soap/http services in it.
- slimsag 3y ago
  with 1800 services, do you feel like you are programming/architecting code in the same way a complex monolithic codebase might, working across them all? Or are the services just cogs in the machine, managed by individual devs / cogs in the machine? I imagine the latter and presume that's the main benefit of so many services, but genuinely curious as I've never experience that many services in an architecture
  - bostik 3y ago
    I would say it's right about the middle of those two extremes. A well functioning service mesh (or even just a well maintained and discoverable ingress controller) is essentially invisible to the individual dev teams. Just think how modern stacks work from a frontend dev's perspective: team wants to use an additional feature, so they find the budgeted credit card, sign up to a random third-party provider, get their access token, and go. From the codebase standpoint, they merely added a new roundtrip to a random service and process the responses in their code. From the dev team's point of view, having the same feature available internally, behind "just another URL", makes no big difference. Maybe less politics around vendor spend and, with luck, easier integration with the remote service auth. Almost certainly less wrangling with compliance and legal. Whether that URL is provided by an ingress with a proper FQDN, or a service mesh entry with otherwise unresolvable name, is (and should be) irrelevant. Modern distributed systems have long since become too large for any single person to fully comprehend them through and through. There is no Grand Design[tm], they are all results of organic changes and evolution. Service discovery and routing can be architected. Individual services within the system can be architected. The complete system where hundreds or even thousands of services interact can not.
de6u99er 3y ago
The trick is to know which code you want to maintain yourself and which stuff you want to get off the shelf. Going the off the shelf way, also means being stateless and using distributed transactions with all their pitfalls (eventual consistency, fire and forget, ...). Good architects will know how and when to do what E.g. start with a modulith instead of a monolith, to ease refactoring into micro-services once user count goes through the roof and vertical scaling won't do it any more.
__turbobrew__ 3y ago
> How do they manage these kinds of K8s setups at scale? You check in configs into the monorepo and there is tooling to continuously sync the configs with the actual state of the infrastructure. The advantage of microservices at scale is that team X breaking the build doesn’t affect team Y. This scale is probably not until you have 1000+ engineers however.
- pclmulqdq 3y ago
  To be clear, that is the advantage of services. Microservices are a culture of taking that isolation as far as you can.
- jakupovic 3y ago
  You do deployments as one used to do with big C++ programs also, you can delete everything and deploy again to make sure everything is new. The latter is preferred. Doing continuous deployments is very hard and not sure possible, at least I haven't seen it work well even though it's touted as fundamental k8s feature. K8s groups resources and presents a platform, how you use it it's up to you, and the management.
pm90 3y ago
> How is it better have service A request to a proxy, which requests to another proxy, which requests to service B? I get the security benefits of that, but the network architecture is boggling. How many PBs of data are sent each day for what could be a monolithic service? The tradeoff here is to decouple services in order to allow them to be developed somewhat independently of each other. Monoliths remove the overhead of network requests but they present their own challenges. You have a lot of implicit dependencies and feature development becomes complicated with changes having unexpected effects very far from the source. Ultimately engineering organizations need to decide the model that works best for them. Neither is inherently better, they’re solving different problems.
- deathanatos 3y ago
  > The tradeoff here is to decouple services in order to allow them to be developed somewhat independently of each other. You don't need a service mesh for that, though? Heck, you don't even need an ingress for service to service traffic. ingress-nginx does the job well, without being overly complex, and most importantly to me, logs when something is wrong, which I cannot say the same for Istio which I was fighting earlier this week where it was just happily RST'ing a connection and saying nothing about why it was deciding to do that.
  - campbel 3y ago
    There are a lot of good and bad reasons to adopt a mesh. Some of which might relate to your concerns. The things I like most about them, working in infrastructur: 1. I can have a unified set of metrics for all services, regardless of language/platform or how diligent the team is at instrumenting their apps. 2. I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates. 3. I can implement automation around canary releases without much lift from dev teams. Other projects leverage these capabilities and do it for you as well. 4. I can get the equivalent of tcpdump for a pod pretty easily which I can use to help app teams debug issues. 5. I can improve app reliability with automatic retries and timeouts. Probably some other things as well... That said, it can be a big increase in complexity to your system the pains of which aren't always distributed to the folks getting the benefits.
paulddraper 3y ago
> A request to a proxy, which requests to another proxy, which requests to service B? I hesitate to tell you how many layers are between me an HN's servers right now.
- phildenhoff 3y ago
  Right but there are all those layers so that HNs one server can service your request and send a response. How much traffic would be generated if, for every request to HN, six other requests fire? And every time you comment a cascade of requests fire in HNs imaginary K8s cluster? It’s not that there is anything wrong with this, or that the tradeoff isn’t worth it… it’s just so much data flying back and forth over the wire.
  - aetimmes 3y ago
    Ultimately, in the age of massive cloud compute, the constraining resource in an organization is engineering hours, not CPU/memory/bandwidth. And even then, I've yet to encounter a system (outside of massively parallel MPI-based HPC) where saturating network pipes became the bottleneck for a system before CPU/memory utilization. Microservice architecture means there's lot of data flying around, but it keeps local resource utilization predictable.
    - com 3y ago
      I thought that the constraining resource was money in all but the top 1000 or so businesses in the world with effectively 99%+ margins, mostly because they control their markets absolutely and can raise prices or change demand like the big three algorithmic advertising companies. Everybody else in the cloud goes to the wall without very careful cost control, which is by no means automated or low cost, either.
  - eptcyka 3y ago
    Most often, the wire is virtual, its just copying bytes between processes.
  - paulddraper 3y ago
patcon 3y ago
Heh wait until you hear how the cell signalling and any biological process in your body works :) It's an absolute clusterfuck of dependencies and side-effects. It's like space bar heating functionality[1] all the way down...! In my thinking, we might as well get used to the levels of indirection and complexity that AIs will be comfortable with. I suspect it will be more akin to what biological computation is "comfortable" with, and less like what our minds happen to prefer But I digress :) yes, microservices strike me as wiiild [1]: https://xkcd.com/1172/ https://xkcd.com/1172/
15457345234 3y ago
You're asking some really good questions here I find that microservices and this type of architecture have become a religion - you do it this way because you do it this way. You add another layer of complication because that's what you do now. You add this product because that's what you do now. Now you do it this way. Now you stop doing this thing and do this thing instead. It's all proclamations and a truly insane level of complexity and often a truly stunningly low level of performance achieved from some very powerful hardware because everything is behind at least twenty layers of abstraction and you're like, encrypting traffic which is just being passed between VMs which are on the same hardware, but because you can't guarantee that they're always on the same hardware you have to encrypt and use a proxy and... oh wow Watching it from the outside is a bit exhausting, it just seems to be so much churn and overhead.
deleted 3y ago
[deleted]
threeseed 3y ago
> for what could be a monolithic service On larger projects: a) It is sometimes impossible to run the entire application on a laptop. And so having a micro-service means you can quickly iterate and test before embedding it into the wider system. b) You will commonly run into conflicting transitive dependencies which you simply can't work around. Classic example being Spark on the JVM which brings in years old Hadoop libraries. c) The inter-relationships between component can become so complex that you really want to be able to use canary or green/blue deployment techniques to reduce risk.
deathanatos 3y ago
> Edit: lots of great discussion below but I really meant how do they manage network TRAFFIC, not microservices in general :) I don't really understand the question you're asking, but I think maybe the answer is that network pipes are just bigger than the scale most people are operating at? I don't think anything I've ever done has really had that many qps, and if it has, it is more likely to raise an eyebrow that says "who's spamming requests" more that "I guess we've made it to the big time". REST & protobufs are orthogonal. Empirically, literally nobody is doing REST, and most things are just ad hoc, poorly to not-at-all defined JSON/HTTP with a few HTTP verbs sprinkled in to make everyone feel good. It could be protobuf, too, if you like, but unless you have some truly gargantuan JSON, it really won't matter in the end. Compression will make up enough of the difference in size on the network. Some languages don't have to allocate the keys a billion times, too, though even in Python, it's a while before it starts to hurt. What I see more of is processes just inexplicably using gigabytes upon gigabytes of RAM, burning through whole years of CPU time for no particular reason before just dropping back to nominal levels like nothing happened, and dev teams that can't coherently understand the disconnect between just how much power a modern machine has, and what their design doc says their process is supposed to do (hint: something that shouldn't take that many resources). I like microservices, but there should be strong areas of responsibility to them. For most companies, I think that's ~2–3 services. At my current company, it's ~2 services + a database, with the rest being things like cron jobs or really small services that are just various glue or infra tooling.
foota 3y ago
Traffic within a data center is relatively cheap, is it not? Of course, I think managed network proxies tend to be expensive, but I think that mostly comes from the machine costs, not the network itself. You pay a bit of a tax for network serialization, but it's unlikely to be the bottleneck in most things ime. Plus, applications built on an external database (e.g., postgres not sqlite) will already have 1 hop. Going from 1 to 2 hops is less dramatic than going from no hops to 1. And I guess implicitly any service sending lots of data to the client already has 1 hop. Imo the only case where a network proxy would be egregious is an in memory database kind of workload where the response size is small relative to the size of the data accessed (maybe something like a custom analytics engine), but that's pretty niche.
- ozr 3y ago
  > Traffic within a data center is relatively cheap, is it not? Complexity is very, very expensive.
- manfre 3y ago
  > Traffic within a data center is relatively cheap, Most organizations will run their services spread across multiple data centers / AZs.
  - foota 3y ago
    Hm, wouldn't you want to set it up to route traffic preferentially to a nearby location?
lukeschlather 3y ago
I really hate this pattern in principle, but it isn't as bad as you're making it out to be. Most of the service meshes operate as sidecars, which is to say that if service A is calling service B through a mesh, there are two proxies in between the services, but proxy A is on the same machine as A, and proxy B is on the same machine as proxy B. So it is kind of offensive to have 3 network hops involved, but actually only one of those is over the wire, and in most cases I don't think the proxies are actually causing any measurable latency. (At least, whatever latency it's causing is smaller than the latency caused by the TLS encryption, which is necessary and the whole point.)
- deleted 3y ago
  [deleted]
- phildenhoff 3y ago
  That makes sense! I didn't realise the proxies were sidecars, but that seems obvious in retrospect.
dilyevsky 3y ago
They manage traffic with “service meshes” which includes anything from relatively” basic” distributed iptables/ipvs VIPs (kube-proxy) + dns to full-blown intercepting proxies a la istio/linkerd
geraldwhen 3y ago
It’s all http with hundreds of milliseconds of latency per request.
solatic 3y ago
> How many PBs of data are sent each day for what could be a monolithic service? The companies that need service meshes couldn't possibly run everything in a single monolith. They already have several if not dozens of monoliths, each one typically coming into the architecture when a large enterprise acquires another company and its monolith, or simply different business units / product lines that don't talk to each other because of sheer organizational scale. It's called a service mesh, not a microservice mesh. First you get the benefits wrapping, securing, and monitoring each of your monoliths, then you reap more benefit when you start to break up each of those monoliths so common concerns can be addressed by common services and provide a unified experience to the enterprise's customers/users.
deleted 3y ago
[deleted]

pm90 3y ago

> Istio has become infamous among k8s infrastructure staff as being the cause of more problems than any other part of the stack True in my opinion. Its very complicated and the docs are confusing af.

throwitaway222 3y ago
At my last company the devops guy was installing istio for 2 years before he gave up. K8s by itself was just fine.
jq-r 3y ago
I've attended a talk on Kubecon last year on how one company adopted Istio service mesh. I've lost the guy in the first 10 minutes of the talk as it was so complicated, and decided that service mesh is 100% not going into our k8s clusters. Recently an overly confident security engineer came to us and demanded that we get service mesh because thats a SOX requirement. I have no idea from where these people get pipe dreams like that.
AlecBG 3y ago
I think Istio docs are great! I do agree that it's complicated, and I think their API is more confusing that it needs to be. The ontology of DestinationRules, VirtualServices, ServiceEntrys, Gateways (as in the K8s resource), gateways (as in the istio gateway Helm chart) is not the best.
rexarex 3y ago
The docs are TERRIBLE once you need to actually use them in prod.

pclmulqdq 3y ago

I am sort of surprised that the solution to encrypting your traffic isn't using HTTPS internally, and people rolled these wild systems of proxies. It makes sense thinking about it now because you have no idea whether you need the encryption or not (two containers may be on the same host), but this seems to be a pyrrhic victory. Once again, the bad implications of a seemingly good idea come to bite us.

sweetjuly 3y ago
It does have some advantages in that you don't have key material on every single server. How much this risk actually matters will vary. I suspect most enterprises would actually be relatively safe with their frontend certificates being compromised, though the same cannot be said about the actual compromise of the backend application.
- pclmulqdq 3y ago
  I assume you would either use self-signed certs or run a private CA to do this. You wouldn't get the attestation benefits, but there's no material of any value in each container if you do this. The encryption is the only benefit. Reading that comment, I sort of understand why people don't do this because it takes some understanding of HTTPS and cryptography to do this properly.
cyberax 3y ago
It boils down to the complexity of running an internal DNS and of issuing certificates. If you don't do that, HTTPS is useful only for encryption against passive eavesdroppers.
flumpcakes 3y ago
The solution _is_ running HTTPS internally. But as an application developer, do you want to manage certificates? Do you want to make sure you're using the "correct" TLS versions and that your app can talk to other TLS versions? What happens when team Z deploys their legacy Java app that doesn't? Where are you getting your certificates from? All of these go away with a service mesh and sidecar model where the everything is encrypted for you and certificates are completely managed by the service mesh. No need to roll your own PKI. Your app only needs to speak plain HTTP and from it's perspective every other app is also just speaking plain HTTP, no TLS in sight. It makes developers' jobs a lot easier, only concentrating on shipping features vs. wrangling with TLS. Saying all of that, you could just roll your own fine. Depending on the competency of your company.

deleted 3y ago

[deleted]

throwawaaarrgh 3y ago

Just like you probably shouldn't be using K8s, you probably shouldn't be using a service mesh. Only add it when you actually need it. You'll know when that happens.

parhamn 3y ago

I think people often don't realize that depending on the language runtime, micro-services can easily be a must. Most service boundaries at organizations are "I need a different version of a pinned package we can't upgrade.". This is common in languages where there is support for only using one version of a given package, and it's worse if there isn't a culture of preserving function APIs. E.g. any python company with pandas/numpy in the stack will need to split the environments at some point in the future, no ifs ands or buts!

pclmulqdq 3y ago
I have heard that the reason Docker (and containers in general) took off was that they solved the problem of Python's awful package management. I didn't believe it until I saw people put Python in production and have to deal with this. At this point, I would rather have a physical snake in my server racks than any Python code.
- nurettin 3y ago
  I have used python in production for years, multiple servers, multiple racks, and deployment has always been as simple as ./deploy.sh pull sync migrate seed restart pull calls git pull, sync runs pipenv sync, migrate runs django migrate, seed runs django management command seed, restart calls systemctl --user restart
  - toomuchtodo 3y ago
    This was not my experience building infra for a startup heavily leveraging a Python monolith. It was painful AF (both when developing locally and deploying to VMs) and Docker made the deployment story palatable (build, push to hundreds of VMs, run).
    - nurettin 3y ago
      Any details as to why?

jmspring 3y ago

The article is about service meshes and the tradeoffs amongst them. Going back several years, teams at companies ask about feature X - mtls a big one. The discussion goes to - should we use a service mesh, often the answer was no. K8s is a great platform with many options, but many decision makers have little knowledge (or don’t research) the implication of their choices.

mianos 3y ago
It is worse than that. Many decision makers are making their decisions based on advice from people who fired up k8s and all its gadgets for their pet project or google.
gnarbarian 3y ago
this is a personal attack
- jmspring 3y ago
  How so?
  - DonHopkins 3y ago
    I take what he said as admitting that he is one of the "many decision makers [who] have little knowledge (or don’t research) the implication of their choices", and that objective fact stings him personally.
    - gnarbarian 3y ago
      yes it was also a joke
billfor 3y ago
I read the article as being about service meshes now being a cost item whereas they were free or low cost. I’m not sure debating the technical merits speaks to that.

n3storm 3y ago

To me every of these sounds like "why don't we put 10ys(tm) inside ZFWs(tm) inside Funnylettes(tm) proxy-edge-borderline stuff so stuff is so messhy and uncomprehensible that you need an IA to explain your application deployment"

lpapez 3y ago
Same. I've never worked in infrastructure teams because I simply don't care about this stuff, and I always find it funny how the infrastructure details tend to leak out to us application developers. Come on, I just want to deploy my app somewhere. I don't want to know about yamls and kubernetes, I just want a button where I deploy my branch. It often feels like people working on infrastructure and platform development do it for it's own sake, forgetting that they were supposed to be enablers for other teams building on top of it.
- root_axis 3y ago
  You say you don't want to understand infrastructure and just want a button that magically deploys your app, yet in the same breath you complain that devops folks are "working on infrastructure and platform development for it's own sake". These are contradictory sentiments. "Leaking" infrastructure details to developers is actually less work for them because they are deferring the finer details of the app deployment to the developers who made it. Your request for a magic button that doesn't require you to understand anything about infrastructure is exactly why we have so much complexity.
  - jackpeterfletch 3y ago
    I agree that you never escape the requirement to have knowledge of how your application is deployed. Things like autoscaling, security, performance etc, all intrinsically effect how you actually write your application. I think there's a broader point though, in that very often, infra teams will pursue solutions that solve problems in the perspective of their own lens and interests without good oversight from the broader organisation, for developer experience, and economies of scale. I've seen and worked in environments at both ends of that scale and the gap in dev-ex, and agility as a result can be absolutely staggering. So often its to avoid 'vendor lock-in', only for infra teams to become the 'vendor', its complex, so they grow by necessity to be expensive, but then still lack the resources to be able provide a clean experience that can be easily migrated off of, resulting in lock-in. As a dev, the cost isn't my concern, but whats frustrating is knowing that its possible to deploy a new service, with all the bells and whistles, in an hour, and being unable to.

FridgeSeal 3y ago

> When you start using Kubernetes one of the first suggestions you'll get is to install a service mesh. Who on earth is giving you this advice? Service-messages are squarely in the “you’ll know when you need it, and it’s not day 1” bucket. Is this the experience people are having with K8s? Welcome to Kubernetes, here’s a service mesh gl;hf. If this advice is common, no wonder some people think K8s is over complicated. To anyone not aware: it _does not_ have to be this complicated. Default ingress controller and a normal Deployment will carry you really, really far.

politelemon 3y ago
To answer your question, yes this is the majority of the experience and is almost a given decision in the initial setup, in part because of the security association. Especially in enterprises it's easier to just say service mesh than have to think the decision through. Regardless of service mesh though, although it's a part of it, the complicated reputation is not undeserved and not solely due to meshes.
sofixa 3y ago
> If this advice is common, no wonder some people think K8s is over complicated I mean, if even Google who were behind it in the first place, are saying it's complicated and have 3 levels of managed Kubernetes services, I'd say it's pretty clear to everyone it is indeed complicated. Some of the complexity is simply due to the complex problems it solves, some is footguns, some is layers upon layers of abstractions to glue around design deficiencies. > To anyone not aware: it _does not_ have to be this complicated. Default ingress controller and a normal Deployment will carry you really, really far You're absolutely right. However, Kubernetes is rarely chosen after a careful evaluation of requirements; it's more often than not because it's considered necessary or for resume driven development. In that case, service mesh is easy to tack on, regardless of need.
vbezhenar 3y ago
I have small cluster and the only reason I’d want service mesh is to inspect req/resp between services for better debug. And I’m not even sure they can do that. Retries for free are good I guess, but not something essential. MTLS is something I don’t want at all.

qazxcvbnm 3y ago

Not having worked with K8s, it seems to me a number of things that service meshes are capable of can be done by SDN (e.g. Tailscale, ZeroTier). As far as I'm aware, SDN can do encryption and service discovery (via things like custom DNS) just fine. Can someone explain to me the differences and tradeoffs involved?

ibotty 3y ago
Cilium is a service mesh via being a SDN. That's hinted at in the article.
atombender 3y ago
That's been my thinking for a while, too. I work extensively on Kubernetes-hosted apps, but our org has (wisely, probably) eschewed service meshes in favour of ingress-based solutions. However, the simplicity of those solutions make things creaky and error-prone. Rather than injecting sidecar containers that set up networking and so on, having pods join an existing SDN that just works with no app-side config would be a much more elegant solution. Other than Cilium, I'm not aware of an SDN like ZeroTier or Wireguard that works seamlessly with Kubernetes this way (and which also works on managed Kubernetes like GKE and EKS).

throwitaway222 3y ago

K8S is quickly turning into BPML

xyz-x 3y ago
Seems to be it's an ebb and flow of complexity; sometimes it increases like with sidecar proxies and their issues, like how they might listen on NICs differently, or how they cause grey failures, and sometimes it decreases like when the sidecar moves to the kernel and gets rewritten in eBPF and we can remove another ip filtering-firewall-chain. So right now it seems to be a bit of an ebb, as these things are becoming moduralised.

andsens 3y ago

I never understood the appeal of service meshes. Half of their reason to exist is covered by vanilla kubernetes, the rest is inter-node VPN (e.g. wireguard) and tracing (cilium hubble). Unless I’m missing something encrypting intra-node traffic is pretty silly. K8S has service routing rules, network policies, access policies, and can be extended up the wazoo with whatever CNI you choose. It’s similar to Helm, in that Helm puts a DSL (values.yaml) on top of a DSL (go templates) on top of a DSL (k8s yaml), just that it is routing, authentication, and encryption on top.. well, routing (service route keys), authentication (netpols), and encryption. It boggles the mind!

fragmede 3y ago
Look, back in the day, things weren't encrypted, so you could listen in on your neighbor's phone calls, read their email, hack their bank accounts. Wireshark and etherdump and the most fun of all, driftnet. So, since then, everything has to be encrypted, lest someone hack there way to the family jewels. Never mind that the number of breaks to get there means there are usually bigger fish to fry. The important thing is to sprinkle magic encryption dust on everything because then we know it's Very Secure. (That's not to deride the fact that encryption is important, because it is, but sometimes it goes a bit far when there are other gaping holes that should be patched first.)
- anonzzzies 3y ago
  Usually, unless someone is really doing naive things, you will need to have access to a lot of almost physical things to sniff traffic. You almost need to physically have access to room where either the server or the client is, even with unencrypted traffic. People say; 'but they can sniff it at level3'; they sure can, IF they have actual access to level3 on a higher level than just using them for normal traffic. Hacked switch or router or so. Probably state actors can and do pull that off, but outside that, it's really not so easy to get to unencrypted traffic of just a random target. You still should encrypt things of course when you can, but you don't have to get quite that paranoid about it. All major hacks are 0-days (well, not updated Wordpress is not necessarily 0-day; a lot of 0-days are exploited months or years later), stolen credentials (social engineering usually), brute force password hacks or applications that are left open (root/root for mysql with 3306 open to the world). Those have nothing to do with (un)encrypted traffic.

romantomjak 3y ago

> Infrastructure teams can usually implement a feature faster than every app team in a company, so this tends to get solved by them. Well, that's comparing apples to oranges. Product teams have completely different goals, e.g. adoption/retention/engagement, so naturally internal cluster encryption is so far out of scope that in fact only the platform team can reasonably implement it. I don't see how that statement is relevant. You don't send an electrician to build a brick wall

flumpcakes 3y ago
Application security should be everyone's responsibility. Architects, developers, and operations. Too many times have I seen architects and developers completely ignore it to make their jobs easier, leaving it to operations/infrastructure to implement. It's easy to twist the arm of business people with a "I can't ship feature X if you want me to look at security Y". If everyone took this seriously perhaps we would have fewer issues.
- romantomjak 3y ago
  I agree, I was just making a point that different teams have different priorities and thus different scope. Saying "PodA can only talk to PodB over mTLS" is very different to "Users need to login using oauth". Who is going to build the product if product team is working on the service mesh?
  - arccy 3y ago
    you can implement mtls (and almost all the other service mesh features) without service meshes, and it's usually better because of lower overhead, less total complexity (see for example the fat client libraries in use by google, netflix, etc.). but people don't want to think about this so leave it to infra teams to plaster a service mesh over everything.
    - solatic 3y ago
      > mtls without service meshes You can, but it's absolutely a pain in the neck. Services need to load the certs from the filesystem on boot-up and trust the certs provided by other services. To manage trust, you need a certificate authority. Now you need to load the certificate authority's cert, and you need to manage rotation of certs. You need to help developers set up laptop-local certificate authorities and get them to issue certs so that you have Dev/Prod parity. You need to ensure that developers are enforcing modern ciphersuites, not doing bullshit "insecure-skip-verify" kind of toggles that make their jobs easier (because remember, their job isn't security, it's shipping features), not accepting self-signed certs or other certs not signed by the certificate authority. You need to make sure all this stuff is put in the testing suite to make sure it keeps getting maintained, and you need the files for these tests marked in CODEOWNERS to be under InfoSec control to ensure nobody rips them out just because they're inconvenient. And you need to copy this for every single service you run in production and every single development team. You know what else you can do? Write your own web server (/sarcasm). I mean, who needs nginx? Probably writing your own will have lower overhead and less total complexity, not running a bunch of features that you don't use. And probably it will not be anywhere close to as good as a battle-hardened web server used by millions of engineers that gets regular support. Personally I think it's debatable whether services really need mTLS within a private network. It's mostly a question of what scale you're running at; probably there's higher benefit-to-effort-ratio InfoSec projects to tackle. But if you do decide you need it, unless you can prove that the overhead is unworkable for your requirements, really you need to bite the bullet and put in a service mesh.

kosolam 3y ago

This article gives an overview of the 5 or so most popular services mesh options for Kubernetes for Enterprise installations. So if you are reading it and not the target audience, keep that in mind. Also, people in the comments arguing against service mesh with “it seems”, “i assume”, “i think”, etc. Please read the list of concerns that a service mesh is good at resolving, also in this article. If you don’t have these concerns, it doesn’t mean they dont exist elsewhere, right?

deleted 3y ago
[deleted]

secondcoming 3y ago

Assuming that this is yet another DevOps tech I'll be expected to be an expert in, I looked up Istio [0]: As organizations accelerate their moves to the cloud, they are, by necessity, modernizing their applications as well. But shifting from monolithic legacy apps to cloud-native ones can raise challenges for DevOps teams. Developers must learn to assemble apps using loosely coupled microservices to ensure portability in the cloud. This just isn't true. You don't need microservices to use the Cloud. [0] https://cloud.google.com/learn/what-is-istio https://cloud.google.com/learn/what-is-istio

baq 3y ago
I think they mean ‘you must be able to run your monolith in a way which only starts a small subset of services’ to leverage the cloud cost model, otherwise just rent a colo.
geodel 3y ago
Well it is true in same sense many think they need meal kit delivery service if they want to avoid eating outside.

trailrunner46 3y ago

This was a good read, as someone using K8s a lot in the last two years but not service meshes yet, it gave me a lot to think about. I understand there are various advantages like metrics, etc, but the encrypted traffic between pods and services is the one that I can see many orgs demanding. If not a service mesh what other options are there?

kapilvt 3y ago

[flagged]

deleted 3y ago
[deleted]

nyrikki 3y ago

Service meshes as a default pattern is harmful. Ideally you would be using immutable events over a message bus as a default pattern. Just as anti corruption layers have a use case. So do service meshes. But service meshes they are synchronous communication, with indirection typically through a sidecar, they will have the very real costs of synchronous communication in distributed systems. IMHO the problem isn't that they are problematic when applied to the correct use case, it is that as a default pattern they inherently will cause problems with little to no benefit for systems that can use events. It is simply the same as any tight coupling, it should be avoided unless a specific use case justifies the costs. Cargo culting it in because it is superficially 'easier' will typically result in code that doesn't have the scaffolding in place to support events and is very expensive to pivot away from if you don't intentionally write your code to support looser coupling in the future. Istio, Envoy, Linkerd, etc... currently cater to the synchronous request/response communication between microservices. It doesn't matter how decetralized their implementation is, it is still essentially sync operations over a network. We have known about the costs of that decision for a long time irrespective of the implementation complexity. It is a 'sometimes' food.

sporkland 3y ago

As someone that got a lot of value out of client side load balancers as a language neutral network library (e.g. we ran envoy under synapse for a while) From the start the Istio design choices tended to address problems we didn't have / felt like architectural dream states while making problems we did have harder to address. The messaging and overall focus seems to have improved and focused more on meat and potatoes features but there's still a number of things that are hard to tweak via the standard istio apis that we have to backdoor in there.

BlackjackCF 3y ago
Istio reminds me of the saying on Sawbones (a medical history podcast) when it comes to “cure-alls”: Cure-alls cure nothing.

whatthesmack 3y ago

As great as reducing complexity is, I just don't see how it's possible to avoid implementing a service mesh in a FedRAMP moderate or high impact level environment. You essentially need to implement mTLS to meet SC-8(1), and to implement mTLS at scale, you need something like a service mesh. Are there other ways of going about this for FedRAMP moderate or high IL?

remram 3y ago
> to implement mTLS at scale, you need something like a service mesh What makes you think that?