3 ms·
> The RPC service has two methods and that is the only way _anything_ can interact with the real service, so faking out those two RPC calls let you write focuse
by philwelch 8y ago
> The RPC service has two methods and that is the only way _anything_ can interact with the real service, so faking out those two RPC calls let you write focused tests easily.
Except now you have to cover cases like “the service is down”, “the service is too latent”, “the actual outputs from the service differ from the documentation”, “the service is behind a reverse proxy that mutates headers in a surprising way”, “you are behind a reverse proxy that mutates headers in a surprising way”, etc.
- jrockway 8y agoYes, this is a set of problems that you need to care about. This is why there are so many Kubernetes-types working on "the service mesh". They all do a little bit too much for my taste (linkerd wants to provide its own Grafana dashboard and touch every TCP connection so as to measure it, Istio wants to molest _everything_ in your cluster and make even basic network connections be a full-fledged Kubernetes API object), but there are more reasonable solutions available. Envoy, specifically, is very good. You can use it for outgoing or incoming requests, and it can be configured to, say, retry retryable gRPC requests (solving your "the service is down" issue, at least if only one replica is down). It also scales cleanly to the amount of complexity you want; you can define your clusters in a file... or write a program that speaks a certain gRPC API so it can discover them for you automatically. Latency is always going to be a problem and moving things to another computer certainly doesn't decrease it. Everything these days has pretty good support for observability; opentracing to inspect slow requests, prometheus to see how things are doing in general. You can get a handle on it and it doesn't cost much. My team is moving from a PHP monolith that has so much framework code that an empty HTTP response takes 100ms minimum to generate. None of our microservices are that slow, even when 3 or 4 backends and gRPC-web <-> http translation are involved. But it does set an upper bound and that's a reasonable concern to have. Monolithic apps are not freed from latency; they read from disk, they talk to a database server, etc. So application developers already have this under control (or have filed it away in the "don't care" bucket); for example, every function in Go that does I/O probably takes a context. The context times out and it cancels your local operation just as easily as it cancels a remote operation. So I don't think this is a new concern, or one that people should be too afraid of, other than getting that last bit of performance out of the system. As for proxies inside your cluster intercepting traffic to other pods and mutating headers in surprising ways... I recommend not running one of those. (Yes, those magical service meshes are some of those. If you don't know why you need one, I recommend living without one until you know you need one. It may be never!) If you don't control your network, you won't have good luck talking to services on the network. It is orthogonal to microservices; you will be using the network more, so one that's bad will hurt you more. But in general, if you control your infrastructure and the nodes on it, you won't run into a "reverse proxy that mutates headers in a surprising way”. If there is one of those, I recommend killing it rather than not splitting up logical services into separate jobs.
- williamallthing 8y agoFYI Linkerd's Grafana dashboards are purely additive. If you don't like them, ignore them. We also work hard to make Linkerd incremental. It doesn't "touch every TCP connection" so much as "touch every TCP connection that you explicitly tell it to".
- jrockway 8y agoNot sure I agree. Running things in production is not free. There is resource usage, attack surface, learning curve, and the chance for something to go wrong. Especially prometheus; it is a high resource usage service using 2G of RAM on nodes that only have 3. I am happy to have one of those in my cluster, but an extra one "just because" actually costs something. Pods are not free either, at least not on Amazon. You get 18 pods per node (at least on t3.medium, which is what I use), and daemonsets quickly eat into that and make every additional node less useful as you increase cluster capacity. In a world where you're already running aws-node, kube-proxy, jaeger-agent, prometheus-node-exporter, and fluentd, you have to be judicious about the value of additional per-node services. I see the benefit in linkerd, but not all the extra stuff it comes with. Having an envoy cluster do gRPC load-balancing between services is enough; yes, you can't tcpdump the streams, it doesn't transparently add TLS, it doesn't configure itself through Kubernetes objects, and it doesn't quite insert the observability that linkerd does... but it does work well and comes with less tooling and resource cost.