4 ms·
I hoped from the intro that they would be suggesting an approach that was standard at Google a decade ago, that the rest of the world never adopted. But no suc
by btilly 8y ago
I hoped from the intro that they would be suggesting an approach that was standard at Google a decade ago, that the rest of the world never adopted.
But no such luck.
However if anyone is creating a distributed product today in 2019, here is an excellent suggestion. Free of charge.
In whatever common code you have for your RPCs, add the ability for an RPC message to be traced. If it is traced, then you will log when it is received, when you reply to it, and also cause all RPC messages you send while handling it to also be traced.
At your front end you can then pick an arbitrary fraction of your traffic, say 1%, and trace it.
Tracing such a small fraction of traffic means that you can instrument it in real time, gather the logs, and create a complete view of how each web request cascaded through your system. And then whenever you run into a slow request that only happens for a small fraction of your traffic you can find a request that was both instrumented and slow, and then dig into those rare problems that happen here and there, then add up.
If you don't do this, then good luck tracking down the performance problems that happen 5% of the time, 3 RPC calls from the front.
- Groxx 8y agoThis is basically what https://www.jaegertracing.io/ https://www.jaegertracing.io/ / https://opentracing.io/ https://opentracing.io/ are intended to do. Not sure who widespread they are, but e.g. Uber uses it: https://eng.uber.com/distributed-tracing/ https://eng.uber.com/distributed-tracing/
- jsiepkes 8y agoIn my experience its pretty common? Aside from projects like https://opencensus.io https://opencensus.io a lot of other technologies not primarily about tracing like service proxies (Envoy, linkerd, etc.) also support various types of tracing based on message id's.
- btown 8y agoNewRelic's Python library monkey-patches low-level urllib3 etc. libraries to accomplish something very similar to this, albeit without communicating a trace ID/intent to the recipient (which you'd want to do if you controlled the endpoint being called): https://github.com/edmorley/newrelic-python-agent/blob/master/newrelic/newrelic/hooks/external_urllib3.py https://github.com/edmorley/newrelic-python-agent/blob/maste... (note: linked repo is not open source)
- RexM 8y agoI believe this is the idea behind AWS X-Ray. I was researching it recently for work but haven’t had a chance to implement anything with it, yet.
- stingraycharles 8y agoThis idea is quite popular in distributed / cloud systems nowadays, and there are tons of open source projects. Every cloud vendor has a product that does it (AWS X-Ray, Google Stackdriver Trace, etc), and there are plenty of open source products (Jaeger, OpenTracing), as well as third party vendors (we're using Datadog ourselves). Having said that, I completely agree with this being very important, and it is the backbone of our ability to debug issues. I would add that it is important to have the ability to correlate things as well, specifically your logs and traces ("give me all the logs that relate to this trace X").
- GorgeRonde 8y ago> In whatever common code you have for your RPCs, add the ability for an RPC message to be traced. Because I realized API calls (which are just RPCs) need this (which is a defacto feature of programming language through exceptions and stacktraces), I now think it is better to start at the programming language level, write your function normally, and then compile it to call/respond as an API endpoint. Actually I think it's best to write a monolith and then cut it s different pieces into independent microservices using code generation/manipulation, even though one's first intention is to build an ecosystem of microservices.