> Taken together, the issues were problematic enough that tracing was underowned and arguably unowned for years. Some individuals did work in their spare time t

> Taken together, the issues were problematic enough that tracing was underowned and arguably unowned for years. Some individuals did work in their spare time to keep the lights on or improve things, but the lack of obvious value from tracing led to a vicious cycle where the high barrier to getting value out of tracing made it hard to fund organizationally, which made it hard to make tracing more usable. Omg, this is 100% the situation at $DAYJOB. I actually think this is generally true for any new infrastructure that devs are expected to interface with. E.g. for metrics, most engineers I work with also don't really understand Prometheus, despite it being incredibly useful with a robust query language, but looking at Grafana graphs gives them a good enough picture. The big "wtf" moment with tracing is always the sampling, its really hard to build a good sampling system that collects meaningful traces ahead of time, much less explain it. The article's "problems" list rings really true. For my story: we tried to roll our own naive system backed by ELK that piggybacked off our logging stack, which largely failed due to time/resource constraints (kibana and elasticsearch actually worked fairly well for some types of aggregations, but some really useful visualizations like a service graph aren't possible). I agree with the post in stating that building a good distributed tracing system from scratch definitely isn't trivial, but it's not very difficult given the tools that exist today. Also, the article mentions things like clock skew not mattering in practice, which if you were to approach this from a "blank slate" would definitely be a bit counter-intuitive. Anyway, due to resourcing constraints (i.e. we had no observability team, and even if we did they wouldn't have worked on this particular problem 1+ years ago), we use a vendor (Lightstep) which is generally solid & requires very little maintenance, but there are still issues where simple conceptual queries like "show me traces that had this tag" aren't possible. E.g. they don't support things like querying `anno_index` from the article for historical data, you can only query in-memory data (so last ~10-20 mins, helpful for immediate oncall scenarios but not much else). The really interesting thing about distributed tracing is that its really still in its infancy in terms of potential applications. The article focuses on performance/oncall scenarios, but we: - Built a post-CI flow that captures integration test failures and links developers to a snap shot so they can see the exact service their test failed - no digging through logs. - Can see traces during local development. - In OpenTracing, there's the pattern of injecting untyped "baggage items" that make it through to downstream systems. We integrated this into our _logging_ clients, so you get free metadata from upstream services in logs without having to redeploy N microservices (this lets us get around some of the query limitations in Lightstep). - I'm also shooting around ideas to leverage baggage items to inject sandbox data for writing tests against plolyglot, manyrepo services. This lets test data live next to test driver code (e.g. a Go service calls into a Python service & we want to mock the Python service's test data but don't want to have to update + merge a change to that repo). Kudos to Twitter for a great tech + organizational initiative.