4 ms·

At a recent start-up like situation we paid licensing for a monitoring and metrics provider that provided out of the box tracing infrastructure for us. It turn

At a recent start-up like situation we paid licensing for a monitoring and metrics provider that provided out of the box tracing infrastructure for us. It turned out that the licensing was quite expensive for what we where getting (couldn't even get outlier traces), we didn't have much control of our infrastructure (couldn't control the sampling rate in such a way to keep all the traces we needed), and ultimately the learning curve for making the infrastructure scalable was a challenge as well. Even after setting up tracing in our entire system successfully, I ultimately found it of minimal value and we ended up relying on ELK and log aggregation for more insight. Tbh, it was a disappointing experience and just reading how much custom dev was required in this post to make tracing work for you guys makes me disappointed. Given the frame of work I do (consulting), I doubt tracing will ever prove more valuable than spending money on log aggregation and other monitoring efforts and tools.

matsemann 6y ago
A quick win I've seen and myself implemented as a consultant, is the use of correlationId in some form. If an incoming request has a correlation-id-header, then use it. If not generate a unique one. When calling other services make sure to include the current correlationId, and thus it spreads throughout the services for a given request. Most logging frameworks allow you to add something to every log message, so add the correlationId there. Then it's easy to track between multiple services. It also makes it easy to distinguish which log messages belong to a given request for a single service, so even without lots of services it can be a nice little feature.
- coredog64 6y ago
  It’s easy for this to fall over. We were doing this where the entry point would make hundreds or even thousands of calls. Unfortunately, I couldn’t convince anyone that you needed spans and not just traces

matsemann 6y ago

A quick win I've seen and myself implemented as a consultant, is the use of correlationId in some form. If an incoming request has a correlation-id-header, then use it. If not generate a unique one. When calling other services make sure to include the current correlationId, and thus it spreads throughout the services for a given request. Most logging frameworks allow you to add something to every log message, so add the correlationId there. Then it's easy to track between multiple services. It also makes it easy to distinguish which log messages belong to a given request for a single service, so even without lots of services it can be a nice little feature.

coredog64 6y ago
It’s easy for this to fall over. We were doing this where the entry point would make hundreds or even thousands of calls. Unfortunately, I couldn’t convince anyone that you needed spans and not just traces