5 ms·
How do you send all errors? The way tracing works, as I understand it, is that each microservice gets a trace header which indicates if it should sample and eac
by fastest963 3y ago
How do you send all errors? The way tracing works, as I understand it, is that each microservice gets a trace header which indicates if it should sample and each microservice itself records traces. If microservice A calls microservice B and B returns successfully but then A ends up erroring, how can you retroactively tell B to record the trace that it already finished making and threw away? Or do you just accept incomplete traces when there are errors?
- phamilton 3y agohttps://opentelemetry.io/docs/concepts/sampling/ https://opentelemetry.io/docs/concepts/sampling/ describes it as Head/Tail sampling, but in practice with vendors I see it as Ingestion sampling and Index sampling. We send all our spans to be ingested, but have a sample rate on indexing. That allows us to override the sampling at index and force errors and other high value spans to always be indexed.
- fastest963 3y agoMaybe the Go client doesn't support that? https://opentelemetry.io/docs/instrumentation/go/sampling/ https://opentelemetry.io/docs/instrumentation/go/sampling/
- phillipcarter 3y agoIt does, but the docs aren't clear on that yet. TraceIdRatioBased is the "take X% of traces" sampler that all SDKs support today.
- sigwinch28 3y agoYou can do head-based sampling and tail-based sampling. With head sampling, the first service in the request chain can make the decision about whether to trace, which can reduce tracing overhead on services further down. With tail-based sampling, the tracing backend can make a determination about whether to persist the trace after the trace has been collected. This has tracing overheads, but allows you to make decisions like “always keep errors”.