3 ms·
Tracing should be sampled at a rate that makes sense for the dollar impact of an error in the service, versus the dollar cost of your tracing system. $thousand
by andrewstuart2 5y ago
Tracing should be sampled at a rate that makes sense for the dollar impact of an error in the service, versus the dollar cost of your tracing system.
$thousands per transaction? Sample 100%. Cheaper to refund a customer than debug the problem every time? Then sample. You'll have plenty of sampled traces with the bug/error shown as soon as the problem is expensive enough to matter.
Metrics are great for demonstrating a problem exists (and alerting), and tracing is great for helping you understand how and why it happened once it's at a big enough scale to matter.
Beyond that, with a lot of what's coming out of Open Telemetry, you can get fancier like sampling a higher percent of traces to a localized processor middleware that then filters out any traces you doesn't care about (e.g. dropping non-errors or low-quantile latencies).
- xg15 5y agoYeah, but once again, this requires you to already know there is an error, have a dollar impact calculated, etc. There is a lot of talk in this thread about the performance impact of logging. I don't see how sampling 100% wouldn't have a far greater performance impact.