3 ms·

Turns out X-Ray is sampling service, it drops data. You can use it to debug recurring problems, but it's no use when debugging a particular incident with the c

by DenisM 10y ago

Turns out X-Ray is sampling service, it drops data. You can use it to debug recurring problems, but it's no use when debugging a particular incident with the customer on the phone. Bummer. To provide a performant and cost-effective experience, X-Ray does not collect data for every request that is sent to an application. Instead, it collects data for a statistically significant number of requests. X-Ray should not be used as an audit or compliance tool because it does not guarantee data completeness. https://aws.amazon.com/xray/faqs/ https://aws.amazon.com/xray/faqs/

dkuebric 10y ago
Yeah, there's a big difference between something that's pure distributed tracing (this) and something that does metrics/monitoring as well (TraceView, New Relic, ...). Sampling is a valid way to make distributed tracing scalable and performant, but it can also limit the use-cases for the data. I imagine this being used for episodic debug cases, where one could turn up the sample rate and pay only for the traces captured/queried during an incident. But you would need to do your monitoring and trending separately. I wouldn't be surprised if they eventually start integrating this better with CloudWatch for that reason, though it doesn't seem to be doing any of that today. Disclosure: I work on TraceView (traceview.solarwinds.com) which is distributed tracing based APM product. We're inspired by Google Dapper and x-trace, both mentioned elsewhere in this thread.
- DenisM 10y ago
  > difference between something that's pure distributed tracing (this) and something that does metrics/monitoring as well (TraceView, New Relic, ...) I don't see the distinction you're making, as neither seems to record all traces for accurate audit. Am I missing it?
  - dkuebric 10y ago
    I was speaking to a more general monitoring approach where you might want to know p99 latency, request volume, error rate, etc, for each service to use in alerting and trending. This is a common use-case for application monitoring that isn't addressed well by a pure tracing approach. If you're looking specifically for the 100% audit trail case, I'd look at DynaTrace or potentially Instana.
tonyhb 10y ago
With Zipkin and OpenTracing you can force spans to be recorded despite the sampling, so if you know something errors (middelware, catches etc.) you can almost guarantee that you'll record your trace if you set things up correctly.
nathankunicki 10y ago
I was just in the 5:30pm re:Invent session at The Venetian. The service works via the SDK's in various languages, that report tracing information to a local daemon that runs on the host, over UDP. The daemon then batches the data, applies sampling (Which is configurable, all the way to 100% - report everything), and sends it en-masse to AWS. Edit: See Sampling Rules section here - http://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-nodejs.html#xray-sdk-nodejs-configuration http://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-nod...
- svdgraaf 10y ago
  geh, I was there as well :) I concur, it was specifically stated that 100% sampling is an option (and is by default?)

by DenisM 10y ago

dkuebric 10y ago

Yeah, there's a big difference between something that's pure distributed tracing (this) and something that does metrics/monitoring as well (TraceView, New Relic, ...). Sampling is a valid way to make distributed tracing scalable and performant, but it can also limit the use-cases for the data. I imagine this being used for episodic debug cases, where one could turn up the sample rate and pay only for the traces captured/queried during an incident. But you would need to do your monitoring and trending separately. I wouldn't be surprised if they eventually start integrating this better with CloudWatch for that reason, though it doesn't seem to be doing any of that today. Disclosure: I work on TraceView (traceview.solarwinds.com) which is distributed tracing based APM product. We're inspired by Google Dapper and x-trace, both mentioned elsewhere in this thread.

DenisM 10y ago
> difference between something that's pure distributed tracing (this) and something that does metrics/monitoring as well (TraceView, New Relic, ...) I don't see the distinction you're making, as neither seems to record all traces for accurate audit. Am I missing it?
- dkuebric 10y ago
  I was speaking to a more general monitoring approach where you might want to know p99 latency, request volume, error rate, etc, for each service to use in alerting and trending. This is a common use-case for application monitoring that isn't addressed well by a pure tracing approach. If you're looking specifically for the 100% audit trail case, I'd look at DynaTrace or potentially Instana.

tonyhb 10y ago

With Zipkin and OpenTracing you can force spans to be recorded despite the sampling, so if you know something errors (middelware, catches etc.) you can almost guarantee that you'll record your trace if you set things up correctly.

nathankunicki 10y ago

I was just in the 5:30pm re:Invent session at The Venetian. The service works via the SDK's in various languages, that report tracing information to a local daemon that runs on the host, over UDP. The daemon then batches the data, applies sampling (Which is configurable, all the way to 100% - report everything), and sends it en-masse to AWS. Edit: See Sampling Rules section here - http://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-nodejs.html#xray-sdk-nodejs-configuration http://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-nod...

svdgraaf 10y ago
geh, I was there as well :) I concur, it was specifically stated that 100% sampling is an option (and is by default?)