4 ms·

Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an

by hrpnk 1y ago

Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.

zdc1 1y ago
I've tried and failed at tracing transactions that span multiple queues (with different backends). At the end I just published some custom metrics for the transaction's success count / failure count / duration and moved on my with life.
madduci 1y ago
I use Otel running in a GKE cluster and tracking Jenkins jobs, whose spans/traces can track long time running jobs pretty well
makeavish 1y ago
You can use SpanLinks to analyse your async processes. This guide might be helpful introduction: https://dev.to/clericcoder/mastering-trace-analysis-with-span-links-using-opentelemetry-and-signoz-a-practical-guidepart-2-1amc https://dev.to/clericcoder/mastering-trace-analysis-with-spa... Also SigNoz supports rendering practically unlimited number of spans in trace detail UI and allows filtering them as well which has been really useful in analyzing batch processes: https://signoz.io/blog/traces-without-limits/ https://signoz.io/blog/traces-without-limits/ You can further run aggregation on spans to monitor failures and latency. PS: I am SigNoz maintainer
- ai-christianson 1y ago
  Is this better than Honeycomb?
  - mdaniel 1y ago
    "Better" is always "for what metric" but if nothing else having the source code to the stack is always "better" IMHO even if one doesn't choose to self-host, and that goes double for SigNoz choosing a permissive license, so one doesn't have to get lawyers involved to run it --- While digging into Honeycomb's open source story, I did find these two awesome toys, one relevant to the otel discussion and one just neato https://github.com/honeycombio/refinery https://github.com/honeycombio/refinery (Apache 2) -- Refinery is a tail-based sampling proxy and operates at the level of an entire trace. Refinery examines whole traces and intelligently applies sampling decisions to each trace. These decisions determine whether to keep or drop the trace data in the sampled data forwarded to Honeycomb. https://github.com/honeycombio/gritql https://github.com/honeycombio/gritql (MIT) -- GritQL is a declarative query language for searching and modifying source code
- brunoqc 1y ago
  Is SpanLinks supported by jaeger?
  - makeavish 1y ago
    Yes Jaeger supports SpanLinks
dboreham 1y ago
It doesn't matter how long things take. The best way to understand this is to realize that OTel tracing (and all other similar things) are really "fancy logging systems". Some agent code emits a log message every time something happens (e.g. batch job begins, batch job ends). Something aggregates those log messages into some place they can be coherently scanned. Then something scans those messages generating some visualization you view. Everything could be done with text messages in text files and some awk script. A tracing system is just that with batteries included and a pretty UI. Understood this way it should now be clear why the duration of a monitored task is not relevant -- once the "begin task" message has been generated all that has to happen is the sampling agent remembers the span ID. Then when the "end task" message is emitted it has the same span ID. That way the two can be correlated and rendered as a task with some duration. There's always a way to propagate the span ID from place to place (e.g. in a http header so correlation can be done between processes/machines). This explains sibling comments about not being able to track tasks between workflows: the span ID wasn't propagated.
- imiric 1y ago
  That's a good way of looking at it, but it assumes that both start and end events will be emitted and will successfully reach the backend. What happens if one of them doesn't?
  - lijok 1y ago
    Depends on the visualization system. It can either not display the entire trace or communicate to the user that the start of the trace hasn’t been received or the trace hasn’t yet concluded. It really is just a bunch of structured log lines with a common attribute to tie them together.
  - candiddevmike 1y ago
    AIUI, there aren't really start or end messages, they're spans. A span is technically an "end" message and will have parent or child spans.
    - 1y ago
sethammons 1y ago
We had a hell of a time attempting to roll out OTel for that kind of work. Our scale was also billions of requests per day. We ended up taking tracing out of these jobs, and only using on requests that finish in short order, like UI web requests. For our longer jobs and fanout work, we started passing a metadata object around that appended timing data related that specific job and then at egress, would capture the timing metadata and flag abnormalities.
atypeoferror 1y ago
We have, and I won’t lie, it’s kinda a pain. The reality we discovered is that async flows are second-class citizens in otel tracing. We essentially propagate the context manually between APIs and lambdas, through HTTP headers, SNS/SQS and even storage. We also ended up pulling Tempo’s Parquet files (we’re self-hosting the grafana stack) into Redash to be able to do real analysis. We got a lot of great insights and were able to do a lot of tuning that would have been impossible otherwise, but it was quite an investment. Would love to know if there is anything out there that would have made this less painful.
drzaiusx11 1y ago
yes, the only tricky part for us were the short async jobs where the otel queue didn’t have time to process its events before the tasks exited. we just added a drain wait loop in our at_exit callbacks

by hrpnk 1y ago

Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.

zdc1 1y ago

I've tried and failed at tracing transactions that span multiple queues (with different backends). At the end I just published some custom metrics for the transaction's success count / failure count / duration and moved on my with life.

madduci 1y ago

I use Otel running in a GKE cluster and tracking Jenkins jobs, whose spans/traces can track long time running jobs pretty well

makeavish 1y ago

You can use SpanLinks to analyse your async processes. This guide might be helpful introduction: https://dev.to/clericcoder/mastering-trace-analysis-with-span-links-using-opentelemetry-and-signoz-a-practical-guidepart-2-1amc https://dev.to/clericcoder/mastering-trace-analysis-with-spa... Also SigNoz supports rendering practically unlimited number of spans in trace detail UI and allows filtering them as well which has been really useful in analyzing batch processes: https://signoz.io/blog/traces-without-limits/ https://signoz.io/blog/traces-without-limits/ You can further run aggregation on spans to monitor failures and latency. PS: I am SigNoz maintainer

ai-christianson 1y ago
Is this better than Honeycomb?
- mdaniel 1y ago
  "Better" is always "for what metric" but if nothing else having the source code to the stack is always "better" IMHO even if one doesn't choose to self-host, and that goes double for SigNoz choosing a permissive license, so one doesn't have to get lawyers involved to run it --- While digging into Honeycomb's open source story, I did find these two awesome toys, one relevant to the otel discussion and one just neato https://github.com/honeycombio/refinery https://github.com/honeycombio/refinery (Apache 2) -- Refinery is a tail-based sampling proxy and operates at the level of an entire trace. Refinery examines whole traces and intelligently applies sampling decisions to each trace. These decisions determine whether to keep or drop the trace data in the sampled data forwarded to Honeycomb. https://github.com/honeycombio/gritql https://github.com/honeycombio/gritql (MIT) -- GritQL is a declarative query language for searching and modifying source code
brunoqc 1y ago
Is SpanLinks supported by jaeger?
- makeavish 1y ago
  Yes Jaeger supports SpanLinks

dboreham 1y ago

It doesn't matter how long things take. The best way to understand this is to realize that OTel tracing (and all other similar things) are really "fancy logging systems". Some agent code emits a log message every time something happens (e.g. batch job begins, batch job ends). Something aggregates those log messages into some place they can be coherently scanned. Then something scans those messages generating some visualization you view. Everything could be done with text messages in text files and some awk script. A tracing system is just that with batteries included and a pretty UI. Understood this way it should now be clear why the duration of a monitored task is not relevant -- once the "begin task" message has been generated all that has to happen is the sampling agent remembers the span ID. Then when the "end task" message is emitted it has the same span ID. That way the two can be correlated and rendered as a task with some duration. There's always a way to propagate the span ID from place to place (e.g. in a http header so correlation can be done between processes/machines). This explains sibling comments about not being able to track tasks between workflows: the span ID wasn't propagated.

imiric 1y ago
That's a good way of looking at it, but it assumes that both start and end events will be emitted and will successfully reach the backend. What happens if one of them doesn't?
- lijok 1y ago
  Depends on the visualization system. It can either not display the entire trace or communicate to the user that the start of the trace hasn’t been received or the trace hasn’t yet concluded. It really is just a bunch of structured log lines with a common attribute to tie them together.
- candiddevmike 1y ago
  AIUI, there aren't really start or end messages, they're spans. A span is technically an "end" message and will have parent or child spans.
  - 1y ago

sethammons 1y ago

We had a hell of a time attempting to roll out OTel for that kind of work. Our scale was also billions of requests per day. We ended up taking tracing out of these jobs, and only using on requests that finish in short order, like UI web requests. For our longer jobs and fanout work, we started passing a metadata object around that appended timing data related that specific job and then at egress, would capture the timing metadata and flag abnormalities.

atypeoferror 1y ago

We have, and I won’t lie, it’s kinda a pain. The reality we discovered is that async flows are second-class citizens in otel tracing. We essentially propagate the context manually between APIs and lambdas, through HTTP headers, SNS/SQS and even storage. We also ended up pulling Tempo’s Parquet files (we’re self-hosting the grafana stack) into Redash to be able to do real analysis. We got a lot of great insights and were able to do a lot of tuning that would have been impossible otherwise, but it was quite an investment. Would love to know if there is anything out there that would have made this less painful.

drzaiusx11 1y ago

yes, the only tricky part for us were the short async jobs where the otel queue didn’t have time to process its events before the tasks exited. we just added a drain wait loop in our at_exit callbacks