4 ms·
Looks very nice! Have built something similar by forking the Datadog tracing library to also send each Span as a JSON to Kinesis Firehose, and then to S3. From
by cube2222 4y ago
Looks very nice!
Have built something similar by forking the Datadog tracing library to also send each Span as a JSON to Kinesis Firehose, and then to S3. From there you can easily query it using AWS Glue (for schema inference) + Athena (as the actual SQL engine).
It's indeed really nice for digging deep or doing performance analysis, though I'd say for 99% of cases, the Datadog APM UI + query editor is much preferred. In practice, we'd occasionally see that we can't express a query in the DD UI and run it through SQL, but it happens very rarely. It's also nice for cheaply achieving 100% sampling if you ever have the need to find a single specific request that's happenned.
I wonder what the UX would be of using something like this with a tool like Metabase for the UI (with an actually fast query engine underneath, like Timescale or ClickHouse).
- chrnola 4y agoSomewhat relatedly, the Uptrace[1] project is building a tool for querying observability data from a ClickHouse storage layer. [1]: https://get.uptrace.dev/ https://get.uptrace.dev/
- derN3rd 4y agoI've recently stumbled upon Signoz.io which also builds ontop of ClickHouse, but this seems interesting as well. Wondering why uptrace is creating modules for their node/web tracer, while it does the same thing as OTEL instrumentations
- ramonguiu 4y ago(Promscale team member) That's great! SQL definitely opens lots of possibilities to analyze the data. And lots of visualization tools already integrate with PostgreSQL/TimescaleDB which Promscale uses to store data so you have a lot of options to pick from. Would you mind sharing some of the problems you've had to use SQL for?
- cube2222 4y agoHey, two main use cases: 1. Find a trace where there is a span of type A with tag `xyz: true`, that has a direct child span of type B with tag `abc: 42`, that has at least one non-direct child span of type C with tag `lmn: forty-two`. So basically complex queries where I'm looking for a trace while making assertions about tags in different spans, possibly spans that are very far from each other in the trace (non-direct child, just joining on trace_id in that case, not child.parent_id = parent.span_id). 2. Analytical performance analysis across all spans of a certain type in a specified time span (a week for example). I have spans of type A, each of these spans has multiple children of type B, now I want to make some calculations involving the duration of A, the count of B, durations of B, and then slice and dice (group by) this by specific tags in those spans.
- ramonguiu 4y agoThanks for sharing. Makes total sense. As soon as you have to query across span attributes AND span parent/child relationships you need a more sophisticated query language.