3 ms·

I think giving metrics and logging a location in a trace is really useful. But I still dislike OTel every time I have to deal with it.

hinkley 2y ago
You can’t do fine grained tracing in OTEL because if you hit 500 spans in a single trace it starts dropping the trace. Basically a toy solution for brownfield work.
- phillipcarter 2y ago
  ...huh? I work with customers who (through a mistake) have created literally multi-million span traces using OTel. Are you referring to a particular backend?
  - hinkley 2y ago
    AWS
    - phillipcarter 2y ago
      Well that's a shame, I'm going to ask some folks about that. 500 spans per trace is ridiculously small and I can't imagine any good reason to have that limitation since it's just not that big of a footprint. OTel doesn't define any limits on the # of spans in a trace (nor the # of attributes on a span!) but it will be bound by the limits of whatever backend you use. In the case of the one I work for, we do limit the total size of a span to be 1MB or less with 64KB per attribute before truncation. Other backends have different limitations. This is the first I've heard of such a small limitation on the total number of spans in a trace though. Traces are just (basically) collections of structured logs with in-built correlation IDs. I can't imagine why you'd limit them like this.
      - hinkley 2y ago
        That was two years ago (we tried spans before metrics), so it’s fuzzy. I believe the collector sidecar was fine with it but the backend was not, which complicated debugging. There’s not a clear feedback path in OpenTelemetry that we could find. I completely forgot to mention the tendency toward silent failures. That’s a cardinal sin for telemetry. I would take it out back and shoot it for that fact alone. The other problem I noticed looking at the wire protocol was that the data for the parent trace doesn’t seem to get sent until the trace closes. That seems like a bookkeeping nightmare to me. There should be a start of trace packet and an update at the end. I shouldn’t have finished spans showing up before the parent trace has been registered. And that’s what it looked like in the dumps my OPs people sent me to debug.
    - mdaniel 2y ago
      Practically a given outcome, then; we could knock their Managed Prometheus offering off the Internet on the regular. It was just laughable for a company that prides itself in one trillion IAM transactions to 429 some metric ingest
- pranay01 2y ago
  As mentioned by philip below, 500 spans is a very small amount. I have seen customers send 1000s of spans in a trace very easily
- IneffablePigeon 2y ago
  This is just not true. We have traces with hundreds of thousands of spans. Those are not very readable but that’s another problem.
  - PeterCorless 2y ago
    How are you storing them, and what do you use to read/visualize/analyze them? I'd imagine just putting them up in a UI becomes a needle-in-a-haystack issue. Are you programmatically analyzing them?
    - IneffablePigeon 2y ago
      Honeycomb. For shorter traces (most of them), a waterfall view is great. For those long ones, we try to split them up if it makes sense but you can also just run queries scoped to that trace to answer questions about it (how many of the spans are db queries, how many are this query, are they quick, etc etc)

hinkley 2y ago

You can’t do fine grained tracing in OTEL because if you hit 500 spans in a single trace it starts dropping the trace. Basically a toy solution for brownfield work.

phillipcarter 2y ago
...huh? I work with customers who (through a mistake) have created literally multi-million span traces using OTel. Are you referring to a particular backend?
- hinkley 2y ago
  AWS
  - phillipcarter 2y ago
    Well that's a shame, I'm going to ask some folks about that. 500 spans per trace is ridiculously small and I can't imagine any good reason to have that limitation since it's just not that big of a footprint. OTel doesn't define any limits on the # of spans in a trace (nor the # of attributes on a span!) but it will be bound by the limits of whatever backend you use. In the case of the one I work for, we do limit the total size of a span to be 1MB or less with 64KB per attribute before truncation. Other backends have different limitations. This is the first I've heard of such a small limitation on the total number of spans in a trace though. Traces are just (basically) collections of structured logs with in-built correlation IDs. I can't imagine why you'd limit them like this.
    - hinkley 2y ago
      That was two years ago (we tried spans before metrics), so it’s fuzzy. I believe the collector sidecar was fine with it but the backend was not, which complicated debugging. There’s not a clear feedback path in OpenTelemetry that we could find. I completely forgot to mention the tendency toward silent failures. That’s a cardinal sin for telemetry. I would take it out back and shoot it for that fact alone. The other problem I noticed looking at the wire protocol was that the data for the parent trace doesn’t seem to get sent until the trace closes. That seems like a bookkeeping nightmare to me. There should be a start of trace packet and an update at the end. I shouldn’t have finished spans showing up before the parent trace has been registered. And that’s what it looked like in the dumps my OPs people sent me to debug.
  - mdaniel 2y ago
    Practically a given outcome, then; we could knock their Managed Prometheus offering off the Internet on the regular. It was just laughable for a company that prides itself in one trillion IAM transactions to 429 some metric ingest
pranay01 2y ago
As mentioned by philip below, 500 spans is a very small amount. I have seen customers send 1000s of spans in a trace very easily
IneffablePigeon 2y ago
This is just not true. We have traces with hundreds of thousands of spans. Those are not very readable but that’s another problem.
- PeterCorless 2y ago
  How are you storing them, and what do you use to read/visualize/analyze them? I'd imagine just putting them up in a UI becomes a needle-in-a-haystack issue. Are you programmatically analyzing them?
  - IneffablePigeon 2y ago
    Honeycomb. For shorter traces (most of them), a waterfall view is great. For those long ones, we try to split them up if it makes sense but you can also just run queries scoped to that trace to answer questions about it (how many of the spans are db queries, how many are this query, are they quick, etc etc)