3 ms·

This is less likely an issue with OTel itself but just that AWS still doesn't natively support it. The x-ray daemon is similar to the collector, but it's run fo

This is less likely an issue with OTel itself but just that AWS still doesn't natively support it. The x-ray daemon is similar to the collector, but it's run for free in the background while the OTel collector is in the user's lambda resources. It was fun having freedom to work on OTel within x-ray to provide better instrumentation users but was always frustrating how whenever pushing for more native support within the team such as ingesting otlp directly, the answer was always no since it meant losing control. Note that I don't think the actual reason is to overcharge users (otherwise why invest in things like graviton?), though the result does end up being that.

hinkley 3y ago
Ours ends up in Prometheus and I wish I had just used that instead. But one of our coworkers was gung ho about tracing, so I obliged not knowing what I had gotten myself into. And the moment I turned tracing on is got quenched because in a mature app the amount of tracing you want to do is about ten times the limit for messages per second per sender. It’s a toy and that functionality is now turned off, though the call graph changes to support it are still there and make our stack traces ugly.
- phillipcarter 3y ago
  I'm definitely curious how data volume was a big concern for traces. I work with several customers with enormous trace volume, and while they absolutely need to do tail sampling so their bill doesn't explode, the actual export of spans from their services isn't typically the problem to solve.
  - hinkley 3y ago
    We don’t have a monorepo so I can’t tell you how many lines of code we have anymore, but if it’s less than 300k lines I’ll eat my hat. It’s gotta be half a million. It’s a beast. No insane fanout, but unless I very carefully cherry picked a couple of points in the code instead of piggy backing in our already pretty good telemetry choke points, then I’d hit the limit even in preproduction. We don’t historically do a lot of things well, but everything does correlationID propagation properly and some have pretty good telemetry, and have done since before I got here. Most other things in the “well” column were heavy lifting by myself and a handful of other instigators, some of whom gave up and left. That’s probably what upsets me so much about OTEL. It made one of our strengths into another thing to complain about.
- bilalq 3y ago
  Tracing is pretty life-changing when it works well though. It's worth spending some engineering resources to try and get it if you can, even if it's at a lower sampled rate.

hinkley 3y ago

Ours ends up in Prometheus and I wish I had just used that instead. But one of our coworkers was gung ho about tracing, so I obliged not knowing what I had gotten myself into. And the moment I turned tracing on is got quenched because in a mature app the amount of tracing you want to do is about ten times the limit for messages per second per sender. It’s a toy and that functionality is now turned off, though the call graph changes to support it are still there and make our stack traces ugly.

phillipcarter 3y ago
I'm definitely curious how data volume was a big concern for traces. I work with several customers with enormous trace volume, and while they absolutely need to do tail sampling so their bill doesn't explode, the actual export of spans from their services isn't typically the problem to solve.
- hinkley 3y ago
  We don’t have a monorepo so I can’t tell you how many lines of code we have anymore, but if it’s less than 300k lines I’ll eat my hat. It’s gotta be half a million. It’s a beast. No insane fanout, but unless I very carefully cherry picked a couple of points in the code instead of piggy backing in our already pretty good telemetry choke points, then I’d hit the limit even in preproduction. We don’t historically do a lot of things well, but everything does correlationID propagation properly and some have pretty good telemetry, and have done since before I got here. Most other things in the “well” column were heavy lifting by myself and a handful of other instigators, some of whom gave up and left. That’s probably what upsets me so much about OTEL. It made one of our strengths into another thing to complain about.
bilalq 3y ago
Tracing is pretty life-changing when it works well though. It's worth spending some engineering resources to try and get it if you can, even if it's at a lower sampled rate.