4 ms·

Has anyone found a solution to automatically adding spans automatically for nodejs applications that doesn't involve littering startSpan or startActiveSpans eve

Has anyone found a solution to automatically adding spans automatically for nodejs applications that doesn't involve littering startSpan or startActiveSpans everywhere? I'm currently looking into monkey patching, but that seems dirty.

pondidum 3y ago
Depends what you're trying to accomplish. We're using middleware to start spans for http and message handlers, and then adding `startSpan` where we need. I don't see a problem with `startSpan` everywhere as it's not much noiser than the `log.info` that would be there instead if we didn't have otel.
- _boffin_ 3y ago
  I started my APM journey when using Spring w/ Java and fell in love with how i can trace the entire flow through the entire application and then just config on how many spans i want to send over and what sample rate i want collected. I'd love to accomplish that. What you mentioned is all nice and well (optimal route), but right now, i'm working with some applications that needs it, but has i don't even know how many methods / classes that i'd need to go through and implement it on.
- Spivak 3y ago
  I think my major pain point with them being everywhere is unlike a log they push a new stack frame and closure. They make tracebacks so much more annoying, yes I know the error happened in a start span, thank you. I wish observability actually observed more than it contributed. Once https://peps.python.org/pep-0669/ https://peps.python.org/pep-0669/ is available I'm gonna try my damndest to get otel working through it. Just give me a config file that says what functions you're interested in and I'll do the rest.
  - pondidum 3y ago
    Agreed the amount of stuff showing in stack traces can be annoying - but it does depend on language. I agree 100% in javascript/typescript its annoying, and I would love to get rid of them, In go however, there isn't an extra stack frame. Nor in C# thinking about it. The config file of functions to trace is a really interesting idea. How would you handle wanting to add data to the spans from inside those functions though? e.g. I want to add all kinds of props to the spans that can't be known except inside the function executing.
phillipcarter 3y ago
There isn't a way to automatically add spans for arbitrary functions in Node, no. There's currently a proposal to add some stuff that makes creating the span easier to make it auto-closing based on scope. FWIW the philosophy here is that observability is a part of the application rather than something separate. That's distinctly different from the APM philosophy, which is that a separate process "does the observability" and your app is "clean" from that. I think there's quite a benefit to manually instrumenting in your codebase intentionally rather than having an automated process do it for you. But I can understand not wanting to go through and do that.
- CubsFan1060 3y ago
  That's an interesting distinction. Given some of the older, larger code bases, manually instrumenting seems like it would require a reasonably large effort. I think that's one of the huge benefits that some of the APM's bring. I wonder if there is any reason OTEL can't have both.
  - _boffin_ 3y ago
    For stuff on the JVM, you just load the APM plugin and you're pretty much good to go as it inspects the bytecode, i believe.
    - CubsFan1060 3y ago
      Oh, right. I guess my question was more around "why can't OTEL _also_ have something that does that"?
  - phillipcarter 3y ago
    OTel kind of has both today. Most languages support autoinstrumentation via either an agent, libraries, or both. So for example, if you've got a big Spring Boot app, it'll instrument requests/responses and DB calls for you. Some languages also have lightweight "sprinkle some spans on it" things you can put in code, like Java method annotations. But none have truly automatic "span for every method call" instrumentation.
vadman97 3y ago
Have you seen the OTEL auto instrumentation package[1]? It supports a number of common frameworks (as middleware) and libraries (via monkeypatches) setting up spans for you to capture things like external http requests, database/cache queries, etc. [2] [1] https://opentelemetry.io/docs/instrumentation/js/automatic/ https://opentelemetry.io/docs/instrumentation/js/automatic/ [2] https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/metapackages/auto-instrumentations-node#supported-instrumentations https://github.com/open-telemetry/opentelemetry-js-contrib/t...
- _boffin_ 3y ago
  Tested those before, but still doesn't get what i'm looking for, which is inside the black box without manually adding the spans. The implementations for bunyan or winston would come the closet, but still doesn't achieve what i'm looking for.
  - eep_social 3y ago
    I think the Ruby auto-instrumentation library does wrap practically every method but we ended up finding it too verbose. I think this is a bit of a goldilocks problem insofar as getting it just right is not easy and varies per application.
    - _boffin_ 3y ago
      One of the things i think that can be done is also limit the number of spans collected via configuration.
      - eep_social 3y ago
        Aye, but then you’re trading off between a centralized config that does sampling vs customized per-service (or whatever boundary) configs. There’s no free lunch, was mostly my point.

pondidum 3y ago

Depends what you're trying to accomplish. We're using middleware to start spans for http and message handlers, and then adding `startSpan` where we need. I don't see a problem with `startSpan` everywhere as it's not much noiser than the `log.info` that would be there instead if we didn't have otel.

_boffin_ 3y ago
I started my APM journey when using Spring w/ Java and fell in love with how i can trace the entire flow through the entire application and then just config on how many spans i want to send over and what sample rate i want collected. I'd love to accomplish that. What you mentioned is all nice and well (optimal route), but right now, i'm working with some applications that needs it, but has i don't even know how many methods / classes that i'd need to go through and implement it on.
Spivak 3y ago
I think my major pain point with them being everywhere is unlike a log they push a new stack frame and closure. They make tracebacks so much more annoying, yes I know the error happened in a start span, thank you. I wish observability actually observed more than it contributed. Once https://peps.python.org/pep-0669/ https://peps.python.org/pep-0669/ is available I'm gonna try my damndest to get otel working through it. Just give me a config file that says what functions you're interested in and I'll do the rest.
- pondidum 3y ago
  Agreed the amount of stuff showing in stack traces can be annoying - but it does depend on language. I agree 100% in javascript/typescript its annoying, and I would love to get rid of them, In go however, there isn't an extra stack frame. Nor in C# thinking about it. The config file of functions to trace is a really interesting idea. How would you handle wanting to add data to the spans from inside those functions though? e.g. I want to add all kinds of props to the spans that can't be known except inside the function executing.

phillipcarter 3y ago

There isn't a way to automatically add spans for arbitrary functions in Node, no. There's currently a proposal to add some stuff that makes creating the span easier to make it auto-closing based on scope. FWIW the philosophy here is that observability is a part of the application rather than something separate. That's distinctly different from the APM philosophy, which is that a separate process "does the observability" and your app is "clean" from that. I think there's quite a benefit to manually instrumenting in your codebase intentionally rather than having an automated process do it for you. But I can understand not wanting to go through and do that.

CubsFan1060 3y ago
That's an interesting distinction. Given some of the older, larger code bases, manually instrumenting seems like it would require a reasonably large effort. I think that's one of the huge benefits that some of the APM's bring. I wonder if there is any reason OTEL can't have both.
- _boffin_ 3y ago
  For stuff on the JVM, you just load the APM plugin and you're pretty much good to go as it inspects the bytecode, i believe.
  - CubsFan1060 3y ago
    Oh, right. I guess my question was more around "why can't OTEL _also_ have something that does that"?
- phillipcarter 3y ago
  OTel kind of has both today. Most languages support autoinstrumentation via either an agent, libraries, or both. So for example, if you've got a big Spring Boot app, it'll instrument requests/responses and DB calls for you. Some languages also have lightweight "sprinkle some spans on it" things you can put in code, like Java method annotations. But none have truly automatic "span for every method call" instrumentation.

vadman97 3y ago

Have you seen the OTEL auto instrumentation package[1]? It supports a number of common frameworks (as middleware) and libraries (via monkeypatches) setting up spans for you to capture things like external http requests, database/cache queries, etc. [2] [1] https://opentelemetry.io/docs/instrumentation/js/automatic/ https://opentelemetry.io/docs/instrumentation/js/automatic/ [2] https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/metapackages/auto-instrumentations-node#supported-instrumentations https://github.com/open-telemetry/opentelemetry-js-contrib/t...

_boffin_ 3y ago
Tested those before, but still doesn't get what i'm looking for, which is inside the black box without manually adding the spans. The implementations for bunyan or winston would come the closet, but still doesn't achieve what i'm looking for.
- eep_social 3y ago
  I think the Ruby auto-instrumentation library does wrap practically every method but we ended up finding it too verbose. I think this is a bit of a goldilocks problem insofar as getting it just right is not easy and varies per application.
  - _boffin_ 3y ago
    One of the things i think that can be done is also limit the number of spans collected via configuration.
    - eep_social 3y ago
      Aye, but then you’re trading off between a centralized config that does sampling vs customized per-service (or whatever boundary) configs. There’s no free lunch, was mostly my point.