3 ms·
A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (altho
by cyberax 2mo ago
A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars).
The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink.
It's just outbound HTTP or GRPC, and it doesn't have to go over public Internet.
> In a runtime like Lambda, how would you avoid the need to run an edge collector?
Here's my setup (in Go, very simplified):
> // Instantiate a new slog logger
> logger := otelslog.NewLogger("root", otelslog.WithLoggerProvider(otelLogger))
> // Use the logger as needed
My code uses proper Go loggers exclusively. I also redirected the stdout and stderr to a goroutine (via the usual close(2)+open() trick) to serve as a catch-all sink for anything that slips the net.
- bilalq 2mo agoIn a lambda runtime, are you blocking client responses until logs/traces/metrics flush?
- Game_Ender 2mo agoThis is what I have done with CLI apps the directly send to the OTEL vendor. It works great.
- ojkelly 2mo agoUse the lambda layer [0] it sends the telemetry after the response is sent, so it doesn’t block. [0] https://github.com/open-telemetry/opentelemetry-lambda https://github.com/open-telemetry/opentelemetry-lambda
- bilalq 2mo agoThat lambda layer comes with an incredibly heavy performance penalty. It doesn't block, but it does consume compute/memory resources and takes forever to startup[0][1]. To be fair, Rotel is promising in this regard[2]. [0]: https://github.com/open-telemetry/opentelemetry-lambda/issues/263 https://github.com/open-telemetry/opentelemetry-lambda/issue... [1]: https://github.com/aws-observability/aws-otel-lambda/issues/228 https://github.com/aws-observability/aws-otel-lambda/issues/... [2]: https://github.com/rotel-dev/rotel https://github.com/rotel-dev/rotel
- cyberax 2mo agoI don't use Lambda anymore, but yes. I submitted traces to AWS XRay in a background goroutine with a small timeout.
- bilalq 2mo agoIf you're sending data purely to X-Ray, there's already a daemon running on lambda that you can forward to with low overhead if you don't use OTel. You also get near zero-cost logging and metric to Cloudwatch and EMF. But if you want bring destinations in the mix or do anything other than Cloudwatch , you have to pay the OTel tax. And even if you were content with a pure AWS setup, OTel is still being pushed on you now. The X-Ray daemon and SDKs are all deprecated now in favor of OTel. Things like enchrichment of resource level traces for things like the DynamoDB client in v3 of the AWS JS SDK don't work with the X-Ray SDK. And they never will now. You're now recommended to use the AWS Distro for OpenTelemetry setup and OTel SDKs. The performance overhead of this is heavy, with big cold-start penalties. Compare this with how the Datadog layer does adaptive flushing and performs relatively much better. Rotel is also promising in this space. But right now, OTel feels immature and things are being deprecated without the replacement being fully baked.
- cyberax 2mo agoWhy do you even _need_ these "layers"? It's a simple RPC protocol that submits data tagged with Span and Trace IDs. That's really all there is to it. You can just submit it directly, without involving any layers.
- bilalq 2mo agoBecause blocking on OTel data to flush before sending a response back is often unacceptable. The layers run a standalone process using the Lambda extension API so they can keep running after your function has responded to a request.