6 ms·

Observability in WebAssembly

neuronicnobody 3y ago
Extract logs, traces, and metrics from Wasm!
kbknapp 3y ago
I wonder how well (or if it's even possible?) to pair this with hosted WASM runners (like CloudFlare Workers, or Fastly Runners, etc.). It looks like there is also (limited?) support for tracing functions and allocations in WASM binaries you didn't compile. I'm looking forward to trying this out!
- nilslice 3y ago
  Yes, it should work in Cloudflare, you'd just need to use the JS adapter for whichever APM you use (Datadog, Honeycomb, Lightstep w/ more coming soon). The library provides import functions you pass into your module instance, and as long as the Wasm was instrumented before you deploy the Worker, you should have no issues. Please let us know if you hit any though!
  - bhelx 3y ago
    One caveat that Datadog (specifically traces) probably won't work on the edge just yet. But if this is something you're interested in kbknapp feel free to post your questions or requests here https://github.com/dylibso/observe-sdk/discussions https://github.com/dylibso/observe-sdk/discussions
_boffin_ 3y ago
Was just thinking of something like this last night. Now, wonder if I can compile JS and automatically wire up apm for all spans.
- deleted 3y ago
  [deleted]
- bhelx 3y ago
  I do have some experimental code working with QuickJS that does this. Feel free to reach out of if you're interested in trying it out or start a discussion https://github.com/dylibso/observe-sdk/discussions https://github.com/dylibso/observe-sdk/discussions
  - _boffin_ 3y ago
    I’ll be reviewing. Question: are you wiring it up like how APM is wired up in JVM applications or what exactly are you doing? Does it effect stack trace outputs or leave those alone?
    - bhelx 3y ago
      I have an instrumented version of QuickJS that can be used in the javy toolchain: https://github.com/bytecodealliance/javy https://github.com/bytecodealliance/javy There are two primary goals: 1. Emit metrics directly about the QuickJS runtime similar to node metrics 2. Expose the Observe API as js bindings With the Observe API exposed to JS, you can do manual instrumentation. I have also instrumented the pieces of QuickJS that invoke JS functions in order to automatically emit spans and it works, but still trying to fill some information gaps to make it more usable than manual instrumentation. > are you wiring it up like how APM is wired up in JVM applications or what exactly are you doing? Does it effect stack trace outputs or leave those alone? I can't speak to how JVM does it, but no this does not effect the call stack.
      - _boffin_ 3y ago
        Awesome work and looking forward to checking it out. One thing I’m also thinking about for node and friends is looking into is using v8 profiling and seeing if I can hook up something from that. But yeah, great stuff
lukevp 3y ago
This is by the same team behind Extism, a WASM plug-in library implemented in multiple languages. I’ve been active on their Discord and they’ve been really awesome and supportive of any questions I have. I’m excited to see where they go with this observability SDK. WASM has a major usability problem right now, and we need things like this and Extism to make it smoother to produce and consume wasm programs!
- nilslice 3y ago
  you're too kind! and excited to see your WebAssembly projects here on HN sometime too :)
- wikiwong 3y ago
  Thanks Luke, we greatly appreciate your engagement and help in hardening this ecosystem!
naillo 3y ago
What a crazy choice of background color. On the other hand it kinda signals that the developers themselves designed it which is generally a good sign of other characteristics of a project.
- nilslice 3y ago
  i'm the developer (non-designer) responsible for this, and dying laughing, so thank you!
- syndicatedjelly 3y ago
  lol i didnt even notice the change, just that i felt more agitated by the end
- onemoresoop 3y ago
  Somebody is a big fan of mustard
  - nilslice 3y ago
    twomoresoops plz
- deleted 3y ago
  [deleted]
titzer 3y ago
Great to see more focus on observability for Wasm now. This was a major impetus and still a major goal of the Wizard Research Engine, which has engine-level hooks that allow programmable instrumentation.
- bhelx 3y ago
  Thanks Ben! Really looking forward to your WasmCon talk.
- nilslice 3y ago
  huge fan of what you’re doing with Wizard & Virgil. and thanks for all you’ve done for wasm!
- wikiwong 3y ago
  We're inspired by this project, linking for posterity: https://github.com/titzer/wizard-engine https://github.com/titzer/wizard-engine
xena 3y ago
I love this. I wish it supported something self-hosted like Jaeger though.
- nilslice 3y ago
  we will have a pure HTTP & gRPC OTel adapter soon - just focused on the platforms first for this announcement but follow the repo for updates: https://github.com/dylibso/observe-sdk https://github.com/dylibso/observe-sdk
- wikiwong 3y ago
  Thanks for the feedback, we've done some internal work to verify open source, self hosted platforms like Jaeger, Zipkin, and OpenObserve. Official support is coming soon!
pranay01 3y ago
Not sure if OP is from Dylibso, but would love to see telemetry data emitted in OpenTelemetry format as well so different tools (including open source ones) which support Otel format can also consume this data. I am one of the maintainers at SigNoz [1], an otel native open source observability platform and would love to enable our users to see telemetry data from WASM [1] https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz
- nilslice 3y ago
  It shall be done! we already have the OTel formatting ready to go, but focused on these platform specific adapters for launch. In fact, Lightstep and Honeycomb adapters are fairly thin wrappers around the OTel formatting we do out of our instrumentation layer. We’ve got a ticket open on the GitHub repo to track in case you’d like to follow along. Happy to get your users supported!
  - serverlessmom 3y ago
    Oh neat! (I was here to ask the same question as above) would love to share the project when OTel support is released!
  - pranay01 3y ago
    Awesome! Would check out the github issue
    - bhelx 3y ago
      Yep, should be really soon. Also, we will check out signoz too. Could you see a reason to have an official adapter at some point? The adapter can tie things together for you that the platform expects, like set extra metadata, set trace ids, etc.
derek12 3y ago
[dead]

Extract logs, traces, and metrics from Wasm!

I wonder how well (or if it's even possible?) to pair this with hosted WASM runners (like CloudFlare Workers, or Fastly Runners, etc.). It looks like there is also (limited?) support for tracing functions and allocations in WASM binaries you didn't compile. I'm looking forward to trying this out!

nilslice 3y ago
Yes, it should work in Cloudflare, you'd just need to use the JS adapter for whichever APM you use (Datadog, Honeycomb, Lightstep w/ more coming soon). The library provides import functions you pass into your module instance, and as long as the Wasm was instrumented before you deploy the Worker, you should have no issues. Please let us know if you hit any though!
- bhelx 3y ago
  One caveat that Datadog (specifically traces) probably won't work on the edge just yet. But if this is something you're interested in kbknapp feel free to post your questions or requests here https://github.com/dylibso/observe-sdk/discussions https://github.com/dylibso/observe-sdk/discussions

_boffin_ 3y ago

Was just thinking of something like this last night. Now, wonder if I can compile JS and automatically wire up apm for all spans.

deleted 3y ago
[deleted]
bhelx 3y ago
I do have some experimental code working with QuickJS that does this. Feel free to reach out of if you're interested in trying it out or start a discussion https://github.com/dylibso/observe-sdk/discussions https://github.com/dylibso/observe-sdk/discussions
- _boffin_ 3y ago
  I’ll be reviewing. Question: are you wiring it up like how APM is wired up in JVM applications or what exactly are you doing? Does it effect stack trace outputs or leave those alone?
  - bhelx 3y ago
    I have an instrumented version of QuickJS that can be used in the javy toolchain: https://github.com/bytecodealliance/javy https://github.com/bytecodealliance/javy There are two primary goals: 1. Emit metrics directly about the QuickJS runtime similar to node metrics 2. Expose the Observe API as js bindings With the Observe API exposed to JS, you can do manual instrumentation. I have also instrumented the pieces of QuickJS that invoke JS functions in order to automatically emit spans and it works, but still trying to fill some information gaps to make it more usable than manual instrumentation. > are you wiring it up like how APM is wired up in JVM applications or what exactly are you doing? Does it effect stack trace outputs or leave those alone? I can't speak to how JVM does it, but no this does not effect the call stack.
    - _boffin_ 3y ago
      Awesome work and looking forward to checking it out. One thing I’m also thinking about for node and friends is looking into is using v8 profiling and seeing if I can hook up something from that. But yeah, great stuff

lukevp 3y ago

This is by the same team behind Extism, a WASM plug-in library implemented in multiple languages. I’ve been active on their Discord and they’ve been really awesome and supportive of any questions I have. I’m excited to see where they go with this observability SDK. WASM has a major usability problem right now, and we need things like this and Extism to make it smoother to produce and consume wasm programs!

nilslice 3y ago
you're too kind! and excited to see your WebAssembly projects here on HN sometime too :)
wikiwong 3y ago
Thanks Luke, we greatly appreciate your engagement and help in hardening this ecosystem!

naillo 3y ago

What a crazy choice of background color. On the other hand it kinda signals that the developers themselves designed it which is generally a good sign of other characteristics of a project.

nilslice 3y ago
i'm the developer (non-designer) responsible for this, and dying laughing, so thank you!
syndicatedjelly 3y ago
lol i didnt even notice the change, just that i felt more agitated by the end
onemoresoop 3y ago
Somebody is a big fan of mustard
- nilslice 3y ago
  twomoresoops plz
deleted 3y ago
[deleted]

titzer 3y ago

Great to see more focus on observability for Wasm now. This was a major impetus and still a major goal of the Wizard Research Engine, which has engine-level hooks that allow programmable instrumentation.

bhelx 3y ago
Thanks Ben! Really looking forward to your WasmCon talk.
nilslice 3y ago
huge fan of what you’re doing with Wizard & Virgil. and thanks for all you’ve done for wasm!
wikiwong 3y ago
We're inspired by this project, linking for posterity: https://github.com/titzer/wizard-engine https://github.com/titzer/wizard-engine

xena 3y ago

I love this. I wish it supported something self-hosted like Jaeger though.

nilslice 3y ago
we will have a pure HTTP & gRPC OTel adapter soon - just focused on the platforms first for this announcement but follow the repo for updates: https://github.com/dylibso/observe-sdk https://github.com/dylibso/observe-sdk
wikiwong 3y ago
Thanks for the feedback, we've done some internal work to verify open source, self hosted platforms like Jaeger, Zipkin, and OpenObserve. Official support is coming soon!

pranay01 3y ago

Not sure if OP is from Dylibso, but would love to see telemetry data emitted in OpenTelemetry format as well so different tools (including open source ones) which support Otel format can also consume this data. I am one of the maintainers at SigNoz [1], an otel native open source observability platform and would love to enable our users to see telemetry data from WASM [1] https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz

nilslice 3y ago
It shall be done! we already have the OTel formatting ready to go, but focused on these platform specific adapters for launch. In fact, Lightstep and Honeycomb adapters are fairly thin wrappers around the OTel formatting we do out of our instrumentation layer. We’ve got a ticket open on the GitHub repo to track in case you’d like to follow along. Happy to get your users supported!
- serverlessmom 3y ago
  Oh neat! (I was here to ask the same question as above) would love to share the project when OTel support is released!
- pranay01 3y ago
  Awesome! Would check out the github issue
  - bhelx 3y ago
    Yep, should be really soon. Also, we will check out signoz too. Could you see a reason to have an official adapter at some point? The adapter can tie things together for you that the platform expects, like set extra metadata, set trace ids, etc.

derek12 3y ago

[dead]