2 ms·

If you get to the end you find that the pain was all self-inflicted. I found it to be very easy in Python with standard stacks (mysql, flask, redis, requests, e

If you get to the end you find that the pain was all self-inflicted. I found it to be very easy in Python with standard stacks (mysql, flask, redis, requests, etc), because you literally just do a few imports at the top of your service and it automatically hooks itself up to track everything without any fuss.

etimberg 2y ago
Until you run your server behind something like gunicorn and all of the auto imports stop working and you have to do it all yourself.
- BugsJustFindMe 2y ago
  It works with uwsgi just fine though.
- jdsleppy 2y ago
  I found this to work fine https://opentelemetry-python.readthedocs.io/en/latest/examples/fork-process-model/README.html#gunicorn-post-fork-hook https://opentelemetry-python.readthedocs.io/en/latest/exampl...
  - jdsleppy 2y ago
    ...but with manually running autoinstrumentation in the post fork hook. I guess there is a lot of undocumented magic in OTel...
verall 2y ago
So recently I needed to this up for a very simple flask app. We're running otel-collector-contrib, jaeger-all-in-one, and prometheus on a single server with docker compose (has to be all within the corpo intranet for reasons..) Traces work, and I have the spanmetrics exporter set up, and I can actually see the spanmetrics in prometheus if I query directly, but they won't show up in the jaeger "monitor" tab, no matter what I do. I spent 3 days on this before my boss is like "why don't we just manually instrument and send everything to the SQL server and create a grafana dashboard from that" and agh I don't want to do that either. Any advice? It's literally the simplest usecase but I can't get it to work. Should I just add grafana to the pile?
- BugsJustFindMe 2y ago
  Yeah the biggest trouble really is on the dashboarding side of things, not the sending side, and is why there are popular SaaS products like datadog. If you're amenable to saas, datadog is probably the best way. Otherwise, look into SigNoz for a one-stop solution with minimal effort even if there are some rough edges still.
  - verall 2y ago
    We absolutely have to run it ourselves (...corporate reasons...), it's a lightweight service with only a few hundred users so we haven't had to worry much about perf (yet). SigNoz does look interesting, I may give this a shot, thank you. I'm a bit concerned about it conflicting with other things going on in our docker-compose but it doesn't look too bad..
- pdimitar 2y ago
  Try https://github.com/openobserve/openobserve https://github.com/openobserve/openobserve, it's extremely easy to self-host and it's an all-in-one solution, dashboards included (though admittedly I've seen prettier ones).
baby_souffle 2y ago
> I found it to be very easy in Python with standard stacks (mysql, flask, redis, requests, etc), because you literally just do a few imports at the top of your service and it automatically hooks itself up to track everything without any fuss. Yes, but only if everything in your stack is supported by their auto instrumentation. Take `aiohttp` for example. The latest version is 3.11.X and ... their auto instrumentation claims to support `3.X` [0] but results vary depending on how new your `aiohttp` is versus the auto instrumentation. It's _magical_ when it all just works, but that ends up being a pretty narrow needle to thread! [0]: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-aiohttp-server/pyproject.toml#L38 https://github.com/open-telemetry/opentelemetry-python-contr...
- BugsJustFindMe 2y ago
  > their auto instrumentation claims to support `3.X` Semver should never be treated as anything more than some tired programmer's shrug and prayer that nobody else notices the breakages they didn't notice themselves. Pin strict dependencies instead of loose ones, and upgrade only after integration testing. There are only two kinds of updates, ones that intend to break something and ones that don't intend to break something, and neither one guarantees that the intent matches the outcome.
  - baby_souffle 2y ago
    > Semver should never be treated as anything more than some tired programmer's shrug and prayer that nobody else notices the breakages they didn't notice themselves. That's precisely my point, but you said it better :). I have had _mixed_ results getting auto instrumentation working reliably with packages that are - technically - supported.

etimberg 2y ago

Until you run your server behind something like gunicorn and all of the auto imports stop working and you have to do it all yourself.

BugsJustFindMe 2y ago
It works with uwsgi just fine though.
jdsleppy 2y ago
I found this to work fine https://opentelemetry-python.readthedocs.io/en/latest/examples/fork-process-model/README.html#gunicorn-post-fork-hook https://opentelemetry-python.readthedocs.io/en/latest/exampl...
- jdsleppy 2y ago
  ...but with manually running autoinstrumentation in the post fork hook. I guess there is a lot of undocumented magic in OTel...

verall 2y ago

So recently I needed to this up for a very simple flask app. We're running otel-collector-contrib, jaeger-all-in-one, and prometheus on a single server with docker compose (has to be all within the corpo intranet for reasons..) Traces work, and I have the spanmetrics exporter set up, and I can actually see the spanmetrics in prometheus if I query directly, but they won't show up in the jaeger "monitor" tab, no matter what I do. I spent 3 days on this before my boss is like "why don't we just manually instrument and send everything to the SQL server and create a grafana dashboard from that" and agh I don't want to do that either. Any advice? It's literally the simplest usecase but I can't get it to work. Should I just add grafana to the pile?

BugsJustFindMe 2y ago
Yeah the biggest trouble really is on the dashboarding side of things, not the sending side, and is why there are popular SaaS products like datadog. If you're amenable to saas, datadog is probably the best way. Otherwise, look into SigNoz for a one-stop solution with minimal effort even if there are some rough edges still.
- verall 2y ago
  We absolutely have to run it ourselves (...corporate reasons...), it's a lightweight service with only a few hundred users so we haven't had to worry much about perf (yet). SigNoz does look interesting, I may give this a shot, thank you. I'm a bit concerned about it conflicting with other things going on in our docker-compose but it doesn't look too bad..
pdimitar 2y ago
Try https://github.com/openobserve/openobserve https://github.com/openobserve/openobserve, it's extremely easy to self-host and it's an all-in-one solution, dashboards included (though admittedly I've seen prettier ones).

baby_souffle 2y ago

> I found it to be very easy in Python with standard stacks (mysql, flask, redis, requests, etc), because you literally just do a few imports at the top of your service and it automatically hooks itself up to track everything without any fuss. Yes, but only if everything in your stack is supported by their auto instrumentation. Take `aiohttp` for example. The latest version is 3.11.X and ... their auto instrumentation claims to support `3.X` [0] but results vary depending on how new your `aiohttp` is versus the auto instrumentation. It's _magical_ when it all just works, but that ends up being a pretty narrow needle to thread! [0]: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-aiohttp-server/pyproject.toml#L38 https://github.com/open-telemetry/opentelemetry-python-contr...

BugsJustFindMe 2y ago
> their auto instrumentation claims to support `3.X` Semver should never be treated as anything more than some tired programmer's shrug and prayer that nobody else notices the breakages they didn't notice themselves. Pin strict dependencies instead of loose ones, and upgrade only after integration testing. There are only two kinds of updates, ones that intend to break something and ones that don't intend to break something, and neither one guarantees that the intent matches the outcome.
- baby_souffle 2y ago
  > Semver should never be treated as anything more than some tired programmer's shrug and prayer that nobody else notices the breakages they didn't notice themselves. That's precisely my point, but you said it better :). I have had _mixed_ results getting auto instrumentation working reliably with packages that are - technically - supported.