32 ms·

OpenTelemetry in 2023

341 points by kevinslin 3y ago

oaiey 3y ago
I am very happy with the progress of OpenTelemetry. When I pushed for it years ago, my developers where hesitant (it was new and they never heard about it) but when I revisited the topic a year ago, OpenTelemetry was everywhere in our systems and the log/tracability vendor we have was switching over to it. Thanks to this amazing group!
phito 3y ago
OTEL is awesome. I was part of the team integrating it at my job, it went really smoothly, and it saved us so much time debugging our microservice application. I would say the hardest part was getting other devs to use it, a lot of them are stuck in their own way and did not want to go through the relatively small learning curve...
dottedmag 3y ago
Ugh, protobuf and gRPC as blessed (?!) transports? Thanks, I always wanted to bring 100MB dependency into my code just to send metrics.
- wdb 3y ago
  Yeah, but you have the same problem when using Prometheus and the native histograms? So far I know it is not supported by the text / openmetrics format only in their protobuf-basaed format version
- kortex 3y ago
  Where are you getting 100MB from? You don't need the whole protoc toolchain. Well I've never worked with C or C++ and gRPC, but insofar as rust, go, and python are concerned, the increase in container/binary size is a few MB.
- Xeoncross 3y ago
  If you're using Rust, Go, C, Java, or some other performant language then it shouldn't be much. If you're using a slower scripting language like Ruby or JavaScript then you might have issues.
  - valyala 3y ago
    Yeah, including the official Opentelemetry parser for metrics just increases sophisticated Go binary size by 10MB - from 19MB to 29MB [1] [1] https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570#issuecomment-1221981838 https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570...
- podoman 3y ago
  Just a quick clarification. OTEL is not just a transport, its a specification.
- malkia 3y ago
  It's 8mb dll when compiled fully as dll(s). Check the https://github.com/open-telemetry/opentelemetry-cpp https://github.com/open-telemetry/opentelemetry-cpp as they've recently added dll support, and I've been keeping a branch that's geared towards more simplicity on deploy (e.g. single .dll, instead of several) - OpenTelemetry's C++ solution though is more flexible, and less size used last I've tried it. Here is mine - https://github.com/malkia/opentelemetry-cpp https://github.com/malkia/opentelemetry-cpp
baz00 3y ago
Is there yet any way of having a front end for this which doesn't significantly dent your revenue stream either in staffing, infrastructure or license fees? We land over 2000 requests/second and it's expensive just keeping logs.
- number101010 3y ago
  Most observability vendors support OTEL at this point. To plug the OSS project I work on that supports OTEL ingestion: https://github.com/grafana/tempo/ https://github.com/grafana/tempo/
  - baz00 3y ago
    Thanks I had no idea Tempo even existed despite using Grafana. Will read into it.
    - candiddevmike 3y ago
      IMO, Jaeger is easier to setup/manage and has a better interface than Grafana/Tempo. It's easy to add Jaeger to your local dev stack so you can have tracing while developing. FWIW I now use Tempo because I have everything else in Grafana (Prometheus, Loki), but I do miss using Jaeger.
      - number101010 3y ago
        > It's easy to add Jaeger to your local dev stack so you can have tracing while developing. Tempo can be spun up with docker compose using a local disk for ephemeral storage/querying: https://github.com/grafana/tempo/blob/main/example/docker-compose/local/docker-compose.yaml https://github.com/grafana/tempo/blob/main/example/docker-co... Maybe this meets your needs? > Jaeger is easier to setup/manage and has a better interface than Grafana/Tempo What do you enjoy about the Jaeger interface? Perhaps it's a gap in Tempo we can improve.
      - sofixa 3y ago
        Jaeger can use multiple backends for storage, including Tempo, so it's not an either/or situation. I'm fairly sure there was an official Grafana-provided Jaeger gRPC plugin for Tempo, but can't easily find it, only this one: https://github.com/flitnetics/jaeger-tempo https://github.com/flitnetics/jaeger-tempo
candiddevmike 3y ago
CNCF needs an OTEL log aggregator
- lokar 3y ago
  The otel collector/agent can do this. Not storage/query, but aggregation and processing.
monlockandkey 3y ago
I'm new to telemetry. Can someone please explain the relationship between otel, Prometheus, Grafana agent?
- wdb 3y ago
  It's a standardised way to send metrics to Prometheus. You can have a metrics exporter for Prometheus, e.g. https://opentelemetry.io/ecosystem/registry/?s=prometheus&component=&language= https://opentelemetry.io/ecosystem/registry/?s=prometheus&co... [1] https://opentelemetry.io/ecosystem/registry/?s=prometheus&component=&language= https://opentelemetry.io/ecosystem/registry/?s=prometheus&co... [2] https://opentelemetry.io/docs/specs/otel/metrics/sdk_exporters/otlp/ https://opentelemetry.io/docs/specs/otel/metrics/sdk_exporte...
  - podoman 3y ago
    Not only prometheus fwiw. There's lots of providers that support it, e.g.: - datadog - honeycomb.io - highlight.io (I'm a founder) - sentry.io
- skrtskrt 3y ago
  Prometheus: metrics collector/databases with a defined metrics format. Many metrics/timeseries databases make an effort to be "Prometheus-compatible" as it was kind of the unofficial standard. OTEL: new open standards which are supposed to provide vendor-independent formats which are compatible and composable between metrics, logs, and traces, and easily enable things like deriving metrics from traces (latency would be an obvious one here). Grafana Agent: a metrics? maybe also logs and traces? collector that supports Prometheus, OTEL and other metrics formats, which can forward, sample, and transform the data. Made by Grafana, open source, etc. Grafana's metrics DB Mimir and maybe some others are essentially "more scalable prometheus", and use prometheus metrics format on disc, so one large concern of the Grafana Agent would be converting OTEL metrics to prometheus metrics format for ingestion into the Grafana databases - but the agent has a whole bunch of other functions supported as well. OTEL Collector - non-vendor-specific collector, like Grafana Agent but largely just concerned with allowing collection/ingestion of OTEL and coverting other formats into OTEL. Allows extensions and plugins to be added for other purposes.
sheshe42 3y ago
[flagged]
waynesonfire 3y ago
like a virus, the complexity spreads to fill head count.
- zaat 3y ago
  Like a reflex, comments blaming solutions for complex challenges are being emitted. Alas, no alternatives are ever offered. Shall we limit all technology to what can be contained in a single box? should we just ban countries, companies and communities from growing past the 500k head count? Please, go on, do say what shall we do with the inherent complexity of the universe, shall we stop physics at Newton and ban particle accelerators? De-legalize storage systems exceeding petabyte? Please, let us hear how complexity can be put back to its old box
deleted 3y ago
[deleted]
skrtskrt 3y ago
since I added OTEL instrumentation to an app, I already easily swapped out the backends from Prometheus and Jaeger to Grafana Mimir and Tempo. The lack of lock-in is fantastic, don't know if I've ever just switched technologies that easily, even SQL databases.
_boffin_ 3y ago
Has anyone found a solution to automatically adding spans automatically for nodejs applications that doesn't involve littering startSpan or startActiveSpans everywhere? I'm currently looking into monkey patching, but that seems dirty.
- pondidum 3y ago
  Depends what you're trying to accomplish. We're using middleware to start spans for http and message handlers, and then adding `startSpan` where we need. I don't see a problem with `startSpan` everywhere as it's not much noiser than the `log.info` that would be there instead if we didn't have otel.
  - _boffin_ 3y ago
    I started my APM journey when using Spring w/ Java and fell in love with how i can trace the entire flow through the entire application and then just config on how many spans i want to send over and what sample rate i want collected. I'd love to accomplish that. What you mentioned is all nice and well (optimal route), but right now, i'm working with some applications that needs it, but has i don't even know how many methods / classes that i'd need to go through and implement it on.
  - Spivak 3y ago
    I think my major pain point with them being everywhere is unlike a log they push a new stack frame and closure. They make tracebacks so much more annoying, yes I know the error happened in a start span, thank you. I wish observability actually observed more than it contributed. Once https://peps.python.org/pep-0669/ https://peps.python.org/pep-0669/ is available I'm gonna try my damndest to get otel working through it. Just give me a config file that says what functions you're interested in and I'll do the rest.
    - pondidum 3y ago
      Agreed the amount of stuff showing in stack traces can be annoying - but it does depend on language. I agree 100% in javascript/typescript its annoying, and I would love to get rid of them, In go however, there isn't an extra stack frame. Nor in C# thinking about it. The config file of functions to trace is a really interesting idea. How would you handle wanting to add data to the spans from inside those functions though? e.g. I want to add all kinds of props to the spans that can't be known except inside the function executing.
jeffchao 3y ago
Otel is great for avoiding vendor lock-in and the spec stability is a long time coming — great to see. Also good there’s architecture options aside from side car (there’s also proxy mode), which is nice for situations like deploying on a PaaS. That said the main trouble I’ve had in the past is instability in the SDKs. Java one’s decent. Go, not so much, for example. Traditionally Otel has been awesome for tracing but not so much logs, events, and even metrics (obviously depending on your language). Other than that, auto-instrumentation has been nice. We had this exact functionality when I worked on observability at Netflix and made starting up and maintaining microservices easy and really helped with adoption.
pphysch 3y ago
So I have a VictoriaMetrics (i.e. Prometheus) setup that I am happy with, but haven't touched OpenTelemetry; why should I care about it? Is it a serious solution for log aggregation, but do I need a separate database for logs?
- hagen1778 3y ago
  You shouldn't unless you want to use the new open source standard for telemetry. You won't benefit from simplicity or performance improvements. It would be quite the opposite. You can check what is the actual cost of open telemetry adoption here [0] But if you ever decide to go this path - VictoriaMetrics supports OpenTelemetry protocol for metrics [1] [0] https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570 https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570 [1] https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#sending-data-via-opentelemetry https://docs.victoriametrics.com/Single-server-VictoriaMetri...
throwaway894345 3y ago
I'd be very interested to hear others experience (either as comments or good blog posts / write ups that they've read) with using Open Telemetry. I haven't used OTEL stuff directly, but I've had always been very disappointed with the telemetry vendors in the past (although as far as I can tell the problems weren't "these vendors don't talk to each other" kinds of problems that I think OTEL aims to solve).
- dboreham 3y ago
  Experience has been positive. We had to understand (and in places fix/enhance) the agent used in our system but the benefits of it not being an expensive black box were huge. E.g. you can run the whole stack on a laptop and therefore use the same tooling for perf analysis in dev as production.
- vmarchaud 3y ago
  I've used OpenTelemetry since it's original alpha in 2020. Originally the main issue i had was supporting tracing across common libraries (there wasn't a lot of libraries supported back then). Now (and i recently worked with it) i would say is which protocol is supported by which component: your sdk generate spans/metrics in a specific format, then you send that to a collector that accept a range of protocol versions and finally you can send that to your vendor ... but you need to know which protocol/version it supports. That's not actually something you can do much about considering the sheer size of opentelemetry (both in term of implementation and vendors working on it) and i expect for people implementing nowadays, proto should be pretty stable and my experience should theorically not be the case anymore.
CSDude 3y ago
I love OpenTelemetry and we want to trace almost every span happening. We’d be bankrupt if we went any vendor. We wired opentelemetry with Java magic, 0 effort and pointed to a self hosted Clickhouseand store 700m+ span per day with a 100$ EC2. https://clickhouse.com/blog/how-we-used-clickhouse-to-store-opentelemetry-traces https://clickhouse.com/blog/how-we-used-clickhouse-to-store-...
- _boffin_ 3y ago
  Are you making sure that you're doing a sample rate, but send over all errors? At a former place, we were doing 5% of non-error traces.
  - grogenaut 3y ago
    Careful, we've had systems go down under increased load just emitting errors if they didn't emit much in non error state
    - _boffin_ 3y ago
      Can you go into more detail about your comment, please.
      - cwp 3y ago
        Not the GP, but: Imagine you're sampling successful traces at, say, 1%, but sending all error traces. If your error rate is low, maybe also 1%, your trace volume will be about 2% of your overall request volume. Then you push an update that introduces a bug and now all requests fail with an error, and all those traces get sampled. Your trace volume just increased 50x, and your infrastructure may not be prepared for that.
      - Longwelwind 3y ago
        I think what they means is that if you provisioned your system to receive spans for 5% of non-error requests and a few error requests, if for some random act of god, all the requests yield an error, your span collector will suddenyl receive spans for all requests.
shayel 3y ago
[flagged]
kiitos 3y ago
OpenTelemetry is a marketing-driven project, designed by committee, implemented naively and inefficiently, and guided by the primary goal of allowing Fortune X00 CTOs to tick off some boxes on their strategy roadmap documents. It's not something that anyone with a choice in the matter should be using.
- brainbag 3y ago
  It looks like every other comment in this thread is favorable to very positive, can you go into more detail about what specifically isn't good about it?
  - arp242 3y ago
    Not the previous poster, but I had to implement it a few years ago, and I found it unbelievable complex with dense and difficult to read specifications. I've implemented plenty of protocols and formats from scratch using just the specification, but rarely have I had such difficulty than with OpenTelemetry. I guess this is something you don't notice as merely a "user", but IMHO it's horribly overengineered for what it does and I'm absolutely not a fan. I also disliked the Go tooling for it, which is "badly written Java in Go syntax", or something along these lines. This was 2 years ago. Maybe it's better now, but I doubt it. In our case it was 100% a "tick off some boxes on their strategy roadmap documents" project too and we had much much better solutions. OTel is one of those "yeah, it works ... I guess" but also "ewwww".
    - phillipcarter 3y ago
      I'd recommend trying it out today. In 2021, very few things in OTel were GA and there wasn't nearly as much automatic instrumentation. One of the reasons why you had to dive into the spec was because there was also very little documentation, too, indicative of a heavily in-progress project. All of these things are now different.
      - arp242 3y ago
        I'll be happy to take your word that some implementation issues are now improved, but things like "overengineered" and "way to complex for what it needs to do" really are foundational, and can't just be "fixed" without starting from scratch (and presumably this is all by design in the first place).
jensneuse 3y ago
We're soon launching a GraphQL Analytics, Tracing and Metrics stack on top of OTEL. We've built a custom OTEL exporter to Clickhouse in go, so we can export OTEL traces and Prometheus metrics all to Clickhouse. We've built this stack for our federated GraphQL Gateway but it could essentially work for any OTEL service. If you want to learn more, here's some info: https://wundergraph.com/cosmo https://wundergraph.com/cosmo Were soon going to open source this, so just follow me/us of you're interested. What I really like about this stack is that you can use our end to end solution, but you're not locked into it. We provide a full service, but you can also just use your own OTEL backend if you want to eject.
pradn 3y ago
Most of the comments on this thread are talking about using OpenTelemetry to send metrics/logs to self-hosted collector jobs. Though using a standard library supported by a bunch of collector tools like ClickHouse is useful it itself, there's another benefit too. The specification allows transferring trace IDs across system boundaries. If you and your dependencies all implement the OpenTelemetry spec, then you get spans that reveal in granular detail what happened in the journey. For example, you could learn that it was your database loading a page from disk that took so long, or that a Cloud service's metadata plane was the responsible span for high latency.
infogulch 3y ago
Telemetry seems like it would be a great candidate for columnar storage formats like Parquet or arrow. In particular I expect that it would compress very well, which could reduce telemetry bandwidth consumption / allow for a bigger sample rate. Does anybody have any experience with the intersection of these technologies?
- solumos 3y ago
  Honeycomb built their own columnar database[0] to support their product. [0] - https://www.honeycomb.io/resources/why-we-built-our-own-distributed-column-store https://www.honeycomb.io/resources/why-we-built-our-own-dist...
- vmarchaud 3y ago
  Someone from F5 worked on this with OpenTelemetry [0] for Arrow, another effort was done for Parquet but was dropped [1] [0]: https://github.com/open-telemetry/oteps/pull/171 https://github.com/open-telemetry/oteps/pull/171 [1]: https://github.com/open-telemetry/opentelemetry-proto/pull/346 https://github.com/open-telemetry/opentelemetry-proto/pull/3...
  - infogulch 3y ago
    Oh nice, thank you (and also solumos) for the links! It looks like oteps/pull/171 (merged June 2023) expanded and superseded the opentelemetry-proto/pull/346 PR (closed Jul 2022) [0]. The former resulted in merging OpenTelemetry Enhancement Proposal 156 [1], with some interesting results especially for 'Phase 2' where they implemented columnar storage end-to-end (see the Validation section [2]): * For univariate time series, OTel Arrow is 2 to 2.5 better in terms of bandwidth reduction ... and the end-to-end speed is 3.1 to 11.2 times faster * For multivariate time series, OTel Arrow is 3 to 7 times better in terms of bandwidth reduction ... Phase 2 has [not yet] been .. estimated but similar results are expected. * For logs, OTel Arrow is 1.6 to 2 times better in terms of bandwidth reduction ... and the end-to-end speed is 2.3 to 4.86 times faster * For traces, OTel Arrow is 1.7 to 2.8 times better in terms of bandwidth reduction ... and the end-to-end speed is 3.37 to 6.16 times faster Pretty exciting results! The OTEL-Arrow adapter has subsequently been donated to the otel community; here's a comment that does a good job of summarizing the results and the recommendations that came out of the test [3]. [0]: https://github.com/open-telemetry/opentelemetry-proto/pull/346#issuecomment-1176314436 https://github.com/open-telemetry/opentelemetry-proto/pull/3... [1]: https://github.com/open-telemetry/oteps/blob/main/text/0156-columnar-encoding.md https://github.com/open-telemetry/oteps/blob/main/text/0156-... [2]: https://github.com/open-telemetry/oteps/blob/main/text/0156-columnar-encoding.md#validation https://github.com/open-telemetry/oteps/blob/main/text/0156-... [3]: https://github.com/open-telemetry/community/issues/1332#issuecomment-1538916171 https://github.com/open-telemetry/community/issues/1332#issu...
TwentyPosts 3y ago
Can someone tell me roughly at which point a tool like OpenTelemetry becomes interesting? It seems complicated, at which point should you bother with it instead of just handrolling simple stuff and eyeballing it?
- paulddraper 3y ago
  OpenTelemetry is several different things, so you'd have to be more specific. But, for example, if you write a library and you want your downstream users to be able to see the telemetry. OpenTelemetry provides a standardized interface, so you don't need to make assumptions.
no_wizard 3y ago
The OTEL official project libraries don't work well on the web frontend yet. No way of correlating errors to source-maps for instance, at least out of the box. The web browser collector published by the OTEL project uses Zone.js to hijack just about everything in the browser into contexts. If you used modern Angular before, you may recognize zone.js, its a real pain sometimes, and messes with globals, which isn't great, as it can create situations where behavior isn't predictable. I don't know that OTEL has any standard around things like session replays either. Lots of telemetry platforms support this (Sentry, Rollbar, DataDog etc) I think alot of backend teams have really like it. I do like the cross boundary nature of spans where you can follow them by a unique tag across your entire system. I personally find its extremely verbose at times, in terms of the payload it generates, some logging platforms are more compact in this regard, but in practice, I haven't noticed it to be an issue
- renke1 3y ago
  I think with native Promises (and async/await?) there is currently no way to implement something like Zone.js properly. I've tried to instrument my code manually, but it's really error-prone and verbose. We really need something like https://nodejs.org/api/async_context.html#class-asynclocalstorage https://nodejs.org/api/async_context.html#class-asynclocalst... to be implemented in the browser.
  - vmarchaud 3y ago
    You can follow [0] which is currently stage 2 to fix this [0]: https://github.com/tc39/proposal-async-context https://github.com/tc39/proposal-async-context
    - renke1 3y ago
      Thanks! I was only aware of a Zones proposal which was withdrawn I think.
    - no_wizard 3y ago
      In addition to this, is the new (stage 3 even!)explicit resource management proposal[0], supported by TypeScript version >= 5.2[1] Though I agree that async context is better fit for this generally, the ERM should be good for telemetry around objects that have defined lifetime semantics, which is a step in the right direction you can use today [0]: https://github.com/tc39/proposal-explicit-resource-management https://github.com/tc39/proposal-explicit-resource-managemen... [1]: https://www.totaltypescript.com/typescript-5-2-new-keyword-using https://www.totaltypescript.com/typescript-5-2-new-keyword-u...
nickstinemates 3y ago
I run a full OSS otel stack in my home lab. It was a lot of fun to set up, with redundancy and all of the extras. It hasn't been particularly useful, but centralized logging and metrics are pretty to look at in dashboards, which was my motivation. Let's hear some great debugging stories that have been powered by OTEL. I'd love to hear from the horses mouth without marketing speak how it was worth $$$$ collecting and storing all of this info.
nerdjon 3y ago
Slight tangent for a moment, I really really hate the "subscribe" popup that comes up on this blog. It is not clear at all that you can just close it and not give your email since there is no "x" button. Instead it has the incredibly unintuitive "continue reading" under the subscribe button that I did not think would work. Clicking out of it also did not work. Seriously we can and should be better than this. On the topic of open telemetry. I have been long wanting to play with it and see if it offers all of the capabilities when we send the data to datadog. But I have been reluctant to add in another thing to manage and train on if it means that the datadog agent is still necessary for anything outside of the basics. Has anyone else actually tried hooking this up to datadog? Edit: just to be clear, my driving goal of this is not necessarily to keep it with datadog. But that is currently where much of our alerting and logs are now. So the idea would be to switch to open telemetry which would then allow us to (theoretically) move to something else down the line.
- monocasa 3y ago
  WRT the popup, that's a medium.com thing. I agree that it's very annoying.
  - luispauloml 3y ago
    It's not Medium, it's Substack.
- remram 3y ago
  Personally I found the poll much worse. Got asked for my opinion, clicked on one of the option, and got rewarded with a full-screen "create an account". Hit the back button and hit taken back to HN. Nice.
hu3 3y ago
Speaking about OpenTelemetry, has anyone used https://uptrace.dev https://uptrace.dev ? From a quick glance it seems to be simple, free and open-source deployed as a single Go binary. They use ClickHouse to store data. Almost too good to be true. I'm contemplating them for a new project.
- podoman 3y ago
  Fwiw, we do the same (https://highlight.io https://highlight.io). Heard good things about uptrace as well.
renke1 3y ago
A few of my colleagues and I had the silly (?) idea that you don't really need logs anymore. Instead of log messages you just attach span events [0]. You then just log the span title and a link to that span in Jaeger; something like [1]. I've only really tried that in my private project, but it felt pretty good. The UI of Jaeger could be a bit better to support that usage, though. Edit: Actually, those colleagues are doing a talk about that topic. So, if you are in Germany and Hannover area, have a look at [2] and search for "Nie wieder Log-Files!". [0]: https://opentelemetry.io/docs/instrumentation/ruby/manual/#add-span-events https://opentelemetry.io/docs/instrumentation/ruby/manual/#a... [1]: tracing.ts:38 Usecase: Handle Auth tracing.ts:47 http://localhost:16686/trace/ec7ffb1e23ddbb8dd770a3f08028666b tracing.ts:38 Adapter: Find Personal Board tracing.ts:47 http://localhost:16686/trace/e22d342316ab0d7d23230864008e27bc tracing.ts:38 Adapter: Find Starred Board List tracing.ts:47 http://localhost:16686/trace/129f89cee26d54cfdc38abea368d9b4e tracing.ts:38 Adapter: Find Personal Board List tracing.ts:47 http://localhost:16686/trace/97948127d77501ff0c65a5db21b21b5a [2]: https://javaforumnord.de/2023/programm/ https://javaforumnord.de/2023/programm/
- phillipcarter 3y ago
  Depending on how greenfield a project is, you don't even need span events unless you absolutely require a timestamp for a specific operation with no duration. Just using spans for every meaningful operation is like having more powerful structured logs. This approach isn't possible for a lot of systems that have existing logs they need to bring along, but if you're greenfield enough, I'd recommend it.
  - glenjamin 3y ago
    Using only spans is surprisingly effective! I did a talk about this at QCon London last year https://www.infoq.com/presentations/event-tracing-monitoring/ https://www.infoq.com/presentations/event-tracing-monitoring...
  - paulddraper 3y ago
    Agreed. Same is true for metrics derived from spans. (Though for metrics, you don't need to sample, and for spans you might. So keep in mind.)
streblo 3y ago
I'm hugely disappointed with OpenTelemetry. In my experience, its an over-engineered mess and the out-of-the-box experience is super user hostile. What it purports to be is so far away from what it actually is. Otel markets itself as a universal tracing/metrics/logs format and set of plug and play libraries that has adapters for everything you need. It's actually a bunch of half/poorly implemented libraries with a ton of leaky internals, bad adapters, and actually not a lot of functionality.
- cratermoon 3y ago
  Okay but can you point to some specific experience and suggest improvements?
  - caust1c 3y ago
    Try implementing an OTEL Tracer. This interface is insane, and should really just be a struct. https://github.com/open-telemetry/opentelemetry-go/blob/trace/v1.17.0/trace/trace.go#L336-L383 https://github.com/open-telemetry/opentelemetry-go/blob/trac...
    - ramenmeal 3y ago
      Why were you implementing your own tracer? Don't they publish an implementation?
      - phillipcarter 3y ago
        Yes, all SDKs have a tracer you can just use. While you can technically create your own tracer, you're officially in Hard Mode territory - there's no highly extensible system I'm aware of that makes swapping core concepts (that already have a default) easy.
    - skrtskrt 3y ago
      how is this interface insane? It's a list of the behaviors you need to implement if you're rolling your own OTEL Tracer Span implementation, and not using one of the multiple available. In contrast, OpenTracing's interfaces had hardly any required methods, so you had to do a runtime type-cast to the whichever implementation you were using in order to access anything useful on the Span like the OperationName.
bilalq 3y ago
OpenTelemetry is being pushed as a replacement for AWS X-Ray SDKs by AWS, but it's in such a broken state for Lambda right now. A 200-500% performance penalty for using it is insane[1][2]. [1]: https://github.com/aws-observability/aws-otel-lambda/issues/228 https://github.com/aws-observability/aws-otel-lambda/issues/... [2]: https://github.com/open-telemetry/opentelemetry-lambda/issues/263 https://github.com/open-telemetry/opentelemetry-lambda/issue...
- hinkley 3y ago
  How do you use it with Lambda? Delta mode? Or tons of labels?
  - bilalq 3y ago
    I use X-Ray directly via AWS Powertools for Lambda[1] instead of going through OTel. The underlying AWS X-Ray SDK team pushes OTel pretty hard in their README[2] and Github issues, but the current situation is that OpenTelemetry is just not production ready for serverless applications. [1]: https://docs.powertools.aws.dev/lambda/typescript/latest/ https://docs.powertools.aws.dev/lambda/typescript/latest/ [2]: https://github.com/aws/aws-xray-sdk-node https://github.com/aws/aws-xray-sdk-node
podoman 3y ago
Disclaimer: I'm the founder of an observability company (Highlight.io). OpenTelemetry has been INCREDIBLY valuable to us. Not only has it made it super fast to build out SDKs for our customers, but the fact that its maintained actively gives us confidence that we're rolling out stable logic to customers' environments. I agree with the author that OpenTelemetry has succeeded, and its pretty obvious from the fact that most major observability vendors support. In short, to a developer it may not seem like its particularly valuable because a metrics/logs/traces API is quite simple whether or not you use OTEL. But the fact that this is an industry wide spec is where it becomes powerful. A few
- m1keil 3y ago
  It's quite understandable that a "o11y protocol" is very valuable to a o11y service provider (especially the smaller ones) since it reduces the cost of vendor swap.
paulddraper 3y ago
Two problems with OpenTelemetry: 1. It doesn't know what the hell it is. Is it a semantic standard? Is a protocol? It is a facade? It is a library? What layer of abstraction does it provide? Answer: All of the above! All the things! All the layers! 2. No one from OpenTelemetry has actually tried instrumenting a library. And if they have, they haven't the first suggestion on how instrumenters should actually use metrics, traces, and logs. Do you write to all three? To one? I asked this question two years ago, zero answers :( [1] [1] https://github.com/open-telemetry/opentelemetry-specification/issues/2107 https://github.com/open-telemetry/opentelemetry-specificatio...
- jauntywundrkind 3y ago
  Good deck of questions but I don't think they matter. I don't think those are answerable questions for observability, be it OpenTelemetry or other proprietary systems. You can go read the leading observability companies web pages and they'll have a 4 page writeup on custom instrumentation. That's not much, just covers very elementary basics! It's not like OTel is behind. The answer just tends heavily towards "it depends." Once you have experience - OTel or other - you can work through these things that might confound a neophyte.
  - morelisp 3y ago
    The problem is they keep making the OTel tooling worse for working through these things, because the people writing the OTel tooling broadly aren't the people actually trying to monitor things. Even before OTel, plain Prometheus client libraries suffered from this.
- coxley 3y ago
  Agreed. Some things they suggest aren't actually possible with their SDKs. For example, you cannot define a histogram's buckets near where you define the histogram. You have to give the global exporter (or w/e the type is) a list of "overrides" that map each histogram name => their buckets. This makes it extremely ugly when you have libraries that emit metrics. https://github.com/open-telemetry/opentelemetry-go/issues/3826 https://github.com/open-telemetry/opentelemetry-go/issues/38...
kevinslin 3y ago
author of the post here - was inspired to write this post after working with OTEL for a few months - realized that OTEL had a ridiculously large surface area that most people (at least myself) might not be aware of I see a lot of comments about how overly complex OTEL is. I don't disagree with this. in some sense, OTEL is very much the k8 for observability (good and bad) The good is that it is a standard that can support every conceivable use case and has wide industry adoption The bad is that there is inherent complexity in needing to support the wide array of use cases
chrisweekly 3y ago
Non-paywalled URL: https://archive.is/NULpZ https://archive.is/NULpZ
enahs-sf 3y ago
Someone needs to build the Segment of APM. Datadog lock-in is real and the bill is high.
Havoc 3y ago
Very split opinions in the comments. Seems people either love it or hate it with nothing in between
enahs-sf 3y ago
I just want Segment but for APM to reduce vendor lock-in.
nijave 3y ago
I've had pretty decent experience with .NET + Grafana Cloud (hosted Tempo) You can generate RED metrics then tail based sample and send a subset of full traces. As with many new things, there's varying maturity in client libraries and still things missing. The Redis .NET client (maybe others?) wasn't very good (maybe alpha/beta quality) but the other .NET stuff seemed reasonable. At least with .NET, it integrates with the existing System.Diagnostics API. I think libraries in some other languages (go?) are a bit clumsier depending on what type of diagnostic/debug/event/performance/reflection APIs that language runtime exposes for interspection.
edenfed 3y ago
Disclaimer: I am one of the maintainers Many comments complain about the complexity of using OpenTelemetry, I recommend checking out Odigos, an open-source project which makes working with OpenTelemetry much easier: https://github.com/keyval-dev/odigos https://github.com/keyval-dev/odigos We combine OpenTelemetry and eBPF to instantly generate distributed traces without any code changes.
- pl4nty 3y ago
  How does this compare to the OTel Operator feature-wise? I'm loving auto-instrumentation with Grafana Cloud, but operator config is painful and I've hit (and helped fix) a lot of bugs
- williamscs 3y ago
  Hi! Just wanted to let you know I'm getting a 500 error on your docs page. https://docs.odigos.io/ https://docs.odigos.io/
  - edenfed 3y ago
    Can you try again please? It works well for me
    - nickstinemates 3y ago
      FWIW also working fine here.
korginator 3y ago
I've been playing with OTEL for a while, with a few backends like Jaeger and Zipkin, and am trying to figure out a way to perform end to end timing measurements across a graph of services triggered by any of several events. Consider this scenario: There is a collection of services that talk to one another, and not all use HTTP. Say agent A0 makes a connection to agent A1, this is observed by service S0 which triggers service S1 to make calls to S2 and S3, which propagate elsewhere and return answers. If we limit the scope of this problem to services explicitly making HTTP calls to other services, we can easily use the Propagators API [1] and use X-B3 headers [2] to propagate the trace context (trace ID, span ID, parent span ID) across this graph, from the origin through to the destination and back. This allows me to query the metrics collector (Jaeger or Zipkin) using this trace ID, look at the timestamps originating at the various services and do a T_end - T_start to determine the overall time taken by one call for a round trip across all the related services. However, this breaks when a subset of these functions cannot propagate the B3 trace IDs for various reasons (e.g., a service is watching a specific state and acts when the state changes). I've been looking into OTEL and other related non-OTEL ways to capture metrics, but it appears there's not much research into this area though it does not seem like a unique or new problem. Has anyone here looked at this scenario, and have you had any luck with OTEL or other mechanisms to get results? [1] https://opentelemetry.io/docs/specs/otel/context/api-propagators/ https://opentelemetry.io/docs/specs/otel/context/api-propaga... [2] https://github.com/openzipkin/b3-propagation https://github.com/openzipkin/b3-propagation [3] https://www.w3.org/TR/trace-context/ https://www.w3.org/TR/trace-context/
samiya0004 3y ago
[dead]

341 points by kevinslin 3y ago

I am very happy with the progress of OpenTelemetry. When I pushed for it years ago, my developers where hesitant (it was new and they never heard about it) but when I revisited the topic a year ago, OpenTelemetry was everywhere in our systems and the log/tracability vendor we have was switching over to it. Thanks to this amazing group!

phito 3y ago

OTEL is awesome. I was part of the team integrating it at my job, it went really smoothly, and it saved us so much time debugging our microservice application. I would say the hardest part was getting other devs to use it, a lot of them are stuck in their own way and did not want to go through the relatively small learning curve...

dottedmag 3y ago

Ugh, protobuf and gRPC as blessed (?!) transports? Thanks, I always wanted to bring 100MB dependency into my code just to send metrics.

wdb 3y ago
Yeah, but you have the same problem when using Prometheus and the native histograms? So far I know it is not supported by the text / openmetrics format only in their protobuf-basaed format version
kortex 3y ago
Where are you getting 100MB from? You don't need the whole protoc toolchain. Well I've never worked with C or C++ and gRPC, but insofar as rust, go, and python are concerned, the increase in container/binary size is a few MB.
Xeoncross 3y ago
If you're using Rust, Go, C, Java, or some other performant language then it shouldn't be much. If you're using a slower scripting language like Ruby or JavaScript then you might have issues.
- valyala 3y ago
  Yeah, including the official Opentelemetry parser for metrics just increases sophisticated Go binary size by 10MB - from 19MB to 29MB [1] [1] https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570#issuecomment-1221981838 https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570...
podoman 3y ago
Just a quick clarification. OTEL is not just a transport, its a specification.
malkia 3y ago
It's 8mb dll when compiled fully as dll(s). Check the https://github.com/open-telemetry/opentelemetry-cpp https://github.com/open-telemetry/opentelemetry-cpp as they've recently added dll support, and I've been keeping a branch that's geared towards more simplicity on deploy (e.g. single .dll, instead of several) - OpenTelemetry's C++ solution though is more flexible, and less size used last I've tried it. Here is mine - https://github.com/malkia/opentelemetry-cpp https://github.com/malkia/opentelemetry-cpp

baz00 3y ago

Is there yet any way of having a front end for this which doesn't significantly dent your revenue stream either in staffing, infrastructure or license fees? We land over 2000 requests/second and it's expensive just keeping logs.

number101010 3y ago
Most observability vendors support OTEL at this point. To plug the OSS project I work on that supports OTEL ingestion: https://github.com/grafana/tempo/ https://github.com/grafana/tempo/
- baz00 3y ago
  Thanks I had no idea Tempo even existed despite using Grafana. Will read into it.
  - candiddevmike 3y ago
    IMO, Jaeger is easier to setup/manage and has a better interface than Grafana/Tempo. It's easy to add Jaeger to your local dev stack so you can have tracing while developing. FWIW I now use Tempo because I have everything else in Grafana (Prometheus, Loki), but I do miss using Jaeger.
    - number101010 3y ago
      > It's easy to add Jaeger to your local dev stack so you can have tracing while developing. Tempo can be spun up with docker compose using a local disk for ephemeral storage/querying: https://github.com/grafana/tempo/blob/main/example/docker-compose/local/docker-compose.yaml https://github.com/grafana/tempo/blob/main/example/docker-co... Maybe this meets your needs? > Jaeger is easier to setup/manage and has a better interface than Grafana/Tempo What do you enjoy about the Jaeger interface? Perhaps it's a gap in Tempo we can improve.
    - sofixa 3y ago
      Jaeger can use multiple backends for storage, including Tempo, so it's not an either/or situation. I'm fairly sure there was an official Grafana-provided Jaeger gRPC plugin for Tempo, but can't easily find it, only this one: https://github.com/flitnetics/jaeger-tempo https://github.com/flitnetics/jaeger-tempo

candiddevmike 3y ago

CNCF needs an OTEL log aggregator

lokar 3y ago
The otel collector/agent can do this. Not storage/query, but aggregation and processing.

monlockandkey 3y ago

I'm new to telemetry. Can someone please explain the relationship between otel, Prometheus, Grafana agent?

wdb 3y ago
It's a standardised way to send metrics to Prometheus. You can have a metrics exporter for Prometheus, e.g. https://opentelemetry.io/ecosystem/registry/?s=prometheus&component=&language= https://opentelemetry.io/ecosystem/registry/?s=prometheus&co... [1] https://opentelemetry.io/ecosystem/registry/?s=prometheus&component=&language= https://opentelemetry.io/ecosystem/registry/?s=prometheus&co... [2] https://opentelemetry.io/docs/specs/otel/metrics/sdk_exporters/otlp/ https://opentelemetry.io/docs/specs/otel/metrics/sdk_exporte...
- podoman 3y ago
  Not only prometheus fwiw. There's lots of providers that support it, e.g.: - datadog - honeycomb.io - highlight.io (I'm a founder) - sentry.io
skrtskrt 3y ago
Prometheus: metrics collector/databases with a defined metrics format. Many metrics/timeseries databases make an effort to be "Prometheus-compatible" as it was kind of the unofficial standard. OTEL: new open standards which are supposed to provide vendor-independent formats which are compatible and composable between metrics, logs, and traces, and easily enable things like deriving metrics from traces (latency would be an obvious one here). Grafana Agent: a metrics? maybe also logs and traces? collector that supports Prometheus, OTEL and other metrics formats, which can forward, sample, and transform the data. Made by Grafana, open source, etc. Grafana's metrics DB Mimir and maybe some others are essentially "more scalable prometheus", and use prometheus metrics format on disc, so one large concern of the Grafana Agent would be converting OTEL metrics to prometheus metrics format for ingestion into the Grafana databases - but the agent has a whole bunch of other functions supported as well. OTEL Collector - non-vendor-specific collector, like Grafana Agent but largely just concerned with allowing collection/ingestion of OTEL and coverting other formats into OTEL. Allows extensions and plugins to be added for other purposes.

sheshe42 3y ago

[flagged]

waynesonfire 3y ago

like a virus, the complexity spreads to fill head count.

zaat 3y ago
Like a reflex, comments blaming solutions for complex challenges are being emitted. Alas, no alternatives are ever offered. Shall we limit all technology to what can be contained in a single box? should we just ban countries, companies and communities from growing past the 500k head count? Please, go on, do say what shall we do with the inherent complexity of the universe, shall we stop physics at Newton and ban particle accelerators? De-legalize storage systems exceeding petabyte? Please, let us hear how complexity can be put back to its old box

deleted 3y ago

[deleted]

skrtskrt 3y ago

since I added OTEL instrumentation to an app, I already easily swapped out the backends from Prometheus and Jaeger to Grafana Mimir and Tempo. The lack of lock-in is fantastic, don't know if I've ever just switched technologies that easily, even SQL databases.

_boffin_ 3y ago

Has anyone found a solution to automatically adding spans automatically for nodejs applications that doesn't involve littering startSpan or startActiveSpans everywhere? I'm currently looking into monkey patching, but that seems dirty.

pondidum 3y ago
Depends what you're trying to accomplish. We're using middleware to start spans for http and message handlers, and then adding `startSpan` where we need. I don't see a problem with `startSpan` everywhere as it's not much noiser than the `log.info` that would be there instead if we didn't have otel.
- _boffin_ 3y ago
  I started my APM journey when using Spring w/ Java and fell in love with how i can trace the entire flow through the entire application and then just config on how many spans i want to send over and what sample rate i want collected. I'd love to accomplish that. What you mentioned is all nice and well (optimal route), but right now, i'm working with some applications that needs it, but has i don't even know how many methods / classes that i'd need to go through and implement it on.
- Spivak 3y ago
  I think my major pain point with them being everywhere is unlike a log they push a new stack frame and closure. They make tracebacks so much more annoying, yes I know the error happened in a start span, thank you. I wish observability actually observed more than it contributed. Once https://peps.python.org/pep-0669/ https://peps.python.org/pep-0669/ is available I'm gonna try my damndest to get otel working through it. Just give me a config file that says what functions you're interested in and I'll do the rest.
  - pondidum 3y ago
    Agreed the amount of stuff showing in stack traces can be annoying - but it does depend on language. I agree 100% in javascript/typescript its annoying, and I would love to get rid of them, In go however, there isn't an extra stack frame. Nor in C# thinking about it. The config file of functions to trace is a really interesting idea. How would you handle wanting to add data to the spans from inside those functions though? e.g. I want to add all kinds of props to the spans that can't be known except inside the function executing.

jeffchao 3y ago

Otel is great for avoiding vendor lock-in and the spec stability is a long time coming — great to see. Also good there’s architecture options aside from side car (there’s also proxy mode), which is nice for situations like deploying on a PaaS. That said the main trouble I’ve had in the past is instability in the SDKs. Java one’s decent. Go, not so much, for example. Traditionally Otel has been awesome for tracing but not so much logs, events, and even metrics (obviously depending on your language). Other than that, auto-instrumentation has been nice. We had this exact functionality when I worked on observability at Netflix and made starting up and maintaining microservices easy and really helped with adoption.

pphysch 3y ago

So I have a VictoriaMetrics (i.e. Prometheus) setup that I am happy with, but haven't touched OpenTelemetry; why should I care about it? Is it a serious solution for log aggregation, but do I need a separate database for logs?

hagen1778 3y ago
You shouldn't unless you want to use the new open source standard for telemetry. You won't benefit from simplicity or performance improvements. It would be quite the opposite. You can check what is the actual cost of open telemetry adoption here [0] But if you ever decide to go this path - VictoriaMetrics supports OpenTelemetry protocol for metrics [1] [0] https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570 https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570 [1] https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#sending-data-via-opentelemetry https://docs.victoriametrics.com/Single-server-VictoriaMetri...

throwaway894345 3y ago

I'd be very interested to hear others experience (either as comments or good blog posts / write ups that they've read) with using Open Telemetry. I haven't used OTEL stuff directly, but I've had always been very disappointed with the telemetry vendors in the past (although as far as I can tell the problems weren't "these vendors don't talk to each other" kinds of problems that I think OTEL aims to solve).

dboreham 3y ago
Experience has been positive. We had to understand (and in places fix/enhance) the agent used in our system but the benefits of it not being an expensive black box were huge. E.g. you can run the whole stack on a laptop and therefore use the same tooling for perf analysis in dev as production.
vmarchaud 3y ago
I've used OpenTelemetry since it's original alpha in 2020. Originally the main issue i had was supporting tracing across common libraries (there wasn't a lot of libraries supported back then). Now (and i recently worked with it) i would say is which protocol is supported by which component: your sdk generate spans/metrics in a specific format, then you send that to a collector that accept a range of protocol versions and finally you can send that to your vendor ... but you need to know which protocol/version it supports. That's not actually something you can do much about considering the sheer size of opentelemetry (both in term of implementation and vendors working on it) and i expect for people implementing nowadays, proto should be pretty stable and my experience should theorically not be the case anymore.

CSDude 3y ago

I love OpenTelemetry and we want to trace almost every span happening. We’d be bankrupt if we went any vendor. We wired opentelemetry with Java magic, 0 effort and pointed to a self hosted Clickhouseand store 700m+ span per day with a 100$ EC2. https://clickhouse.com/blog/how-we-used-clickhouse-to-store-opentelemetry-traces https://clickhouse.com/blog/how-we-used-clickhouse-to-store-...

_boffin_ 3y ago
Are you making sure that you're doing a sample rate, but send over all errors? At a former place, we were doing 5% of non-error traces.
- grogenaut 3y ago
  Careful, we've had systems go down under increased load just emitting errors if they didn't emit much in non error state
  - _boffin_ 3y ago
    Can you go into more detail about your comment, please.
    - cwp 3y ago
      Not the GP, but: Imagine you're sampling successful traces at, say, 1%, but sending all error traces. If your error rate is low, maybe also 1%, your trace volume will be about 2% of your overall request volume. Then you push an update that introduces a bug and now all requests fail with an error, and all those traces get sampled. Your trace volume just increased 50x, and your infrastructure may not be prepared for that.
    - Longwelwind 3y ago
      I think what they means is that if you provisioned your system to receive spans for 5% of non-error requests and a few error requests, if for some random act of god, all the requests yield an error, your span collector will suddenyl receive spans for all requests.

shayel 3y ago

[flagged]

kiitos 3y ago

OpenTelemetry is a marketing-driven project, designed by committee, implemented naively and inefficiently, and guided by the primary goal of allowing Fortune X00 CTOs to tick off some boxes on their strategy roadmap documents. It's not something that anyone with a choice in the matter should be using.

brainbag 3y ago
It looks like every other comment in this thread is favorable to very positive, can you go into more detail about what specifically isn't good about it?
- arp242 3y ago
  Not the previous poster, but I had to implement it a few years ago, and I found it unbelievable complex with dense and difficult to read specifications. I've implemented plenty of protocols and formats from scratch using just the specification, but rarely have I had such difficulty than with OpenTelemetry. I guess this is something you don't notice as merely a "user", but IMHO it's horribly overengineered for what it does and I'm absolutely not a fan. I also disliked the Go tooling for it, which is "badly written Java in Go syntax", or something along these lines. This was 2 years ago. Maybe it's better now, but I doubt it. In our case it was 100% a "tick off some boxes on their strategy roadmap documents" project too and we had much much better solutions. OTel is one of those "yeah, it works ... I guess" but also "ewwww".
  - phillipcarter 3y ago
    I'd recommend trying it out today. In 2021, very few things in OTel were GA and there wasn't nearly as much automatic instrumentation. One of the reasons why you had to dive into the spec was because there was also very little documentation, too, indicative of a heavily in-progress project. All of these things are now different.
    - arp242 3y ago
      I'll be happy to take your word that some implementation issues are now improved, but things like "overengineered" and "way to complex for what it needs to do" really are foundational, and can't just be "fixed" without starting from scratch (and presumably this is all by design in the first place).

jensneuse 3y ago

We're soon launching a GraphQL Analytics, Tracing and Metrics stack on top of OTEL. We've built a custom OTEL exporter to Clickhouse in go, so we can export OTEL traces and Prometheus metrics all to Clickhouse. We've built this stack for our federated GraphQL Gateway but it could essentially work for any OTEL service. If you want to learn more, here's some info: https://wundergraph.com/cosmo https://wundergraph.com/cosmo Were soon going to open source this, so just follow me/us of you're interested. What I really like about this stack is that you can use our end to end solution, but you're not locked into it. We provide a full service, but you can also just use your own OTEL backend if you want to eject.

pradn 3y ago

Most of the comments on this thread are talking about using OpenTelemetry to send metrics/logs to self-hosted collector jobs. Though using a standard library supported by a bunch of collector tools like ClickHouse is useful it itself, there's another benefit too. The specification allows transferring trace IDs across system boundaries. If you and your dependencies all implement the OpenTelemetry spec, then you get spans that reveal in granular detail what happened in the journey. For example, you could learn that it was your database loading a page from disk that took so long, or that a Cloud service's metadata plane was the responsible span for high latency.

infogulch 3y ago

Telemetry seems like it would be a great candidate for columnar storage formats like Parquet or arrow. In particular I expect that it would compress very well, which could reduce telemetry bandwidth consumption / allow for a bigger sample rate. Does anybody have any experience with the intersection of these technologies?

solumos 3y ago
Honeycomb built their own columnar database[0] to support their product. [0] - https://www.honeycomb.io/resources/why-we-built-our-own-distributed-column-store https://www.honeycomb.io/resources/why-we-built-our-own-dist...
vmarchaud 3y ago
Someone from F5 worked on this with OpenTelemetry [0] for Arrow, another effort was done for Parquet but was dropped [1] [0]: https://github.com/open-telemetry/oteps/pull/171 https://github.com/open-telemetry/oteps/pull/171 [1]: https://github.com/open-telemetry/opentelemetry-proto/pull/346 https://github.com/open-telemetry/opentelemetry-proto/pull/3...
- infogulch 3y ago
  Oh nice, thank you (and also solumos) for the links! It looks like oteps/pull/171 (merged June 2023) expanded and superseded the opentelemetry-proto/pull/346 PR (closed Jul 2022) [0]. The former resulted in merging OpenTelemetry Enhancement Proposal 156 [1], with some interesting results especially for 'Phase 2' where they implemented columnar storage end-to-end (see the Validation section [2]): * For univariate time series, OTel Arrow is 2 to 2.5 better in terms of bandwidth reduction ... and the end-to-end speed is 3.1 to 11.2 times faster * For multivariate time series, OTel Arrow is 3 to 7 times better in terms of bandwidth reduction ... Phase 2 has [not yet] been .. estimated but similar results are expected. * For logs, OTel Arrow is 1.6 to 2 times better in terms of bandwidth reduction ... and the end-to-end speed is 2.3 to 4.86 times faster * For traces, OTel Arrow is 1.7 to 2.8 times better in terms of bandwidth reduction ... and the end-to-end speed is 3.37 to 6.16 times faster Pretty exciting results! The OTEL-Arrow adapter has subsequently been donated to the otel community; here's a comment that does a good job of summarizing the results and the recommendations that came out of the test [3]. [0]: https://github.com/open-telemetry/opentelemetry-proto/pull/346#issuecomment-1176314436 https://github.com/open-telemetry/opentelemetry-proto/pull/3... [1]: https://github.com/open-telemetry/oteps/blob/main/text/0156-columnar-encoding.md https://github.com/open-telemetry/oteps/blob/main/text/0156-... [2]: https://github.com/open-telemetry/oteps/blob/main/text/0156-columnar-encoding.md#validation https://github.com/open-telemetry/oteps/blob/main/text/0156-... [3]: https://github.com/open-telemetry/community/issues/1332#issuecomment-1538916171 https://github.com/open-telemetry/community/issues/1332#issu...

TwentyPosts 3y ago

Can someone tell me roughly at which point a tool like OpenTelemetry becomes interesting? It seems complicated, at which point should you bother with it instead of just handrolling simple stuff and eyeballing it?

paulddraper 3y ago
OpenTelemetry is several different things, so you'd have to be more specific. But, for example, if you write a library and you want your downstream users to be able to see the telemetry. OpenTelemetry provides a standardized interface, so you don't need to make assumptions.

no_wizard 3y ago

The OTEL official project libraries don't work well on the web frontend yet. No way of correlating errors to source-maps for instance, at least out of the box. The web browser collector published by the OTEL project uses Zone.js to hijack just about everything in the browser into contexts. If you used modern Angular before, you may recognize zone.js, its a real pain sometimes, and messes with globals, which isn't great, as it can create situations where behavior isn't predictable. I don't know that OTEL has any standard around things like session replays either. Lots of telemetry platforms support this (Sentry, Rollbar, DataDog etc) I think alot of backend teams have really like it. I do like the cross boundary nature of spans where you can follow them by a unique tag across your entire system. I personally find its extremely verbose at times, in terms of the payload it generates, some logging platforms are more compact in this regard, but in practice, I haven't noticed it to be an issue

renke1 3y ago
I think with native Promises (and async/await?) there is currently no way to implement something like Zone.js properly. I've tried to instrument my code manually, but it's really error-prone and verbose. We really need something like https://nodejs.org/api/async_context.html#class-asynclocalstorage https://nodejs.org/api/async_context.html#class-asynclocalst... to be implemented in the browser.
- vmarchaud 3y ago
  You can follow [0] which is currently stage 2 to fix this [0]: https://github.com/tc39/proposal-async-context https://github.com/tc39/proposal-async-context
  - renke1 3y ago
    Thanks! I was only aware of a Zones proposal which was withdrawn I think.
  - no_wizard 3y ago
    In addition to this, is the new (stage 3 even!)explicit resource management proposal[0], supported by TypeScript version >= 5.2[1] Though I agree that async context is better fit for this generally, the ERM should be good for telemetry around objects that have defined lifetime semantics, which is a step in the right direction you can use today [0]: https://github.com/tc39/proposal-explicit-resource-management https://github.com/tc39/proposal-explicit-resource-managemen... [1]: https://www.totaltypescript.com/typescript-5-2-new-keyword-using https://www.totaltypescript.com/typescript-5-2-new-keyword-u...

nickstinemates 3y ago

I run a full OSS otel stack in my home lab. It was a lot of fun to set up, with redundancy and all of the extras. It hasn't been particularly useful, but centralized logging and metrics are pretty to look at in dashboards, which was my motivation. Let's hear some great debugging stories that have been powered by OTEL. I'd love to hear from the horses mouth without marketing speak how it was worth $$$$ collecting and storing all of this info.

nerdjon 3y ago

Slight tangent for a moment, I really really hate the "subscribe" popup that comes up on this blog. It is not clear at all that you can just close it and not give your email since there is no "x" button. Instead it has the incredibly unintuitive "continue reading" under the subscribe button that I did not think would work. Clicking out of it also did not work. Seriously we can and should be better than this. On the topic of open telemetry. I have been long wanting to play with it and see if it offers all of the capabilities when we send the data to datadog. But I have been reluctant to add in another thing to manage and train on if it means that the datadog agent is still necessary for anything outside of the basics. Has anyone else actually tried hooking this up to datadog? Edit: just to be clear, my driving goal of this is not necessarily to keep it with datadog. But that is currently where much of our alerting and logs are now. So the idea would be to switch to open telemetry which would then allow us to (theoretically) move to something else down the line.

monocasa 3y ago
WRT the popup, that's a medium.com thing. I agree that it's very annoying.
- luispauloml 3y ago
  It's not Medium, it's Substack.
remram 3y ago
Personally I found the poll much worse. Got asked for my opinion, clicked on one of the option, and got rewarded with a full-screen "create an account". Hit the back button and hit taken back to HN. Nice.

hu3 3y ago

Speaking about OpenTelemetry, has anyone used https://uptrace.dev https://uptrace.dev ? From a quick glance it seems to be simple, free and open-source deployed as a single Go binary. They use ClickHouse to store data. Almost too good to be true. I'm contemplating them for a new project.

podoman 3y ago
Fwiw, we do the same (https://highlight.io https://highlight.io). Heard good things about uptrace as well.

renke1 3y ago

A few of my colleagues and I had the silly (?) idea that you don't really need logs anymore. Instead of log messages you just attach span events [0]. You then just log the span title and a link to that span in Jaeger; something like [1]. I've only really tried that in my private project, but it felt pretty good. The UI of Jaeger could be a bit better to support that usage, though. Edit: Actually, those colleagues are doing a talk about that topic. So, if you are in Germany and Hannover area, have a look at [2] and search for "Nie wieder Log-Files!". [0]: https://opentelemetry.io/docs/instrumentation/ruby/manual/#add-span-events https://opentelemetry.io/docs/instrumentation/ruby/manual/#a... [1]: tracing.ts:38 Usecase: Handle Auth tracing.ts:47 http://localhost:16686/trace/ec7ffb1e23ddbb8dd770a3f08028666b tracing.ts:38 Adapter: Find Personal Board tracing.ts:47 http://localhost:16686/trace/e22d342316ab0d7d23230864008e27bc tracing.ts:38 Adapter: Find Starred Board List tracing.ts:47 http://localhost:16686/trace/129f89cee26d54cfdc38abea368d9b4e tracing.ts:38 Adapter: Find Personal Board List tracing.ts:47 http://localhost:16686/trace/97948127d77501ff0c65a5db21b21b5a [2]: https://javaforumnord.de/2023/programm/ https://javaforumnord.de/2023/programm/

phillipcarter 3y ago
Depending on how greenfield a project is, you don't even need span events unless you absolutely require a timestamp for a specific operation with no duration. Just using spans for every meaningful operation is like having more powerful structured logs. This approach isn't possible for a lot of systems that have existing logs they need to bring along, but if you're greenfield enough, I'd recommend it.
- glenjamin 3y ago
  Using only spans is surprisingly effective! I did a talk about this at QCon London last year https://www.infoq.com/presentations/event-tracing-monitoring/ https://www.infoq.com/presentations/event-tracing-monitoring...
- paulddraper 3y ago
  Agreed. Same is true for metrics derived from spans. (Though for metrics, you don't need to sample, and for spans you might. So keep in mind.)

streblo 3y ago

I'm hugely disappointed with OpenTelemetry. In my experience, its an over-engineered mess and the out-of-the-box experience is super user hostile. What it purports to be is so far away from what it actually is. Otel markets itself as a universal tracing/metrics/logs format and set of plug and play libraries that has adapters for everything you need. It's actually a bunch of half/poorly implemented libraries with a ton of leaky internals, bad adapters, and actually not a lot of functionality.

cratermoon 3y ago
Okay but can you point to some specific experience and suggest improvements?
- caust1c 3y ago
  Try implementing an OTEL Tracer. This interface is insane, and should really just be a struct. https://github.com/open-telemetry/opentelemetry-go/blob/trace/v1.17.0/trace/trace.go#L336-L383 https://github.com/open-telemetry/opentelemetry-go/blob/trac...
  - ramenmeal 3y ago
    Why were you implementing your own tracer? Don't they publish an implementation?
    - phillipcarter 3y ago
      Yes, all SDKs have a tracer you can just use. While you can technically create your own tracer, you're officially in Hard Mode territory - there's no highly extensible system I'm aware of that makes swapping core concepts (that already have a default) easy.
  - skrtskrt 3y ago
    how is this interface insane? It's a list of the behaviors you need to implement if you're rolling your own OTEL Tracer Span implementation, and not using one of the multiple available. In contrast, OpenTracing's interfaces had hardly any required methods, so you had to do a runtime type-cast to the whichever implementation you were using in order to access anything useful on the Span like the OperationName.

bilalq 3y ago

OpenTelemetry is being pushed as a replacement for AWS X-Ray SDKs by AWS, but it's in such a broken state for Lambda right now. A 200-500% performance penalty for using it is insane[1][2]. [1]: https://github.com/aws-observability/aws-otel-lambda/issues/228 https://github.com/aws-observability/aws-otel-lambda/issues/... [2]: https://github.com/open-telemetry/opentelemetry-lambda/issues/263 https://github.com/open-telemetry/opentelemetry-lambda/issue...

hinkley 3y ago
How do you use it with Lambda? Delta mode? Or tons of labels?
- bilalq 3y ago
  I use X-Ray directly via AWS Powertools for Lambda[1] instead of going through OTel. The underlying AWS X-Ray SDK team pushes OTel pretty hard in their README[2] and Github issues, but the current situation is that OpenTelemetry is just not production ready for serverless applications. [1]: https://docs.powertools.aws.dev/lambda/typescript/latest/ https://docs.powertools.aws.dev/lambda/typescript/latest/ [2]: https://github.com/aws/aws-xray-sdk-node https://github.com/aws/aws-xray-sdk-node

podoman 3y ago

Disclaimer: I'm the founder of an observability company (Highlight.io). OpenTelemetry has been INCREDIBLY valuable to us. Not only has it made it super fast to build out SDKs for our customers, but the fact that its maintained actively gives us confidence that we're rolling out stable logic to customers' environments. I agree with the author that OpenTelemetry has succeeded, and its pretty obvious from the fact that most major observability vendors support. In short, to a developer it may not seem like its particularly valuable because a metrics/logs/traces API is quite simple whether or not you use OTEL. But the fact that this is an industry wide spec is where it becomes powerful. A few

m1keil 3y ago
It's quite understandable that a "o11y protocol" is very valuable to a o11y service provider (especially the smaller ones) since it reduces the cost of vendor swap.

paulddraper 3y ago

Two problems with OpenTelemetry: 1. It doesn't know what the hell it is. Is it a semantic standard? Is a protocol? It is a facade? It is a library? What layer of abstraction does it provide? Answer: All of the above! All the things! All the layers! 2. No one from OpenTelemetry has actually tried instrumenting a library. And if they have, they haven't the first suggestion on how instrumenters should actually use metrics, traces, and logs. Do you write to all three? To one? I asked this question two years ago, zero answers :( [1] [1] https://github.com/open-telemetry/opentelemetry-specification/issues/2107 https://github.com/open-telemetry/opentelemetry-specificatio...

jauntywundrkind 3y ago
Good deck of questions but I don't think they matter. I don't think those are answerable questions for observability, be it OpenTelemetry or other proprietary systems. You can go read the leading observability companies web pages and they'll have a 4 page writeup on custom instrumentation. That's not much, just covers very elementary basics! It's not like OTel is behind. The answer just tends heavily towards "it depends." Once you have experience - OTel or other - you can work through these things that might confound a neophyte.
- morelisp 3y ago
  The problem is they keep making the OTel tooling worse for working through these things, because the people writing the OTel tooling broadly aren't the people actually trying to monitor things. Even before OTel, plain Prometheus client libraries suffered from this.
coxley 3y ago
Agreed. Some things they suggest aren't actually possible with their SDKs. For example, you cannot define a histogram's buckets near where you define the histogram. You have to give the global exporter (or w/e the type is) a list of "overrides" that map each histogram name => their buckets. This makes it extremely ugly when you have libraries that emit metrics. https://github.com/open-telemetry/opentelemetry-go/issues/3826 https://github.com/open-telemetry/opentelemetry-go/issues/38...

kevinslin 3y ago

author of the post here - was inspired to write this post after working with OTEL for a few months - realized that OTEL had a ridiculously large surface area that most people (at least myself) might not be aware of I see a lot of comments about how overly complex OTEL is. I don't disagree with this. in some sense, OTEL is very much the k8 for observability (good and bad) The good is that it is a standard that can support every conceivable use case and has wide industry adoption The bad is that there is inherent complexity in needing to support the wide array of use cases

chrisweekly 3y ago

Non-paywalled URL: https://archive.is/NULpZ https://archive.is/NULpZ

enahs-sf 3y ago

Someone needs to build the Segment of APM. Datadog lock-in is real and the bill is high.

Havoc 3y ago

Very split opinions in the comments. Seems people either love it or hate it with nothing in between

enahs-sf 3y ago

I just want Segment but for APM to reduce vendor lock-in.

nijave 3y ago

I've had pretty decent experience with .NET + Grafana Cloud (hosted Tempo) You can generate RED metrics then tail based sample and send a subset of full traces. As with many new things, there's varying maturity in client libraries and still things missing. The Redis .NET client (maybe others?) wasn't very good (maybe alpha/beta quality) but the other .NET stuff seemed reasonable. At least with .NET, it integrates with the existing System.Diagnostics API. I think libraries in some other languages (go?) are a bit clumsier depending on what type of diagnostic/debug/event/performance/reflection APIs that language runtime exposes for interspection.

edenfed 3y ago

Disclaimer: I am one of the maintainers Many comments complain about the complexity of using OpenTelemetry, I recommend checking out Odigos, an open-source project which makes working with OpenTelemetry much easier: https://github.com/keyval-dev/odigos https://github.com/keyval-dev/odigos We combine OpenTelemetry and eBPF to instantly generate distributed traces without any code changes.

pl4nty 3y ago
How does this compare to the OTel Operator feature-wise? I'm loving auto-instrumentation with Grafana Cloud, but operator config is painful and I've hit (and helped fix) a lot of bugs
williamscs 3y ago
Hi! Just wanted to let you know I'm getting a 500 error on your docs page. https://docs.odigos.io/ https://docs.odigos.io/
- edenfed 3y ago
  Can you try again please? It works well for me
  - nickstinemates 3y ago
    FWIW also working fine here.

korginator 3y ago

I've been playing with OTEL for a while, with a few backends like Jaeger and Zipkin, and am trying to figure out a way to perform end to end timing measurements across a graph of services triggered by any of several events. Consider this scenario: There is a collection of services that talk to one another, and not all use HTTP. Say agent A0 makes a connection to agent A1, this is observed by service S0 which triggers service S1 to make calls to S2 and S3, which propagate elsewhere and return answers. If we limit the scope of this problem to services explicitly making HTTP calls to other services, we can easily use the Propagators API [1] and use X-B3 headers [2] to propagate the trace context (trace ID, span ID, parent span ID) across this graph, from the origin through to the destination and back. This allows me to query the metrics collector (Jaeger or Zipkin) using this trace ID, look at the timestamps originating at the various services and do a T_end - T_start to determine the overall time taken by one call for a round trip across all the related services. However, this breaks when a subset of these functions cannot propagate the B3 trace IDs for various reasons (e.g., a service is watching a specific state and acts when the state changes). I've been looking into OTEL and other related non-OTEL ways to capture metrics, but it appears there's not much research into this area though it does not seem like a unique or new problem. Has anyone here looked at this scenario, and have you had any luck with OTEL or other mechanisms to get results? [1] https://opentelemetry.io/docs/specs/otel/context/api-propagators/ https://opentelemetry.io/docs/specs/otel/context/api-propaga... [2] https://github.com/openzipkin/b3-propagation https://github.com/openzipkin/b3-propagation [3] https://www.w3.org/TR/trace-context/ https://www.w3.org/TR/trace-context/

samiya0004 3y ago

[dead]