8 ms·
OpenTelemetry
- spullara 5y agoI hate that this specification and most of the other ones use spans that have a beginning and end rather than events that start and end the span. What if it crashes before it sends out the span? What if it is taking a very long time to complete?
- rad_gruchalski 5y agoYou can emit multiple spans yourself, no?
- novok 5y agoData query wise it's significantly more complicated to compile 2 db row spans than it is to do queries on one db row spans. The library that makes and manages those spans before uploading can deal with those use cases internally if it wants, a lot of these systems have support for single point events too.
- beeandapenguin 5y agoFor the long running scenario, why not use Links? The specification describes[1] a similar scenario and how links can be leveraged to causally relate spans even across trace boundaries. As for the crashing scenario, this seems like an application-level concern. Ideally it is not waiting until the crash to send traces. Depending on the environment, the application could handle the crash and deliver the telemetry leading up to it before shutting down. [1]: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/overview.md#links-between-spans https://github.com/open-telemetry/opentelemetry-specificatio...
- jeffbee 5y agoYes, logging the beginning and ending of a span to your observability platform is a common and valuable approach, since you can figure out from the open spans of a dead process what requests might have been running when it died, making it easier to diagnose requests of death.
- dyladan 5y agoThe span is simply the way the data is modeled. The way the tracing works is it calls a component called a Span Processor on both span start and end. It would be possible to implement a Span Processor that sends span_start and span_end events to your backend without waiting for the span to start. In terms of crashing and taking a long time to complete, these are problems that are very difficult to solve on the backend as well. Simply having the start event without the end event is not enough info to say for sure that there will _never_ be an end event. For low data volumes and simple use-cases this may not seem like a big deal, but it gets complex extremely quickly.
- unportant 5y agoI think you are right and data model should have been event based rather than span based. https://medium.com/opentracing/open-for-event-based-tracing-a326c295f2a2 https://medium.com/opentracing/open-for-event-based-tracing-... digs into that topic and basically says that distributed tracing is about causality & partial ordering which bring us back to the 70's with Lamport's logical clocks and all following work. However, this design "flaw" is well known and seems assumed. I'm not able to find relevant GitHub issues right now, but I remember this topic being discussed on OpenTracing or OpenTelemetry bug tracker and the outcome was something like "Spans might not be the best data model, but people are now used to it and we have to ship the spec within a reasonable time, so let's stick to it". Edit: https://github.com/open-telemetry/opentelemetry-specification/issues/20 https://github.com/open-telemetry/opentelemetry-specificatio... might be relevant to your concerns.
- dmitrygr 5y agoNot a mention of "opt-in only, because we respect privacy"... Please stop helping spy on users! Be the change you want to see in the world!
- sveiss 5y agoThat's because this tool has nothing to do with "spying on users". It's a set of tools and a specification for instrumenting applications and collecting logs/metrics/traces from them. It's used to ask questions like "why is this page slow to load?" and get answers like "because it talks to the billing microservice which has a really slow SQL query". This isn't the same sort of telemetry that desktop software vendors collect.
- dmitrygr 5y agoSadly, it seems you are wrong: https://opentelemetry.io/docs/js/ https://opentelemetry.io/docs/js/ "the browser" That means they are running code on my machine and exfiltrating data out.
- rad_gruchalski 5y agoDefine “they”, please.
- robbedpeter 5y agoThey as in "they" who are receiving the data being collected about the user. It isn't hard to see who the bad guys are. If this is implemented in a totally transparent, user controlled, consensual way, and the collected data made ephemeral and subject to user request for deletion and viewing on demand while being stored, then there's nothing bad going on. Anything less is unacceptable - a line is being crossed. Right now a majority of the encounters users have with telemetry is totally opaque and behind the scenes, nonconsensual except for the barest shreds of a EULA or button click fig leaf.
- 5y ago
- iddan 5y agoI really hope we’ll have an open standard for tracing. Right now the common solutions are vendor-locking and can’t be integrated in OSS projects
- Thaxll 5y agoOpen tracing works just fine.
- dyladan 5y agoOpenTracing and OpenCensus have been merged into the OpenTelemetry project. I think OT isn't sunset officially yet, but the future is OTel.
- phillipcarter 5y agoThat's definitely a goal of OTel! And it seems to be working. Anecdotally, OTel support has been very important for my workplace (we sell observability tooling https://www.honeycomb.io/ https://www.honeycomb.io/) in terms of prospects. I think a lot of decision makers at companies are realizing that they can't afford not to have vendor-neutral instrumentation. And tracing is GA in most SDKs as well, so you should feel ready to adopt it. Several vendors and OSS tools support it (or you can export your data to another format with the OTel Collector).
- lizthegrey 5y agoPhillip also wrote https://leaddev.com/monitoring-observability/rise-opentelemetry https://leaddev.com/monitoring-observability/rise-openteleme... today, but is too shy to plug it :)
- bastijn 5y agoTook me a while to figure out OTel is OpenTelemetry here and you were not proposing another spec. Comment for others to realize this.
- splittingTimes 5y agoWe are trying to get a better understanding of how customers use our desktop application. We are urged by management to use Qualtrics, but it feels like it is not the right tool for the job. How would you say does opentelemetry compare to Qualtrics? Is Qualtrics a tool where users need to actively fill out surveys and opentelemetry collects passively user interactions? There is also other names floating around in this space like DataDog or OpsTrace, where I am not clear how to place them.
- lizthegrey 5y agoOpenTelemetry allows you to measure performance of your frontend and backend, but it's not a survey tool. DataDog is one of many compatible backends that works with OpenTelemetry, but you can also use Jaeger OSS, Honeycomb, Lightstep, and others.
- splittingTimes 5y agoStrictly only the performance or can it also answer usage questions like: When customers go through this workflow step, they use this option/button 90% of the time and the other options are used rarely?
- lizthegrey 5y agoBI/funnel analytics is a use case that you might have more luck with https://www.tinybird.co/ https://www.tinybird.co/ instead.
- kall 5y agoIt may be possible, but it will not be pleasant to use tracing to answer those kinds of questions. Sounds like something you can do with off the shelf product analytics tools like Amplitude or Mixpanel. Do you already have something like that?
- cagmz 5y agoWe're using New Relic track performance (API response times, DB load, etc) as well as to track actions that users make on our front ends and APIs.
- lizthegrey 5y agoHello, Hacker News! https://share.getcloudapp.com/4guP9P6p https://share.getcloudapp.com/4guP9P6p (stats gathered from opentelemetry-js)
- lizthegrey 5y agofor extra fairness and showcasing interoperability, here's the same data rendered in more than one backend! https://twitter.com/lizthegrey/status/1453525797243797510 https://twitter.com/lizthegrey/status/1453525797243797510
- zls 5y agoI love OpenTelemetry, but for Java, I wish adoption would slow down until the implementation catches up. For example, if you visit the OpenCensus site, it tells you on every page that OpenTelemetry is the way to go now. Yet OpenTelemetry's Java metrics implementation is still listed as alpha, and functionality as basic as tags didn't work when I last tried it. Or, I've been working with Datadog lately, and I want to add dynamic span metadata. My options appear to be either to wire up the (deprecated) OpenTracing client, or set up a full collector/agent suite of OpenTelemetry processes (whose default tutorial configurations appear to be invalid in places). worth I guess, but bumpier than I'd like.
- tonyhb 5y agoYeah, OpenTelemetry was OpenCensus, and before that it was OpenTracing. The go client faces some of the same churn; metrics are in alpha and the API isn't documented, though spans and traces are on more solid ground than the Java client it seems. It's been a lot of change and I can't wait for things to settle down and for companies to stop trying to get their piece. I feel like everyone (Datadog, NewRelic, Google, etc.) saw Lightstep as a competitor and wanted to get involved with a competing spec, which is how we ended up in this consensus mess. Still, it's good progress and it's so much better than raw logs. Would recommend.
- lizthegrey 5y agoLightstep was one of the originators of the combined spec -- but yes, the churn has been really painful, and we're sorry for the pre-1.0 pain. Now that the spec is 1.0 and most of the language SIGs have put out 1.0 API/SDK releases for tracing, hopefully that pain will reced.
- Game_Ender 5y agoDo you have an insight on the timeline for stabilizing the metrics and logging specs?
- phillipcarter 5y ago
- jakozaur 5y agoI belive OpenTelemetry will make distributed tracing mainstream. Traces are the best tool to debug problems, yet as their are expensive (both in dev work and license) to implement they are way less popular than logs. Historicaly APM (application performance monitoring) also owned collection, vendor locked-in and didn’t evolve price per value. Open-source collection standard allows users to choose or switch to the best vendor. Personally, I recommend Sumo Logic (disclaimer: I work there). We have rich full-text query capabilities on spans with parsing and aggreagations on query time…
- heyzk 5y agoI'm curious, who's your target customer? Should small teams be looking early on or is this more of a mid-sized company thing?
- smarx007 5y agoAt almost $100 per GB of logs ingested, I think it’s targeted at SV startups.
- jakozaur 5y agoThough Sumo Logic started by targeting mid-size to enterprise, recently it invests more in self-serve: 1. Free plan includes Tracing. 2. There is monthly self-serve subscription available.
- sjtindell 5y ago
- gclawes 5y agoHow does this relate to OpenMetrics, if at all?
- bastijn 5y agoOpenMetrics relates to the metrics part of OpenTelemetry, where the latter is currently in alpha/experimental phase. OpenTelemetry has a statement on being interoperable with OpenMetrics for both accepting, forwarding and generating in the OpenMetrics format [0]. So they are two separate CNCF projects where OpenMetrics has overlap with a part of OpenTelemetry. [0] https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/openmetrics-guidelines.md https://github.com/open-telemetry/opentelemetry-specificatio...
- jeffbee 5y agoIt is nice that the proto files are unambiguous that trace time is represented as unix epoch nanos, and it is declared `fixed64` the way Jesus intended, but it's too bad the specification made it to 1.0 without simply mandating that timestamps are always in unix epoch nanos. Having uncertainty about the precision of timestamps has caused and will continue to cause loads of bugs. If there is some language platform out there that can't figure out the time in nanos, that language platform should simply be ignored.
- erik_seaberg 5y agoFrom what I’ve read, you have to accommodate ±1 ms NTP clock skew unless you have an atomic clock or GPS source in each datacenter, so the extra precision isn’t reliable.
- jeffbee 5y agoThat's fine. Many clocks are off by much more than 1ms. But the timestamps should be represented as nanos everywhere.
- dikei 5y agoOpen Telemetry seems super complex compared to our current setup of Micrometer with Prometheus and Log4j2 with Json output. I wonder if it's worth it to replace both metrics and logging just to take advantage of OT's tracing capability.
- bradbeattie 5y agoA key benefit you might not be getting that OpenTelemetry (and Fluentd for logs in the interim) provides is routing. The ability to trial multiple downstream storage solutions is useful!
- dikei 5y agoBoth Micrometer and using log4j2 with a log collector (we're using fluent-bit) already support writing to multiple downstream endpoints though, so not a clear win for OTel.
- lizthegrey 5y agoyou could also choose to feed json logs with span and trace ids through, but you'd then have to manually track the context, parent ids, propagation, etc. which OTel takes care of for you.
- edge17 5y agoForgive me for having no idea what this project is, but is this a telemetry library for web infrastructure, or is this an open source telemetry library for monitoring high frequency signals from hardware devices, or...?
- lizthegrey 5y agothe former (web+backend), we're part of the Cloud Native Computing Foundation.
- xgbi 5y agoAnybody has a recommendation on litterature/blogs addressing the subject of observability? I’m contemplating an SRE job offer and although I have my share of knowledge, I am still out of my league whenever I glance at the cncf map in the observability category. I feel that the field has exploded with new components and new ways, and I’m sometimes afraid of taking the wrong path.
- deleted 5y ago[deleted]
- matt2115 5y agoThis is a good place to start if you're interested in OTEL https://www.aspecto.io/opentelemetry-bootcamp/ https://www.aspecto.io/opentelemetry-bootcamp/
- mhausenblas 5y ago* Observability Engineering (Charity Majors, Liz Fong-Jones, George Miranda) * Software Telemetry (Jamie Riedesel)
- yuppiepuppie 5y agoAs someone not familiar with this project, and looking at the landing page, I have no idea what this does. I understand it has something to do with telemetry, but for what? And what would it help me with? How would I use it. I see a telescope. Is it space related? Telemetry of satellites? Just my two cents as someone not familiar with the project, and has no idea how to use it or how it would solve a problem I have.
- Game_Ender 5y agoDoes this section here help? It’s below the fold on Mobile, but it is on the front page of the site. > OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.
- conradludgate 5y agoIt's a collection of open standards and libraries for exporting/injesting Traces and Metrics from your applications (and logging is in the works)