4 ms·

> If there’s anything that’s an absolute must in my Observability to-do list, it’s getting org-wide acceptance of OpenTelemetry (OTel) for data-collection. Ope

> If there’s anything that’s an absolute must in my Observability to-do list, it’s getting org-wide acceptance of OpenTelemetry (OTel) for data-collection. OpenTelemetry exists to allow CTOs to tick boxes on requirements checklists, and help to get the CNCF technology stack into the F500 space. The project is textbook design by committee, with seemingly infinite scope but very little acknowledgement of the actual technical constraints of the problem space. It claims it can solve a lot of problems, but I can say from first hand experience that it does not, and cannot, deliver on those claims. It's not good. Nobody with a choice should be using it.

jalons 5y ago
Recommended alternatives?
- sagichmal 5y ago
  At the moment, the capabilities of technology simply don't permit the level of... generic-ness? that vendors like OTel -- and customers really -- want. If you try to push everything through a single pipeline, as a single kind of data, you necessarily lock yourself out of most of the value you should be getting. You need a heterogeneous stack, with specific tools for specific purposes. Opinions vary but my experience is that Prometheus-style metrics are by far the most important thing to invest in, and deliver the majority of the "observability value" to the broadest set of architectures. Tracing systems like Lightstep can be super useful, too, but to deliver value they need a lot more end-to-end integration effort, and the cost of setting it up can very very easily outweigh the benefits it provides. I've come to believe that logs are a trap. Everyone understands them intuitively, so they feel comfortable using them for basically everything, without thinking very critically about the ramifications. And even when logs are structured, they have no substantive schema, so there's no backpressure, so to speak, on usage. So the signal/noise ratio almost immediately goes negative. And they just occupy enormous amounts of time and memory to manage, process, analyze, etc. Although I'm sure it's not a universal truth, I've found that everything you might normally think to log is actually much better served by in-process request tracing. In general that means maintaining the log events of the most recent N requests for all of a set of application-defined categories. This is basically a real-time view of a system, with history proportional to, I dunno, rarity? of the request class. You don't ship these anywhere, you ask the applications directly. It seems weird to describe but it's just vastly more efficient to manage, and equally if not more useful for debug and triage. YMMV.

jalons 5y ago

Recommended alternatives?

sagichmal 5y ago
At the moment, the capabilities of technology simply don't permit the level of... generic-ness? that vendors like OTel -- and customers really -- want. If you try to push everything through a single pipeline, as a single kind of data, you necessarily lock yourself out of most of the value you should be getting. You need a heterogeneous stack, with specific tools for specific purposes. Opinions vary but my experience is that Prometheus-style metrics are by far the most important thing to invest in, and deliver the majority of the "observability value" to the broadest set of architectures. Tracing systems like Lightstep can be super useful, too, but to deliver value they need a lot more end-to-end integration effort, and the cost of setting it up can very very easily outweigh the benefits it provides. I've come to believe that logs are a trap. Everyone understands them intuitively, so they feel comfortable using them for basically everything, without thinking very critically about the ramifications. And even when logs are structured, they have no substantive schema, so there's no backpressure, so to speak, on usage. So the signal/noise ratio almost immediately goes negative. And they just occupy enormous amounts of time and memory to manage, process, analyze, etc. Although I'm sure it's not a universal truth, I've found that everything you might normally think to log is actually much better served by in-process request tracing. In general that means maintaining the log events of the most recent N requests for all of a set of application-defined categories. This is basically a real-time view of a system, with history proportional to, I dunno, rarity? of the request class. You don't ship these anywhere, you ask the applications directly. It seems weird to describe but it's just vastly more efficient to manage, and equally if not more useful for debug and triage. YMMV.