6 ms·

I've seen it hyped up like crazy, what's so bad about it and what is the superior alternative?

preseinger 4y ago
Instrumentation of a given system is by definition always "larger" than the nominal traffic which flows through that system. That is, given some request R, the possible observable metadata of that request R is essentially infinite. So when you're designing an observability system, the whole ball game is about capturing and constraining the cardinality of that metadata. Observability as a field is _entirely_ about reducing the volume of telemetry data such that it can be usefully captured, aggregated, and queried by operators. That's why we have metrics, traces, and logs as distinct things. They're all optimizations of the same underlying information, each optimized for different consumer use cases. Concretely: if user U1 does an HTTP GET to /spotify/track/123, that's perhaps 10KB of production traffic, but easily 10MB of telemetry traffic. In effect that telemetry can be modeled as a huge key/val map of metadata, but you can't do it that way and remain efficient, you have to optimize for observability use cases. You have to increment a cardinality-bound set of metric counters for the request outcomes, and maybe emit some best-effort trace data for the request ID, and etc. etc. — _as separate things_! The engineering costs dominate the design. But OpenTelemetry says this isn't the case. OpenTelemetry says that whatever requirements are on FX00 company CTO feature checklists are valid a priori, and commits to doing whatever is necessary to satisfy them. That's because OpenTelemetry is evaluated not on any technical merits, but on the adoption rate of the CNCF stack among those FX00 companies. OpenTelemetry is explicitly and exclusively a thing meant to tick off an "observability" checklist item on the checklist of a FX00 CTO's due diligence form. That's it. That's the only goal. Nobody with a choice should be using it. Read the source code, it's abysmal. Alternatives? Write code that leverages each pillar of observability directly. There's no short-cut. That's the whole point.
- phillipcarter 4y ago
  > So when you're designing an observability system, the whole ball game is about capturing and constraining the cardinality of that metadata. If your world is only time series databases that struggle with cardinality, then sure. Fortunately, there's a lot more tools out there, some of which do just fine with high cardinality data. I don't really agree much with your entire comment. I think you're looking at observability through the lens of 2010s-era tools, and falling deeply into the trap of thinking that this is the only way to do things.
  - preseinger 4y ago
    My core claim is that telemetry data for a system is always more than the production data for that system. Yes? No? Assuming yes, everything else I'm claiming is noncontroversial.
    - phillipcarter 4y ago
      I count about a dozen claims, so I'm not really sure what to begin with. But I disagree with your comment here too.
      - preseinger 4y ago
        (shrug) OK. I'm a domain expert here, but that doesn't mean my claims are infallible, of course. Your call. Good luck.
        LewisJEllis 4y ago
        most or all of the observability domain experts i'm familiar with either stopped talking about "pillars" several years ago, or have been actively speaking against the "3 pillars" framing for several years most of your takes here sound like they're from somewhere around 2016-2018
    - remram 4y ago
      I don't think anyone opposes that, but I don't think that's your "core claim". Your core claim is that this is a huge problem, one that makes OpenTelemetry "a technological dead-end", and that "the whole ball game is about capturing and constraining the cardinality of that metadata". That is what it would be nice to have you justify some more. After all, the same could be said of logs, or internal traffic when using microservices/DBMS (request cardinality/traffic will be multiplied).
- harpratap 4y ago
  > Alternatives? Write code that leverages each pillar of observability directly. There's no short-cut. That's the whole point. I think I am not able to follow you correctly. Is your entire point that auto-instrumentation is too much and one should default to manual instrumentation instead?
  - preseinger 4y ago
    You can't delegate "instrumentation" as a monolithic concern to a single authority or package or library like OpenTelemetry and expect to get anything but noise on the other side. And yeah, basically any vendor claiming to do do "auto-instrumentation" is selling snake-oil. Expensive snake-oil, too! Observability is not something that a vendor can provide without meaningful and deep integration in your infrastructure. You have to do some amount of work, and IMO few vendors deliver value beyond what a single engineer can produce with with a basic internal Prometheus infrastructure + short-term log aggregation.
    - harpratap 4y ago
      But OpenTelemetry never claimed to be an auto-instrumentation only library, you can very well just manually instrument your application with the metrics you want and export it your self managed prometheus + log infrastructure. In fact, OTel makes it easier for you if in the future you might want to move to a better self managed TSDB + Log Infra because you know it most likely have an exporter ready with zero effort to re-instrument your metrics or logs
      - preseinger 4y ago
        You can definitely define an abstract concept of telemetry which generalizes over use cases, and which can be expressed as a single schema that can be collected generically. The problem is that this definition is too general to be practically useful. The whole ball game for observability systems is optimization for specific consumption use cases. That _must_ occur at the point of origin, it _cannot_ be deferred. OTel says that it's possible to define a general-purpose exporter for arbitrary telemetry data, and that specialization and optimization of that generalized data can be done later, downstream. This is simply not true.
- mavelikara 4y ago
  Few of the newer vendors have evangelized the "capture everything, ignore costs" mantra leading people to places where angels fear to tread.

Instrumentation of a given system is by definition always "larger" than the nominal traffic which flows through that system. That is, given some request R, the possible observable metadata of that request R is essentially infinite. So when you're designing an observability system, the whole ball game is about capturing and constraining the cardinality of that metadata. Observability as a field is _entirely_ about reducing the volume of telemetry data such that it can be usefully captured, aggregated, and queried by operators. That's why we have metrics, traces, and logs as distinct things. They're all optimizations of the same underlying information, each optimized for different consumer use cases. Concretely: if user U1 does an HTTP GET to /spotify/track/123, that's perhaps 10KB of production traffic, but easily 10MB of telemetry traffic. In effect that telemetry can be modeled as a huge key/val map of metadata, but you can't do it that way and remain efficient, you have to optimize for observability use cases. You have to increment a cardinality-bound set of metric counters for the request outcomes, and maybe emit some best-effort trace data for the request ID, and etc. etc. — _as separate things_! The engineering costs dominate the design. But OpenTelemetry says this isn't the case. OpenTelemetry says that whatever requirements are on FX00 company CTO feature checklists are valid a priori, and commits to doing whatever is necessary to satisfy them. That's because OpenTelemetry is evaluated not on any technical merits, but on the adoption rate of the CNCF stack among those FX00 companies. OpenTelemetry is explicitly and exclusively a thing meant to tick off an "observability" checklist item on the checklist of a FX00 CTO's due diligence form. That's it. That's the only goal. Nobody with a choice should be using it. Read the source code, it's abysmal. Alternatives? Write code that leverages each pillar of observability directly. There's no short-cut. That's the whole point.

phillipcarter 4y ago
> So when you're designing an observability system, the whole ball game is about capturing and constraining the cardinality of that metadata. If your world is only time series databases that struggle with cardinality, then sure. Fortunately, there's a lot more tools out there, some of which do just fine with high cardinality data. I don't really agree much with your entire comment. I think you're looking at observability through the lens of 2010s-era tools, and falling deeply into the trap of thinking that this is the only way to do things.
- preseinger 4y ago
  My core claim is that telemetry data for a system is always more than the production data for that system. Yes? No? Assuming yes, everything else I'm claiming is noncontroversial.
  - phillipcarter 4y ago
    I count about a dozen claims, so I'm not really sure what to begin with. But I disagree with your comment here too.
    - preseinger 4y ago
      (shrug) OK. I'm a domain expert here, but that doesn't mean my claims are infallible, of course. Your call. Good luck.
      - LewisJEllis 4y ago
        most or all of the observability domain experts i'm familiar with either stopped talking about "pillars" several years ago, or have been actively speaking against the "3 pillars" framing for several years most of your takes here sound like they're from somewhere around 2016-2018
  - remram 4y ago
    I don't think anyone opposes that, but I don't think that's your "core claim". Your core claim is that this is a huge problem, one that makes OpenTelemetry "a technological dead-end", and that "the whole ball game is about capturing and constraining the cardinality of that metadata". That is what it would be nice to have you justify some more. After all, the same could be said of logs, or internal traffic when using microservices/DBMS (request cardinality/traffic will be multiplied).
harpratap 4y ago
> Alternatives? Write code that leverages each pillar of observability directly. There's no short-cut. That's the whole point. I think I am not able to follow you correctly. Is your entire point that auto-instrumentation is too much and one should default to manual instrumentation instead?
- preseinger 4y ago
  You can't delegate "instrumentation" as a monolithic concern to a single authority or package or library like OpenTelemetry and expect to get anything but noise on the other side. And yeah, basically any vendor claiming to do do "auto-instrumentation" is selling snake-oil. Expensive snake-oil, too! Observability is not something that a vendor can provide without meaningful and deep integration in your infrastructure. You have to do some amount of work, and IMO few vendors deliver value beyond what a single engineer can produce with with a basic internal Prometheus infrastructure + short-term log aggregation.
  - harpratap 4y ago
    But OpenTelemetry never claimed to be an auto-instrumentation only library, you can very well just manually instrument your application with the metrics you want and export it your self managed prometheus + log infrastructure. In fact, OTel makes it easier for you if in the future you might want to move to a better self managed TSDB + Log Infra because you know it most likely have an exporter ready with zero effort to re-instrument your metrics or logs
    - preseinger 4y ago
      You can definitely define an abstract concept of telemetry which generalizes over use cases, and which can be expressed as a single schema that can be collected generically. The problem is that this definition is too general to be practically useful. The whole ball game for observability systems is optimization for specific consumption use cases. That _must_ occur at the point of origin, it _cannot_ be deferred. OTel says that it's possible to define a general-purpose exporter for arbitrary telemetry data, and that specialization and optimization of that generalized data can be done later, downstream. This is simply not true.
mavelikara 4y ago
Few of the newer vendors have evangelized the "capture everything, ignore costs" mantra leading people to places where angels fear to tread.