7 ms·

OpenTelemetry is moribund and a technological dead-end. Godspeed to any org that builds on it.

by preseinger 4y ago

ren_engineer 4y ago
I've seen it hyped up like crazy, what's so bad about it and what is the superior alternative?
- preseinger 4y ago
  Instrumentation of a given system is by definition always "larger" than the nominal traffic which flows through that system. That is, given some request R, the possible observable metadata of that request R is essentially infinite. So when you're designing an observability system, the whole ball game is about capturing and constraining the cardinality of that metadata. Observability as a field is _entirely_ about reducing the volume of telemetry data such that it can be usefully captured, aggregated, and queried by operators. That's why we have metrics, traces, and logs as distinct things. They're all optimizations of the same underlying information, each optimized for different consumer use cases. Concretely: if user U1 does an HTTP GET to /spotify/track/123, that's perhaps 10KB of production traffic, but easily 10MB of telemetry traffic. In effect that telemetry can be modeled as a huge key/val map of metadata, but you can't do it that way and remain efficient, you have to optimize for observability use cases. You have to increment a cardinality-bound set of metric counters for the request outcomes, and maybe emit some best-effort trace data for the request ID, and etc. etc. — _as separate things_! The engineering costs dominate the design. But OpenTelemetry says this isn't the case. OpenTelemetry says that whatever requirements are on FX00 company CTO feature checklists are valid a priori, and commits to doing whatever is necessary to satisfy them. That's because OpenTelemetry is evaluated not on any technical merits, but on the adoption rate of the CNCF stack among those FX00 companies. OpenTelemetry is explicitly and exclusively a thing meant to tick off an "observability" checklist item on the checklist of a FX00 CTO's due diligence form. That's it. That's the only goal. Nobody with a choice should be using it. Read the source code, it's abysmal. Alternatives? Write code that leverages each pillar of observability directly. There's no short-cut. That's the whole point.
  - phillipcarter 4y ago
    > So when you're designing an observability system, the whole ball game is about capturing and constraining the cardinality of that metadata. If your world is only time series databases that struggle with cardinality, then sure. Fortunately, there's a lot more tools out there, some of which do just fine with high cardinality data. I don't really agree much with your entire comment. I think you're looking at observability through the lens of 2010s-era tools, and falling deeply into the trap of thinking that this is the only way to do things.
    - preseinger 4y ago
      My core claim is that telemetry data for a system is always more than the production data for that system. Yes? No? Assuming yes, everything else I'm claiming is noncontroversial.
      - phillipcarter 4y ago
        I count about a dozen claims, so I'm not really sure what to begin with. But I disagree with your comment here too.
        preseinger 4y ago
        (shrug) OK. I'm a domain expert here, but that doesn't mean my claims are infallible, of course. Your call. Good luck.
        LewisJEllis 4y ago
        most or all of the observability domain experts i'm familiar with either stopped talking about "pillars" several years ago, or have been actively speaking against the "3 pillars" framing for several years most of your takes here sound like they're from somewhere around 2016-2018
      - remram 4y ago
        I don't think anyone opposes that, but I don't think that's your "core claim". Your core claim is that this is a huge problem, one that makes OpenTelemetry "a technological dead-end", and that "the whole ball game is about capturing and constraining the cardinality of that metadata". That is what it would be nice to have you justify some more. After all, the same could be said of logs, or internal traffic when using microservices/DBMS (request cardinality/traffic will be multiplied).
hooverd 4y ago
Oh no. Now what should I use? I can't believe something that's actively in development is considered dead these days. Surely there's a better alternative with as good or better community support, right? Right...?
- preseinger 4y ago
  The problem isn't a lack of community, it's that the project is solving a problem that isn't relevant to anyone except Fortune-X00 companies.
  - phillipcarter 4y ago
    Yeah I'm gonna call bullshit on that one. I work daily with everyone from startups and nonprofits to big banks. They use OTel because it solves problems for them. They don't all use it the same way, but they use it.
  - hooverd 4y ago
    So, public sector here. We use Honeycomb with OTel and it's most definitely solving our problems. It's incredible being able to see traces up and down our whole stack and pinpoint where an issue is.
    - preseinger 4y ago
      What RPS do you serve, net? Everything works to trivial scale, e.g. O(10k) RPS and below. Do you do more than that per server?
      - LewisJEllis 4y ago
        You sound like you're trying to have it both ways: "OTel solves a problem that is irrelevant unless you're massive" but also "well of COURSE it can work if you're not massive, how massive are you?" There are tons of startups and other not-fortune-x00 orgs benefitting from what OTel provides. Your claim that OTel is irrelevant outside fortune-x00 cos is very clearly not true.

OpenTelemetry is moribund and a technological dead-end. Godspeed to any org that builds on it.

ren_engineer 4y ago

I've seen it hyped up like crazy, what's so bad about it and what is the superior alternative?

preseinger 4y ago
Instrumentation of a given system is by definition always "larger" than the nominal traffic which flows through that system. That is, given some request R, the possible observable metadata of that request R is essentially infinite. So when you're designing an observability system, the whole ball game is about capturing and constraining the cardinality of that metadata. Observability as a field is _entirely_ about reducing the volume of telemetry data such that it can be usefully captured, aggregated, and queried by operators. That's why we have metrics, traces, and logs as distinct things. They're all optimizations of the same underlying information, each optimized for different consumer use cases. Concretely: if user U1 does an HTTP GET to /spotify/track/123, that's perhaps 10KB of production traffic, but easily 10MB of telemetry traffic. In effect that telemetry can be modeled as a huge key/val map of metadata, but you can't do it that way and remain efficient, you have to optimize for observability use cases. You have to increment a cardinality-bound set of metric counters for the request outcomes, and maybe emit some best-effort trace data for the request ID, and etc. etc. — _as separate things_! The engineering costs dominate the design. But OpenTelemetry says this isn't the case. OpenTelemetry says that whatever requirements are on FX00 company CTO feature checklists are valid a priori, and commits to doing whatever is necessary to satisfy them. That's because OpenTelemetry is evaluated not on any technical merits, but on the adoption rate of the CNCF stack among those FX00 companies. OpenTelemetry is explicitly and exclusively a thing meant to tick off an "observability" checklist item on the checklist of a FX00 CTO's due diligence form. That's it. That's the only goal. Nobody with a choice should be using it. Read the source code, it's abysmal. Alternatives? Write code that leverages each pillar of observability directly. There's no short-cut. That's the whole point.
- phillipcarter 4y ago
  > So when you're designing an observability system, the whole ball game is about capturing and constraining the cardinality of that metadata. If your world is only time series databases that struggle with cardinality, then sure. Fortunately, there's a lot more tools out there, some of which do just fine with high cardinality data. I don't really agree much with your entire comment. I think you're looking at observability through the lens of 2010s-era tools, and falling deeply into the trap of thinking that this is the only way to do things.
  - preseinger 4y ago
    My core claim is that telemetry data for a system is always more than the production data for that system. Yes? No? Assuming yes, everything else I'm claiming is noncontroversial.
    - phillipcarter 4y ago
      I count about a dozen claims, so I'm not really sure what to begin with. But I disagree with your comment here too.
      - preseinger 4y ago
        (shrug) OK. I'm a domain expert here, but that doesn't mean my claims are infallible, of course. Your call. Good luck.
        LewisJEllis 4y ago
        most or all of the observability domain experts i'm familiar with either stopped talking about "pillars" several years ago, or have been actively speaking against the "3 pillars" framing for several years most of your takes here sound like they're from somewhere around 2016-2018
    - remram 4y ago
      I don't think anyone opposes that, but I don't think that's your "core claim". Your core claim is that this is a huge problem, one that makes OpenTelemetry "a technological dead-end", and that "the whole ball game is about capturing and constraining the cardinality of that metadata". That is what it would be nice to have you justify some more. After all, the same could be said of logs, or internal traffic when using microservices/DBMS (request cardinality/traffic will be multiplied).

hooverd 4y ago

Oh no. Now what should I use? I can't believe something that's actively in development is considered dead these days. Surely there's a better alternative with as good or better community support, right? Right...?

preseinger 4y ago
The problem isn't a lack of community, it's that the project is solving a problem that isn't relevant to anyone except Fortune-X00 companies.
- phillipcarter 4y ago
  Yeah I'm gonna call bullshit on that one. I work daily with everyone from startups and nonprofits to big banks. They use OTel because it solves problems for them. They don't all use it the same way, but they use it.
- hooverd 4y ago
  So, public sector here. We use Honeycomb with OTel and it's most definitely solving our problems. It's incredible being able to see traces up and down our whole stack and pinpoint where an issue is.
  - preseinger 4y ago
    What RPS do you serve, net? Everything works to trivial scale, e.g. O(10k) RPS and below. Do you do more than that per server?
    - LewisJEllis 4y ago
      You sound like you're trying to have it both ways: "OTel solves a problem that is irrelevant unless you're massive" but also "well of COURSE it can work if you're not massive, how massive are you?" There are tons of startups and other not-fortune-x00 orgs benefitting from what OTel provides. Your claim that OTel is irrelevant outside fortune-x00 cos is very clearly not true.