4 ms·

Can you even achieve this with prometheus? Afaik it operates by exposing metrics that are scraped at some interval. High level stuff, not per-trace stuff. How

Can you even achieve this with prometheus? Afaik it operates by exposing metrics that are scraped at some interval. High level stuff, not per-trace stuff. How would you build the "holy grail" map that shows a trace of every sub component in a transaction broken down by start/stop time etc... for instance show the load balancer see a request, the request get handled by middlewares etc, then go onto some kind of handler/controller, the sub-queries inside of that like database calls or cache calls. I don't think that is possible with prometheus?

baby_souffle 2y ago
> Can you even achieve this with prometheus? Afaik it operates by exposing metrics that are scraped at some interval. High level stuff, not per-trace stuff. Correct. Prometheus is just metrics. The main argument for oTel is that instead of one proprietary vendor SDK or importing prometheus and jaeger and whatever you want to use for logging, just import oTel and all that will be done with a common / open data format. I still believe in that dream but it's clear that the whole project needs some time/resources to mature a bit more. If anybody remembers the Terraform/ToFu drama, it's been really wild to see how much support everybody pledged for ToFu but all the traditional observability providers have just kinda tolerated oTel :/
niftaystory 2y ago
Code traces are metrics. Run times per function calls metrics, count of specific function call metrics. Otel is an attempt to package such arithmetic. Web apps have added so many layers of syntax sugar and semantic wank, we’ve lost sight its all just the same old math operations relative to different math objects. Sets are not triangles but both are tested, quantified, and compared with the same old mathematical ops we learn by middle school.
- mikestorrent 2y ago
  No, code traces are not just metrics; and while you can knit together something approximating traces from metrics, you'll quickly run into the reason why traces are a distinct thing. First, in a distributed system, you'll discover that you can't rely on clocks to get the timing of subsecond events correct. Second, you'll be contextless about code paths. So, you might independantly reinvent the idea of passing along a context - and now you're just making your own tracing system but without any of the benefit of building on years of existing discoveries in this field. OTel does feel a little bit heavy, unless you're already used to e.g. New Relic, Dynatrace, etc. where you have to run an agent process and instrumentize your code to some extent; it's never going to be free to audit every function call! This is why (a) you sample down and don't keep every trace, and (b) unless your company is extremely flush with cash you probably don't run tracing in every environment. If you can get away with it just in a staging or perf test env you can reap most of the benefit without the production impact and cost.
  - niftaystory 2y ago
    All those things you describe are computable metrics. They have to be or Otel itself would not be able to compute them for consumption. All you described are cherry picked semantic indirections to obfuscate it’s all just a computer computing metrics of its own memory states. Sorry for knowing how computers actually work (EE grad not a CS grad). I know that can frustrate CS grads who think their preferred OS and favorite programming language is how a computer works. You’re describing how contemporary SWEs view their day job. Edit: teleMETRY …what’s in a name? Oh right …meaning.
    - chupasaurus 2y ago
      As a no grad to EE grad: traces mean a bundle of metrics that varies in structure hence you can't store and process them as effective as a list of counters unless you have a distinct bin for each possible trace, combinatorial explosion y'know.
    - andrewflnr 2y ago
      You know the conversation is going well for you when you resort to citing the "meaning" of a name instead of, you know, base reality. Who needs the territory, I've got my map right here. Speaking of meaning, the best I can make of your point is that you're using a much broader definition of "metrics" than the rest of this conversation, and in particular broader than Prometheus (remember context? very important for "meaning"!) supports. That or you really just don't know what a "trace" is (in this context).
    - mathfailure 2y ago
      To be a smart-ass, one has to be smart first. Quit this.
- chrisweekly 2y ago
  huh? I've always heard and read and experienced that "logs, traces, metrics" are the 3 legs of the observability stool.
  - niftaystory 2y ago
    Open teleMETRY Any guesses as to etymology?
    - ffsm8 2y ago
      By this logic, you can say that logging, metrics and tracing are all fundamentally just different kinds of data and we should be calling it just plain databases and CRUD. They're related, but people have a very specific idea and concept of what each is, you haven't actually provided a good argument why we should throw out these distinctions just because they somewhat resemble each other if you ignore a few details
- atombender 2y ago
  OpenTelemetry's traces are trees of spans. You cannot represent this efficiently without a combinatorial explosion of labels. You may be thinking of metrics in the sense of counters and gauges, but that's not the data model that OpenTelemetry (and before they, Zipkin, Jaeger, and OpenCensus) uses for traces. The data model for tracing is to emit events that provide a span ID and an optional parent span ID. The event collector can piece these together into a tree after the fact, which will work as long as the parent structure is maintained. Prometheus is absolutely not suitable for this. Quibbling about the word "telemetry" doesn't really help here. OpenTelemetry supports three different, completely different subsets of functionality: Metrics (counters, gauges, histograms), traces (span events in a tree structure), and logging (structured log events). They each have completely different client interfaces.
hinkley 2y ago
Yeah part of the problem is it’s called Opentelemetry and half of you are only talking about tracing, not metrics. Telemetry is metrics. It’s been metrics since at least the Mercury Program. Metrics in OTEL is about three years old and it’s garbage for something that’s been in development for three years.
tonyhart7 2y ago
its looks hassle to implement ngl

baby_souffle 2y ago

> Can you even achieve this with prometheus? Afaik it operates by exposing metrics that are scraped at some interval. High level stuff, not per-trace stuff. Correct. Prometheus is just metrics. The main argument for oTel is that instead of one proprietary vendor SDK or importing prometheus and jaeger and whatever you want to use for logging, just import oTel and all that will be done with a common / open data format. I still believe in that dream but it's clear that the whole project needs some time/resources to mature a bit more. If anybody remembers the Terraform/ToFu drama, it's been really wild to see how much support everybody pledged for ToFu but all the traditional observability providers have just kinda tolerated oTel :/

niftaystory 2y ago

Code traces are metrics. Run times per function calls metrics, count of specific function call metrics. Otel is an attempt to package such arithmetic. Web apps have added so many layers of syntax sugar and semantic wank, we’ve lost sight its all just the same old math operations relative to different math objects. Sets are not triangles but both are tested, quantified, and compared with the same old mathematical ops we learn by middle school.

mikestorrent 2y ago
No, code traces are not just metrics; and while you can knit together something approximating traces from metrics, you'll quickly run into the reason why traces are a distinct thing. First, in a distributed system, you'll discover that you can't rely on clocks to get the timing of subsecond events correct. Second, you'll be contextless about code paths. So, you might independantly reinvent the idea of passing along a context - and now you're just making your own tracing system but without any of the benefit of building on years of existing discoveries in this field. OTel does feel a little bit heavy, unless you're already used to e.g. New Relic, Dynatrace, etc. where you have to run an agent process and instrumentize your code to some extent; it's never going to be free to audit every function call! This is why (a) you sample down and don't keep every trace, and (b) unless your company is extremely flush with cash you probably don't run tracing in every environment. If you can get away with it just in a staging or perf test env you can reap most of the benefit without the production impact and cost.
- niftaystory 2y ago
  All those things you describe are computable metrics. They have to be or Otel itself would not be able to compute them for consumption. All you described are cherry picked semantic indirections to obfuscate it’s all just a computer computing metrics of its own memory states. Sorry for knowing how computers actually work (EE grad not a CS grad). I know that can frustrate CS grads who think their preferred OS and favorite programming language is how a computer works. You’re describing how contemporary SWEs view their day job. Edit: teleMETRY …what’s in a name? Oh right …meaning.
  - chupasaurus 2y ago
    As a no grad to EE grad: traces mean a bundle of metrics that varies in structure hence you can't store and process them as effective as a list of counters unless you have a distinct bin for each possible trace, combinatorial explosion y'know.
  - andrewflnr 2y ago
    You know the conversation is going well for you when you resort to citing the "meaning" of a name instead of, you know, base reality. Who needs the territory, I've got my map right here. Speaking of meaning, the best I can make of your point is that you're using a much broader definition of "metrics" than the rest of this conversation, and in particular broader than Prometheus (remember context? very important for "meaning"!) supports. That or you really just don't know what a "trace" is (in this context).
  - mathfailure 2y ago
    To be a smart-ass, one has to be smart first. Quit this.
chrisweekly 2y ago
huh? I've always heard and read and experienced that "logs, traces, metrics" are the 3 legs of the observability stool.
- niftaystory 2y ago
  Open teleMETRY Any guesses as to etymology?
  - ffsm8 2y ago
    By this logic, you can say that logging, metrics and tracing are all fundamentally just different kinds of data and we should be calling it just plain databases and CRUD. They're related, but people have a very specific idea and concept of what each is, you haven't actually provided a good argument why we should throw out these distinctions just because they somewhat resemble each other if you ignore a few details
atombender 2y ago
OpenTelemetry's traces are trees of spans. You cannot represent this efficiently without a combinatorial explosion of labels. You may be thinking of metrics in the sense of counters and gauges, but that's not the data model that OpenTelemetry (and before they, Zipkin, Jaeger, and OpenCensus) uses for traces. The data model for tracing is to emit events that provide a span ID and an optional parent span ID. The event collector can piece these together into a tree after the fact, which will work as long as the parent structure is maintained. Prometheus is absolutely not suitable for this. Quibbling about the word "telemetry" doesn't really help here. OpenTelemetry supports three different, completely different subsets of functionality: Metrics (counters, gauges, histograms), traces (span events in a tree structure), and logging (structured log events). They each have completely different client interfaces.

hinkley 2y ago

Yeah part of the problem is it’s called Opentelemetry and half of you are only talking about tracing, not metrics. Telemetry is metrics. It’s been metrics since at least the Mercury Program. Metrics in OTEL is about three years old and it’s garbage for something that’s been in development for three years.

tonyhart7 2y ago

its looks hassle to implement ngl