3 ms·

I just don’t get this sentiment. How would you represent metrics as traces? You cannot. Even reconstructing traces from logs would be challenging at best. How w

• anygivnthursday 1mo ago
  I am not sure if this is what they mean, but e.g. with Micrometer in Java you can instrument your code once with observations that produces observation events, then you can register handlers that can turn them into metrics, or logs, or traces without having to instrument your code three times. https://docs.micrometer.io/micrometer/reference/observation.html https://docs.micrometer.io/micrometer/reference/observation....
  • Flamkuchlo 1mo ago
    The problem is not the instrumentation but the way everyone of them work. A metric is a point in time. A metric is very small but you have a lot of them. A log is when something is happening but you need to log it out. A logline is heavy and has a lot of context. User id, message, etc. A trace needs to start at the request level and tracing until the response. This is the slowest and heaviest operation. How do you decide when to suddenly do the trace and send it? IF you always do the trace, you have to pay for the overhead of that tracing constantly.
    • spockz 1mo ago
      Technically, you can use the same places in the code where you stop/start/fork traces to also be the places where you increment the counters/gauges, etc. Which I think the GP was alluding to when describing the micrometer solution. Similarly, you can derive metrics for log lines without having to emit the actual log lines. Then separately you can have log levels or verbosity levels that control to which level you actually emit traces/logs and/or roll up metrics.
    • TylerE 1mo ago
      At that point you almost might as well just log everything. The decision logic is likely about as complex as just doing it. Then I suppose you have a watchdog task that fires off every, say, 15 minutes or an hour or something, looks at the collected data, and either decides to keep it or trash it while recording a tiny "nothing interesting" datapoint.
      • Flamkuchlo 1mo ago
        Loghandling is quite resource intensive. All the log ingestion systems i have seen were bigger elastic search clusters.
    • twic 1mo ago
      Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then reconstruct a trace from those log entries. If you want to associate multiple spans, and separate log entries, within a trace, you use a shared ID, which is just the same as a context entry for logging. All three of these pillars are just ways of looking at events. They are not fundamentally different at all. This is a mistaken idea in "Observability 1.0" whose correction is the basis of "Observability 2.0". The pillars still have their uses, but the choice between them is really a non-functional one - storing a log entry for every event might be too expensive, so just store metrics instead, and index every log entry so it can be correlated with nearby ones might be too expensive, so just store specific traces instead.
      • Flamkuchlo 1mo ago
        A metric is not event based. You don't have a metric 'person logged in' because you would need to scrape the metric at the moment a person logged in. You have a metric called 'overall people have logged in so far' and you do math on it. The 'person logged in' is an event you log out.
      • PunchyHamster 1mo ago
        > Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. No, metric is just value. Some are derived from events (like histogram/rate of given event duration) but others are wholly independent (like returning app's CPU/memory usage)
        • krab 1mo ago
          The app's memory usage is an aggregation of the alloc/free events. I think the original point was that all of the metrics, traces and logs are conceptually the same but for efficiency, we store less data in each place, not the full history. Personally, for the systems I work on, having an easy way to turn logs into metrics and vice versa, without deciding up front, would be a slight benefit.
    • jaen 1mo ago
      What? All of this has been solved for a long time. How do you think hyperscalers do this? Search keyword: "Adaptive sampling"
      • Flamkuchlo 1mo ago
        Adaptive sampling is not tracing, its sampling. Tracing traces a particular event. I'm quite aware of the difference between sampling, tracing and profiling.
        • jaen 27d ago
          No... Adaptive sampling is a family of statistical methods to choose an appropriate decimation strategy for arbitrary events based on real-world occurrence distributions. It can be applied to tracing, metrics, logging ("sampling") and profiling.
• sweetgiorni 1mo ago
  > How would you represent metrics as traces? Just instrument your meter implementation so each observation produces a span. Boom, free metric-derived traces.
  • fallingbananna 1mo ago
    Yup. Not a difficult problem to solve. In the code define everything as a span with a name, scope (start-end), description and tags... and then you can easily dynamically produce traces, spans, logs or metrics based on what you need.
    • TylerE 1mo ago
      At some point your monitoring is burning 10x as much CPU as the actual task...
  • jandrewrogers 1mo ago
    "free". The observability system would greatly exceed the workload being observed in many cases.