4 ms·

> Alternatives? Write code that leverages each pillar of observability directly. There's no short-cut. That's the whole point. I think I am not able to follow

> Alternatives? Write code that leverages each pillar of observability directly. There's no short-cut. That's the whole point. I think I am not able to follow you correctly. Is your entire point that auto-instrumentation is too much and one should default to manual instrumentation instead?

preseinger 4y ago
You can't delegate "instrumentation" as a monolithic concern to a single authority or package or library like OpenTelemetry and expect to get anything but noise on the other side. And yeah, basically any vendor claiming to do do "auto-instrumentation" is selling snake-oil. Expensive snake-oil, too! Observability is not something that a vendor can provide without meaningful and deep integration in your infrastructure. You have to do some amount of work, and IMO few vendors deliver value beyond what a single engineer can produce with with a basic internal Prometheus infrastructure + short-term log aggregation.
- harpratap 4y ago
  But OpenTelemetry never claimed to be an auto-instrumentation only library, you can very well just manually instrument your application with the metrics you want and export it your self managed prometheus + log infrastructure. In fact, OTel makes it easier for you if in the future you might want to move to a better self managed TSDB + Log Infra because you know it most likely have an exporter ready with zero effort to re-instrument your metrics or logs
  - preseinger 4y ago
    You can definitely define an abstract concept of telemetry which generalizes over use cases, and which can be expressed as a single schema that can be collected generically. The problem is that this definition is too general to be practically useful. The whole ball game for observability systems is optimization for specific consumption use cases. That _must_ occur at the point of origin, it _cannot_ be deferred. OTel says that it's possible to define a general-purpose exporter for arbitrary telemetry data, and that specialization and optimization of that generalized data can be done later, downstream. This is simply not true.
    - harpratap 4y ago
      Could you point me to docs, architecture notes or code where they explicitly say auto-instrumentation is the ONLY approach to using OTel? AFAIK both auto-instrumentation and manual instrumentation are equally supported - https://opentelemetry.io/docs/concepts/instrumenting/ https://opentelemetry.io/docs/concepts/instrumenting/

preseinger 4y ago

You can't delegate "instrumentation" as a monolithic concern to a single authority or package or library like OpenTelemetry and expect to get anything but noise on the other side. And yeah, basically any vendor claiming to do do "auto-instrumentation" is selling snake-oil. Expensive snake-oil, too! Observability is not something that a vendor can provide without meaningful and deep integration in your infrastructure. You have to do some amount of work, and IMO few vendors deliver value beyond what a single engineer can produce with with a basic internal Prometheus infrastructure + short-term log aggregation.

harpratap 4y ago
But OpenTelemetry never claimed to be an auto-instrumentation only library, you can very well just manually instrument your application with the metrics you want and export it your self managed prometheus + log infrastructure. In fact, OTel makes it easier for you if in the future you might want to move to a better self managed TSDB + Log Infra because you know it most likely have an exporter ready with zero effort to re-instrument your metrics or logs
- preseinger 4y ago
  You can definitely define an abstract concept of telemetry which generalizes over use cases, and which can be expressed as a single schema that can be collected generically. The problem is that this definition is too general to be practically useful. The whole ball game for observability systems is optimization for specific consumption use cases. That _must_ occur at the point of origin, it _cannot_ be deferred. OTel says that it's possible to define a general-purpose exporter for arbitrary telemetry data, and that specialization and optimization of that generalized data can be done later, downstream. This is simply not true.
  - harpratap 4y ago
    Could you point me to docs, architecture notes or code where they explicitly say auto-instrumentation is the ONLY approach to using OTel? AFAIK both auto-instrumentation and manual instrumentation are equally supported - https://opentelemetry.io/docs/concepts/instrumenting/ https://opentelemetry.io/docs/concepts/instrumenting/