3 ms·
Instrumentation isn't solved by any single specific thing. It's a praxis that you apply to your code as your write it, like I guess error handling; it's not a p
by kiitos 3y ago
Instrumentation isn't solved by any single specific thing. It's a praxis that you apply to your code as your write it, like I guess error handling; it's not a product that you can deploy, like I guess Splunk or New Relic or whatever else.
You should "use" metrics, logs, and traces thru dependencies that are specific to your organization. The interface between your business logic and operational telemetry should be abstract, essentially the same as a database or a remote HTTP endpoint or etc. The concrete system(s) collecting and serving telemetry data are the responsibility of your dev/ops or whatever team.
Main point: instrumentation is part of the development process, not something that's automatic or that can be bolted-on.
- pgwhalen 3y agoHave you worked with OTEL before? Basically all of your points about instrumentation are actually quite sympathetic to OTEL's view of the world. The whole point of OTEL is to provide some standards around how these pieces fit together - not to solve for them automatically.
- kiitos 3y agoI've been deeply involved with OTel from even before it was a CNCF jam. My experiences with the project, over time, have made me basically abandon the project as unsound and infeasible since a year or two. Those experiences also inform comments like the ones I've made here.
- pgwhalen 3y agoCan you elaborate on what unsound and infeasible mean? I'm newer to OTel than you (~6 months of working with it in depth), and don't really understand what you're getting at. It's solving real problems in my organization, with only a "regular" amount of pain for a component of its size.
- klysm 3y agoOkay so what’s the interface? Sounds like what OTEL provides to me
- kiitos 3y agoThere are well-defined interfaces for specific sub-classes of telemetry data. Prometheus provides a set of interfaces for metrics which are pretty battle-tested by now. There are similar interfaces for logs and traces, authored by various different parties, and with various different capabilities, trade-offs, etc. There is no one true interface! The interface is a function of the sub-class of telemetry data it serves, the specific properties of the service(s) it supports, the teams it's used by, the organization that maintains it, etc. etc. OTel tries to assert a general-purpose interface. But this is exactly the issue with the project. That interface doesn't exist.
- klysm 3y agoOTEL is a set of interfaces, so I’m not sure your last point applies. I do agree that battle tested things like Prometheus work great, but why not have a set of standardized interfaces? There is clearly a cost to having them; for some projects this may be too much. For the projects I’ve used it in it let me spin up all the traces and telemetry without thinking hard.