7 ms·

Tentative TLDR: It seems like DD pressured a contributor into not shipping a feature that would have made DD just another telemetry vendor. It seems like right

Tentative TLDR: It seems like DD pressured a contributor into not shipping a feature that would have made DD just another telemetry vendor. It seems like right now, data can flow IN datadog libraries/agents but not out. This PR would sort of allow data to flow OUT of datadog's libs/agents? And DD doesn't want that because it removes their lock-in power? Is this correct? This would be extremely crappy of Datadog.

deleted 4y ago
[deleted]
judge2020 4y ago
> This PR would sort of allow data to flow OUT of datadog's libs/agents? Yes. This allows you to expose a Telemetry collector on the datadog agent port 8126[0], allowing you to collect Application Performance Monitoring (APM) traces from any APM-enabled datadog library[1]. If I had to guess, DataDog's argument is that they don't want you using the engineering hours they invest into their libraries to have DD do the heavy lifting of collecting APM traces and send the messages off to another service. <removed OSS comment> 0: https://github.com/boostchicken/opentelemetry-collector-contrib/blob/datadog-receiver/receiver/datadogreceiver/README.md https://github.com/boostchicken/opentelemetry-collector-cont... 1: https://docs.datadoghq.com/tracing/#send-traces-to-datadog https://docs.datadoghq.com/tracing/#send-traces-to-datadog
- cmeacham98 4y ago
  > DataDog's libraries don't seem to be OSS: https://github.com/DataDog/datadogpy/blob/master/LICENSE https://github.com/DataDog/datadogpy/blob/master/LICENSE Is that not a verbatim the 3-clause BSD license (which is an FSF/OSI approved OSS license)?
- deleted 4y ago
  [deleted]
- aeyes 4y ago
  The library you linked is just for sending metric data (gauges, counters etc), the relevant library for APM is called ddtrace and its dual licensed with Apache2 and BSD3. https://github.com/DataDog/dd-trace-py/blob/1.x/LICENSEe https://github.com/DataDog/dd-trace-py/blob/1.x/LICENSEe
- blibble 4y ago
  that LICENSE of theirs is literally 3 clause BSD?
- hosh 4y ago
  When we switched off of AppSignal, since our instrumentation as deeply intertwined with our code, it took a while to change that over to Otel. But at least now, being Otel, theoretically if we want to change it to a different vendor, we can. If we were to change off of DD, and there is an otel collector that can accept DD APM spans, then we can switch now, refactor later. In practice, DD has a lot going for it that I don't see in New Relic. There are also some key features in DD that is not in OTEl -- for example, we can't use DD's APM ingestion controls for controlling sampling rates for OTEL spans, and DD has no incentive to add such a feature. I'm actually working on adding in Otel sampling into our project right now. (In our case, we have to use Otel because DD does not have SDKs for Elixir)
  - technimad 4y ago
    Why sample otel spans and miss out on the important ones?
    - harpratap 4y ago
      Because of cost?
    - hosh 4y ago
      While I would like to ingest every one, cost is a factor. Even if we were self-hosting, there's a cost to ingesting and storing every single span. And even if we are able to pay for ingesting 100%, not everything is practical to be ingested 100%. Our most common request type (heartbeat) generate a span payload size that is a multiple of the original request. We're using Elixir in production, and those can absorb a tremendous amount of traffic, saturating the entire CPU capacity of the hardware if we let it. The agents are not capable of keeping up.
    - phillipcarter 4y ago
      You can sorta have your cake and eat to too. Firstly, not all spans are interesting. When 99.99% of your traffic is just going to serve up an HTTP 200 within your acceptable latency threshold, you don't need every one of those. You probably do want to keep 100% of error spans, or those where the root has a duration beyond a configured threshold. There's tools to be able to sample that way. Secondly, there's ways to also attach your effective sample rate as metadata to spans, and if there's a backend that supports re-weighting counts based on that, you can still get accurate all-up counts of overall traffic. Admittedly, OTel and many other backends don't have the best story for this yet. But it's getting better.
jiggawatts 4y ago
None of the APM providers like New Relic, Data Dog, or Azure App Insights want a truly open ecosystem. The only reason they support "Open" Telemetry is because they're worried about lock-in at the data sources. For example, App Insights supports rich/structured telemetry via its proprietary SDK and various APIs. No open-source developer in their right mind would ever hard code such a proprietary dependency into something published under a truly open license. Now that rich telemetry instead of simple text logging is starting to become an increasingly popular approach, the proprietary APM vendors got nervous that they would get "locked out" of the entire open source ecosystem, to be replaced by a data source that is open and not compatible with their proprietary sinks. Hence Open Telemetry. It was always about making the source open, not the sink.
- api 4y ago
  Open source these days is very often about making the thing you use to consume SaaS open so it can spread as widely as possible, but it’s never made easy to commoditize the SaaS part.
- phillipcarter 4y ago
  I won't speculate on the top-level strategies of companies competing with another, so take this as a grain of salt. From my perspective (maintainer, employed by a vendor), all of us who work for these different companies collaborate very well together. We all recognize that it's both technically tractable and fundamentally user-friendly to make instrumentation be a common standard that anyone can use to point at any of the OSS and commercial tools in this space. There's plenty to differentiate on with telemetry backends, querying experiences, API capabilities, data analysis tools and UX, etc. We have a long way to go to see this vision fully realized, but it's quite far along and I have no doubt we'll arrive at the right outcome here.
  - jiggawatts 4y ago
    Everything you've just said is compatible with what I've said. The vendors are concerned about being left out at the "source", and are seeking to differentiate with their proprietary "sink", usually closed-source SaaS solutions. I'm not even arguing that this is bad, it's just how markets work, and it's currently beneficial to developers in general, including both open-source developers and the type working in a cubicle farm somewhere.
deleted 4y ago
[deleted]
jensensbutton 4y ago
As far as I can tell literally no vendor allows data to flow out? What am i missing?

deleted 4y ago

[deleted]

judge2020 4y ago

> This PR would sort of allow data to flow OUT of datadog's libs/agents? Yes. This allows you to expose a Telemetry collector on the datadog agent port 8126[0], allowing you to collect Application Performance Monitoring (APM) traces from any APM-enabled datadog library[1]. If I had to guess, DataDog's argument is that they don't want you using the engineering hours they invest into their libraries to have DD do the heavy lifting of collecting APM traces and send the messages off to another service. <removed OSS comment> 0: https://github.com/boostchicken/opentelemetry-collector-contrib/blob/datadog-receiver/receiver/datadogreceiver/README.md https://github.com/boostchicken/opentelemetry-collector-cont... 1: https://docs.datadoghq.com/tracing/#send-traces-to-datadog https://docs.datadoghq.com/tracing/#send-traces-to-datadog

cmeacham98 4y ago
> DataDog's libraries don't seem to be OSS: https://github.com/DataDog/datadogpy/blob/master/LICENSE https://github.com/DataDog/datadogpy/blob/master/LICENSE Is that not a verbatim the 3-clause BSD license (which is an FSF/OSI approved OSS license)?
deleted 4y ago
[deleted]
aeyes 4y ago
The library you linked is just for sending metric data (gauges, counters etc), the relevant library for APM is called ddtrace and its dual licensed with Apache2 and BSD3. https://github.com/DataDog/dd-trace-py/blob/1.x/LICENSEe https://github.com/DataDog/dd-trace-py/blob/1.x/LICENSEe
blibble 4y ago
that LICENSE of theirs is literally 3 clause BSD?
hosh 4y ago
When we switched off of AppSignal, since our instrumentation as deeply intertwined with our code, it took a while to change that over to Otel. But at least now, being Otel, theoretically if we want to change it to a different vendor, we can. If we were to change off of DD, and there is an otel collector that can accept DD APM spans, then we can switch now, refactor later. In practice, DD has a lot going for it that I don't see in New Relic. There are also some key features in DD that is not in OTEl -- for example, we can't use DD's APM ingestion controls for controlling sampling rates for OTEL spans, and DD has no incentive to add such a feature. I'm actually working on adding in Otel sampling into our project right now. (In our case, we have to use Otel because DD does not have SDKs for Elixir)
- technimad 4y ago
  Why sample otel spans and miss out on the important ones?
  - harpratap 4y ago
    Because of cost?
  - hosh 4y ago
    While I would like to ingest every one, cost is a factor. Even if we were self-hosting, there's a cost to ingesting and storing every single span. And even if we are able to pay for ingesting 100%, not everything is practical to be ingested 100%. Our most common request type (heartbeat) generate a span payload size that is a multiple of the original request. We're using Elixir in production, and those can absorb a tremendous amount of traffic, saturating the entire CPU capacity of the hardware if we let it. The agents are not capable of keeping up.
  - phillipcarter 4y ago
    You can sorta have your cake and eat to too. Firstly, not all spans are interesting. When 99.99% of your traffic is just going to serve up an HTTP 200 within your acceptable latency threshold, you don't need every one of those. You probably do want to keep 100% of error spans, or those where the root has a duration beyond a configured threshold. There's tools to be able to sample that way. Secondly, there's ways to also attach your effective sample rate as metadata to spans, and if there's a backend that supports re-weighting counts based on that, you can still get accurate all-up counts of overall traffic. Admittedly, OTel and many other backends don't have the best story for this yet. But it's getting better.

jiggawatts 4y ago

None of the APM providers like New Relic, Data Dog, or Azure App Insights want a truly open ecosystem. The only reason they support "Open" Telemetry is because they're worried about lock-in at the data sources. For example, App Insights supports rich/structured telemetry via its proprietary SDK and various APIs. No open-source developer in their right mind would ever hard code such a proprietary dependency into something published under a truly open license. Now that rich telemetry instead of simple text logging is starting to become an increasingly popular approach, the proprietary APM vendors got nervous that they would get "locked out" of the entire open source ecosystem, to be replaced by a data source that is open and not compatible with their proprietary sinks. Hence Open Telemetry. It was always about making the source open, not the sink.

api 4y ago
Open source these days is very often about making the thing you use to consume SaaS open so it can spread as widely as possible, but it’s never made easy to commoditize the SaaS part.
phillipcarter 4y ago
I won't speculate on the top-level strategies of companies competing with another, so take this as a grain of salt. From my perspective (maintainer, employed by a vendor), all of us who work for these different companies collaborate very well together. We all recognize that it's both technically tractable and fundamentally user-friendly to make instrumentation be a common standard that anyone can use to point at any of the OSS and commercial tools in this space. There's plenty to differentiate on with telemetry backends, querying experiences, API capabilities, data analysis tools and UX, etc. We have a long way to go to see this vision fully realized, but it's quite far along and I have no doubt we'll arrive at the right outcome here.
- jiggawatts 4y ago
  Everything you've just said is compatible with what I've said. The vendors are concerned about being left out at the "source", and are seeking to differentiate with their proprietary "sink", usually closed-source SaaS solutions. I'm not even arguing that this is bad, it's just how markets work, and it's currently beneficial to developers in general, including both open-source developers and the type working in a cubicle farm somewhere.

deleted 4y ago

[deleted]

jensensbutton 4y ago

As far as I can tell literally no vendor allows data to flow out? What am i missing?