18 ms·

DataDog asked OpenTelemetry contributor to kill pull request

349 points by raybb 4y ago

Mizza 4y ago
Datadog is stupidly expensive for what it provides. Don't recommend and I hope something open kills it soon.
- deleted 4y ago
  [deleted]
- icelancer 4y ago
  Likewise for New Relic. Unfortunately OTel and everything similar doesn't remotely come close to the usability and ease of configuration that New Relic does. Not yet, anyway...
  - judge2020 4y ago
    Ideally you just implement OTel into your apps and later plug that into any metric/log/trace collector you want, so you have DataDog and Newrelic and other frontends competing based on their feature suite and price. As it stands, this was/is trying to use DataDog's existing APM libraries to turn them into OTel for ingest into other providers.
- kevincox 4y ago
  I'm not so sure to be honest. It is very expensive. But being able to quickly get all of the features is fantastic for a small company. As you start to use more volume (and your bill starts growing) then it makes sense to start considering self-hosting. But for many teams with relatively small data volumes self-hosting is likely not worth the engineering time.
  - zug_zug 4y ago
    Out of curiosity, is there anything self-hosted that compares to data-dog RUM?
- pranay01 4y ago
  you may want to have a look at SigNoz - https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz PS: I am one of the maintainers
- bdcravens 4y ago
  Plus they have (or had, just going off of my personal experience) very unethical sales practices.
- boostchicken 4y ago
  stabs repeatedly
domlaut 4y ago
Related, from Feb 2022, on what this is and why it's important: https://twitter.com/mipsytipsy/status/1494861690059759616 https://twitter.com/mipsytipsy/status/1494861690059759616
dpedu 4y ago
Nice tool. A couple years ago I wrote a similar tool that translates Datadog's agent's metrics/checks into AWS Cloudwatch equivalents.
raybb 4y ago
Link to exact comment: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1404724926 https://github.com/open-telemetry/opentelemetry-collector-co...
- icepat 4y ago
  Not sure I entirely follow what's going on here. Is there some context behind this that's useful to know?
  - tonyhb 4y ago
    If I understand correctly, this is a receiver — which means you can take your existing DD instrumentation in their format and have it translated to OTel using this code. It lowers the switching cost to get off of DD.
    - trynewideas 4y ago
      And Datadog competitor Grafana sponsored it: https://twitter.com/boostchicken/status/1618692475845238784 https://twitter.com/boostchicken/status/1618692475845238784
      - deleted 4y ago
        [deleted]
    - xen2xen1 4y ago
      Protect the moat at all costs!
      - benatkin 4y ago
        It's worse than that, they want security through obscurity too. They feel like someone is inappropriately tinkering with the agent they want customers to install. It's open source: https://github.com/DataDog/datadog-agent https://github.com/DataDog/datadog-agent ...but the downloads are behind a login: https://github.com/DataDog/datadog-agent#datadog-agent https://github.com/DataDog/datadog-agent#datadog-agent
AYBABTME 4y ago
Tentative TLDR: It seems like DD pressured a contributor into not shipping a feature that would have made DD just another telemetry vendor. It seems like right now, data can flow IN datadog libraries/agents but not out. This PR would sort of allow data to flow OUT of datadog's libs/agents? And DD doesn't want that because it removes their lock-in power? Is this correct? This would be extremely crappy of Datadog.
- deleted 4y ago
  [deleted]
- judge2020 4y ago
  > This PR would sort of allow data to flow OUT of datadog's libs/agents? Yes. This allows you to expose a Telemetry collector on the datadog agent port 8126[0], allowing you to collect Application Performance Monitoring (APM) traces from any APM-enabled datadog library[1]. If I had to guess, DataDog's argument is that they don't want you using the engineering hours they invest into their libraries to have DD do the heavy lifting of collecting APM traces and send the messages off to another service. <removed OSS comment> 0: https://github.com/boostchicken/opentelemetry-collector-contrib/blob/datadog-receiver/receiver/datadogreceiver/README.md https://github.com/boostchicken/opentelemetry-collector-cont... 1: https://docs.datadoghq.com/tracing/#send-traces-to-datadog https://docs.datadoghq.com/tracing/#send-traces-to-datadog
  - cmeacham98 4y ago
    > DataDog's libraries don't seem to be OSS: https://github.com/DataDog/datadogpy/blob/master/LICENSE https://github.com/DataDog/datadogpy/blob/master/LICENSE Is that not a verbatim the 3-clause BSD license (which is an FSF/OSI approved OSS license)?
  - deleted 4y ago
    [deleted]
  - aeyes 4y ago
    The library you linked is just for sending metric data (gauges, counters etc), the relevant library for APM is called ddtrace and its dual licensed with Apache2 and BSD3. https://github.com/DataDog/dd-trace-py/blob/1.x/LICENSEe https://github.com/DataDog/dd-trace-py/blob/1.x/LICENSEe
hijinks 4y ago
datadogs APM traces are the one thing that locks companies into Datadog. If this is merged it makes it a lot easier to move out of datadog into something else.
- aeyes 4y ago
  The only possible lock-in I see is if you have annotated code with Datadog specific tracing instructions and I'm pretty sure that most of these features aren't relevant for Open Telemetry. Switching APM providers isn't hard at all. Maybe the Open Telemetry ecosystem lacks a good agent but my understanding is that their APM agent is pretty much already a copy of the Datadog agent. I think New Relic would be a more interesting agent to use because their approach to APM isn't primarily to do sampled traces.
deleted 4y ago
[deleted]
funerr 4y ago
Here is where it started to go downhill: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1032906655 https://github.com/open-telemetry/opentelemetry-collector-co...
- TheRealPomax 4y ago
  And then went back uphill in https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1404724926 https://github.com/open-telemetry/opentelemetry-collector-co...
floatinglotus 4y ago
More vendor lock-in shenanigans. Data dog has always been a proprietary POS. I don’t know why people use it, APM traces? How long before Grafana has these capabilities in OSS? So annoying seeing a company like DD who cannot innovate at all, trying to lock in the average company.
- predakanga 4y ago
  > How long before Grafana has these capabilities in OSS? They've already started down that path with Grafana Tempo[0]. It's functional, but their UX and discoverability need a lot of work. [0]: https://grafana.com/oss/tempo/ https://grafana.com/oss/tempo/
- dalyons 4y ago
  Do you actually use it at scale in a large complicated system? IMHO it’s the opposite - datadog is consistently the most innovative and fastest iterating observability platform out there. Nothing comes close. It’s expensive yes, and there’s some lock-in yes, but they are GOOD.
- Mesverrum 4y ago
  As someone who has been in this space a long time I think you are pretty off base here. The commercial vendors have absolutely been the ones who did all the innovations in tracing. The OSS equivalents were really poor imitations for the longest time. Zipkin/Jaeger/Opentrace have been coming along slowly for many years, but it's really been maybe 2 years since that work has really become competitive with what was available in the commercial APM space. For the most part it's felt like that only happened because the commercial tools have all halted new work on their proprietary tools and told their staff to push Otel over the finish line. None of them actually want to pay teams of engineers to maintain proprietary libraries for every language, the execs are absolutely drooling to find a way to pawn as much of that cost off onto others via Otel. Locking people in on agents is naive when the truly heavy lift in migrating platforms is the swapping dashboards and alerts and retraining all your staff. You see the same thing around integrations, everyone used to have to roll their own proprietary chunks of code that in the end were all querying mostly the same data points back from servers and API's. Now everyone just prefers to wait for the Prometheus exporter and they adopt that instead.
bdcravens 4y ago
I'm always willing to dogpile on DataDog (no pun intended), given bad experiences with their sales, but all I'm reading out of this is that the DD person didn't want to review it out of a potential conflict of interest. It was reviewed however by someone else. Am I missing something here?
- nabakin 4y ago
  You're missing that this was a year-old PR ready to merge but it was stopped from merging by DataDog because it makes it easier for users to migrate from DataDog to another logging service
  - bdcravens 4y ago
    No, I am unsurprised by the DataDog taking that position; it fits with what I know of the company. However as someone pointed out, the comment in question wasn't originally linked, but I see it now.
  - bushbaba 4y ago
    Where did DD stop the merge. I just saw them simply state that the reviewer didn't want to review it as they work at Datadog. It could simply be that they are biased TO merge it even if it contains flaws. Having an independent reviewer ensures the code quality meets the bar.
    - jru 4y ago
      The OP didn't link directly to the comment and it's buried in the thread: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1404724926 https://github.com/open-telemetry/opentelemetry-collector-co...
- s17n 4y ago
  https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1404724926 https://github.com/open-telemetry/opentelemetry-collector-co...
joshxyz 4y ago
I love this bit: > Hello, I'm using this receiver in production for about one year, and left some comments that may be helpful. >> Are you serious? That is intense. Did it scale? Any memory issues? If I remember correctly I tuned all that away. Always astonishing to see broken stuff doing well in prod lol.
- quickthrower2 4y ago
  “Not production ready” stopped noone
capableweb 4y ago
From @mx-psi > Speaking as a Collector contrib maintainer, I just wanted to say that I am not going to continue reviewing this PR or start reviewing other future PRs related to the Datadog APM receiver to avoid any semblance of conflict of interest given my role both as a maintainer on this repository and as a member of Datadog's OpenTelemetry team. And from their GitHub profile: > Open Source Software Engineer at Datadog , focusing on OpenTelemetry What's the conflict of interest exactly? You work at Datadog, supposedly to work on OSS, with a focus on OpenTelemetry and you don't want to review Datadog related code for OpenTelemetry? Sounds weird, that kind of profile is exactly the type of person who should be reviewing the code, they have knowledge of both sides of it. Rather, it sounds like Datadog is walking back and don't want to support OpenTelemetry if it means it'll support their own tooling, instead of just others.
- judge2020 4y ago
  The potential conflict is that people might think/perceive DD to still be out to harm this project given their previous request for the author to stop development. Anything that might be regular code review or discussion for "why did you choose this approach" could be seen as sabotage on this super public issue.
stvnbn 4y ago
How can DataDog threaten open source developers? Can they sue them? what is it?
- davidjfelix 4y ago
  A little lost here as well but my read was that the PR author was working with outside parties (their company??) to get approval (sign a CLA?) and some third party asked for Datadog's opinion and Datadog advised that they wait for a Datadog feature. Due to this discussion, the third party was in "default no" and the PR author did not push the matter, expecting Datadog to release something, when they did not after several months, the author reopened and pushed to get approval. I could be totally wrong, but maybe combined we can make sense of it. From my read it feels like DD misguided this third party which then advised the author to spike.
deleted 4y ago
[deleted]
httgp 4y ago
Good news! The PR is moving again! https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1155145470 https://github.com/open-telemetry/opentelemetry-collector-co...
CSDude 4y ago
We switched to OpenTelemetry agent from DataDog agent. Out of all commercial providers, Datadog is the worst one to show traces properly. Although it captures mostly the same traces, all of them have io.opentelemetry.something on UI making it very hard to read. I’ve tried Honeycomb, NewRelic, Lightstep.
- ChrisCooneyUK 4y ago
  [dead]
boiler_up800 4y ago
Datadog is working on an OAuth API which may eventually enable some of this data to flow out of it https://www.datadoghq.com/blog/oauth/ https://www.datadoghq.com/blog/oauth/
ChrisCooneyUK 4y ago
I want to go for a drink with Moonheart (who used this unmerged, untested, year old PR in Production for a year) and find out how I, too, can become an absolute maverick.
effnorwood 4y ago
[dead]
oigursh 4y ago
Datadog support is hot garbage, and for the amount of money I've sent them over the years they ignore feature requests and just DGAF about customers. They're very hot on licensing though.

349 points by raybb 4y ago

Mizza 4y ago

Datadog is stupidly expensive for what it provides. Don't recommend and I hope something open kills it soon.

deleted 4y ago
[deleted]
icelancer 4y ago
Likewise for New Relic. Unfortunately OTel and everything similar doesn't remotely come close to the usability and ease of configuration that New Relic does. Not yet, anyway...
- judge2020 4y ago
  Ideally you just implement OTel into your apps and later plug that into any metric/log/trace collector you want, so you have DataDog and Newrelic and other frontends competing based on their feature suite and price. As it stands, this was/is trying to use DataDog's existing APM libraries to turn them into OTel for ingest into other providers.
kevincox 4y ago
I'm not so sure to be honest. It is very expensive. But being able to quickly get all of the features is fantastic for a small company. As you start to use more volume (and your bill starts growing) then it makes sense to start considering self-hosting. But for many teams with relatively small data volumes self-hosting is likely not worth the engineering time.
- zug_zug 4y ago
  Out of curiosity, is there anything self-hosted that compares to data-dog RUM?
pranay01 4y ago
you may want to have a look at SigNoz - https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz PS: I am one of the maintainers
bdcravens 4y ago
Plus they have (or had, just going off of my personal experience) very unethical sales practices.
boostchicken 4y ago
stabs repeatedly

domlaut 4y ago

Related, from Feb 2022, on what this is and why it's important: https://twitter.com/mipsytipsy/status/1494861690059759616 https://twitter.com/mipsytipsy/status/1494861690059759616

dpedu 4y ago

Nice tool. A couple years ago I wrote a similar tool that translates Datadog's agent's metrics/checks into AWS Cloudwatch equivalents.

raybb 4y ago

Link to exact comment: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1404724926 https://github.com/open-telemetry/opentelemetry-collector-co...

icepat 4y ago
Not sure I entirely follow what's going on here. Is there some context behind this that's useful to know?
- tonyhb 4y ago
  If I understand correctly, this is a receiver — which means you can take your existing DD instrumentation in their format and have it translated to OTel using this code. It lowers the switching cost to get off of DD.
  - trynewideas 4y ago
    And Datadog competitor Grafana sponsored it: https://twitter.com/boostchicken/status/1618692475845238784 https://twitter.com/boostchicken/status/1618692475845238784
    - deleted 4y ago
      [deleted]
  - xen2xen1 4y ago
    Protect the moat at all costs!
    - benatkin 4y ago
      It's worse than that, they want security through obscurity too. They feel like someone is inappropriately tinkering with the agent they want customers to install. It's open source: https://github.com/DataDog/datadog-agent https://github.com/DataDog/datadog-agent ...but the downloads are behind a login: https://github.com/DataDog/datadog-agent#datadog-agent https://github.com/DataDog/datadog-agent#datadog-agent

AYBABTME 4y ago

Tentative TLDR: It seems like DD pressured a contributor into not shipping a feature that would have made DD just another telemetry vendor. It seems like right now, data can flow IN datadog libraries/agents but not out. This PR would sort of allow data to flow OUT of datadog's libs/agents? And DD doesn't want that because it removes their lock-in power? Is this correct? This would be extremely crappy of Datadog.

deleted 4y ago
[deleted]
judge2020 4y ago
> This PR would sort of allow data to flow OUT of datadog's libs/agents? Yes. This allows you to expose a Telemetry collector on the datadog agent port 8126[0], allowing you to collect Application Performance Monitoring (APM) traces from any APM-enabled datadog library[1]. If I had to guess, DataDog's argument is that they don't want you using the engineering hours they invest into their libraries to have DD do the heavy lifting of collecting APM traces and send the messages off to another service. <removed OSS comment> 0: https://github.com/boostchicken/opentelemetry-collector-contrib/blob/datadog-receiver/receiver/datadogreceiver/README.md https://github.com/boostchicken/opentelemetry-collector-cont... 1: https://docs.datadoghq.com/tracing/#send-traces-to-datadog https://docs.datadoghq.com/tracing/#send-traces-to-datadog
- cmeacham98 4y ago
  > DataDog's libraries don't seem to be OSS: https://github.com/DataDog/datadogpy/blob/master/LICENSE https://github.com/DataDog/datadogpy/blob/master/LICENSE Is that not a verbatim the 3-clause BSD license (which is an FSF/OSI approved OSS license)?
- deleted 4y ago
  [deleted]
- aeyes 4y ago
  The library you linked is just for sending metric data (gauges, counters etc), the relevant library for APM is called ddtrace and its dual licensed with Apache2 and BSD3. https://github.com/DataDog/dd-trace-py/blob/1.x/LICENSEe https://github.com/DataDog/dd-trace-py/blob/1.x/LICENSEe

hijinks 4y ago

datadogs APM traces are the one thing that locks companies into Datadog. If this is merged it makes it a lot easier to move out of datadog into something else.

aeyes 4y ago
The only possible lock-in I see is if you have annotated code with Datadog specific tracing instructions and I'm pretty sure that most of these features aren't relevant for Open Telemetry. Switching APM providers isn't hard at all. Maybe the Open Telemetry ecosystem lacks a good agent but my understanding is that their APM agent is pretty much already a copy of the Datadog agent. I think New Relic would be a more interesting agent to use because their approach to APM isn't primarily to do sampled traces.

deleted 4y ago

[deleted]

funerr 4y ago

Here is where it started to go downhill: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1032906655 https://github.com/open-telemetry/opentelemetry-collector-co...

TheRealPomax 4y ago
And then went back uphill in https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1404724926 https://github.com/open-telemetry/opentelemetry-collector-co...

floatinglotus 4y ago

More vendor lock-in shenanigans. Data dog has always been a proprietary POS. I don’t know why people use it, APM traces? How long before Grafana has these capabilities in OSS? So annoying seeing a company like DD who cannot innovate at all, trying to lock in the average company.

predakanga 4y ago
> How long before Grafana has these capabilities in OSS? They've already started down that path with Grafana Tempo[0]. It's functional, but their UX and discoverability need a lot of work. [0]: https://grafana.com/oss/tempo/ https://grafana.com/oss/tempo/
dalyons 4y ago
Do you actually use it at scale in a large complicated system? IMHO it’s the opposite - datadog is consistently the most innovative and fastest iterating observability platform out there. Nothing comes close. It’s expensive yes, and there’s some lock-in yes, but they are GOOD.
Mesverrum 4y ago
As someone who has been in this space a long time I think you are pretty off base here. The commercial vendors have absolutely been the ones who did all the innovations in tracing. The OSS equivalents were really poor imitations for the longest time. Zipkin/Jaeger/Opentrace have been coming along slowly for many years, but it's really been maybe 2 years since that work has really become competitive with what was available in the commercial APM space. For the most part it's felt like that only happened because the commercial tools have all halted new work on their proprietary tools and told their staff to push Otel over the finish line. None of them actually want to pay teams of engineers to maintain proprietary libraries for every language, the execs are absolutely drooling to find a way to pawn as much of that cost off onto others via Otel. Locking people in on agents is naive when the truly heavy lift in migrating platforms is the swapping dashboards and alerts and retraining all your staff. You see the same thing around integrations, everyone used to have to roll their own proprietary chunks of code that in the end were all querying mostly the same data points back from servers and API's. Now everyone just prefers to wait for the Prometheus exporter and they adopt that instead.

bdcravens 4y ago

I'm always willing to dogpile on DataDog (no pun intended), given bad experiences with their sales, but all I'm reading out of this is that the DD person didn't want to review it out of a potential conflict of interest. It was reviewed however by someone else. Am I missing something here?

nabakin 4y ago
You're missing that this was a year-old PR ready to merge but it was stopped from merging by DataDog because it makes it easier for users to migrate from DataDog to another logging service
- bdcravens 4y ago
  No, I am unsurprised by the DataDog taking that position; it fits with what I know of the company. However as someone pointed out, the comment in question wasn't originally linked, but I see it now.
- bushbaba 4y ago
  Where did DD stop the merge. I just saw them simply state that the reviewer didn't want to review it as they work at Datadog. It could simply be that they are biased TO merge it even if it contains flaws. Having an independent reviewer ensures the code quality meets the bar.
  - jru 4y ago
    The OP didn't link directly to the comment and it's buried in the thread: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1404724926 https://github.com/open-telemetry/opentelemetry-collector-co...
s17n 4y ago
https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1404724926 https://github.com/open-telemetry/opentelemetry-collector-co...

joshxyz 4y ago

I love this bit: > Hello, I'm using this receiver in production for about one year, and left some comments that may be helpful. >> Are you serious? That is intense. Did it scale? Any memory issues? If I remember correctly I tuned all that away. Always astonishing to see broken stuff doing well in prod lol.

quickthrower2 4y ago
“Not production ready” stopped noone

capableweb 4y ago

From @mx-psi > Speaking as a Collector contrib maintainer, I just wanted to say that I am not going to continue reviewing this PR or start reviewing other future PRs related to the Datadog APM receiver to avoid any semblance of conflict of interest given my role both as a maintainer on this repository and as a member of Datadog's OpenTelemetry team. And from their GitHub profile: > Open Source Software Engineer at Datadog , focusing on OpenTelemetry What's the conflict of interest exactly? You work at Datadog, supposedly to work on OSS, with a focus on OpenTelemetry and you don't want to review Datadog related code for OpenTelemetry? Sounds weird, that kind of profile is exactly the type of person who should be reviewing the code, they have knowledge of both sides of it. Rather, it sounds like Datadog is walking back and don't want to support OpenTelemetry if it means it'll support their own tooling, instead of just others.

judge2020 4y ago
The potential conflict is that people might think/perceive DD to still be out to harm this project given their previous request for the author to stop development. Anything that might be regular code review or discussion for "why did you choose this approach" could be seen as sabotage on this super public issue.

stvnbn 4y ago

How can DataDog threaten open source developers? Can they sue them? what is it?

davidjfelix 4y ago
A little lost here as well but my read was that the PR author was working with outside parties (their company??) to get approval (sign a CLA?) and some third party asked for Datadog's opinion and Datadog advised that they wait for a Datadog feature. Due to this discussion, the third party was in "default no" and the PR author did not push the matter, expecting Datadog to release something, when they did not after several months, the author reopened and pushed to get approval. I could be totally wrong, but maybe combined we can make sense of it. From my read it feels like DD misguided this third party which then advised the author to spike.

deleted 4y ago

[deleted]

httgp 4y ago

Good news! The PR is moving again! https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1155145470 https://github.com/open-telemetry/opentelemetry-collector-co...

CSDude 4y ago

We switched to OpenTelemetry agent from DataDog agent. Out of all commercial providers, Datadog is the worst one to show traces properly. Although it captures mostly the same traces, all of them have io.opentelemetry.something on UI making it very hard to read. I’ve tried Honeycomb, NewRelic, Lightstep.

ChrisCooneyUK 4y ago
[dead]

boiler_up800 4y ago

Datadog is working on an OAuth API which may eventually enable some of this data to flow out of it https://www.datadoghq.com/blog/oauth/ https://www.datadoghq.com/blog/oauth/

ChrisCooneyUK 4y ago

I want to go for a drink with Moonheart (who used this unmerged, untested, year old PR in Production for a year) and find out how I, too, can become an absolute maverick.

effnorwood 4y ago

[dead]

oigursh 4y ago

Datadog support is hot garbage, and for the amount of money I've sent them over the years they ignore feature requests and just DGAF about customers. They're very hot on licensing though.