5 ms·

OpenTelemetry is going to be an existential threat to DataDog and other companies that effectively rely on vendor lock-in to exploit customers. Not sure how com

OpenTelemetry is going to be an existential threat to DataDog and other companies that effectively rely on vendor lock-in to exploit customers. Not sure how companies rationalize these types of services at scale when there are so many open source options to run for a fraction of the cost. You could hire 100+ engineers and still save money compared to a 65M bill https://news.ycombinator.com/item?id=34540419 https://news.ycombinator.com/item?id=34540419

ricardobeat 3y ago
I’ve seen an attempted move from DD to OT and it was a nightmare of undocumented features and little compounded issues. Tracing was non functional. It doesn’t seem mature enough yet.
- icelancer 3y ago
  Were you watching me? Because that's what happened. I am a user of New Relic. Not because I'm happy. But because OpenTelemetry doesn't come close to the same features. Fortunately, at least OT is about 10x harder to set up with worse documentation. Wait a minute...
  - VirusNewbie 3y ago
    yup at my last company CTO wanted to switch away from New Relic to save money for Open Telemetry and I can tell you we wasted months of work because OpenTelemetry is dogshit. New Relic is really really good too, so it was even more painful.
    - icelancer 3y ago
      Yeah, New Relic is just an excellent product. No way around it.
- hosh 3y ago
  We had to use otel with Datadog because Datadog did not support Elixir with an official SDK. There are a lot of subtle issues, but we've been able to work through them to get usable traces. (Metrics and log ingestion already have pretty good existing open-source tooling, like statsd).
- rch 3y ago
  There's some purposeful friction with DD when it comes to OTel. Incumbents in this space are in for a rough time as more applications provide meaningful telemetry, beyond just logs. Fortunately for them that timeline is 'fuzzy' at best.
- SomaticPirate 3y ago
  Apparently OT is a threat to DD to the point of them asking contributors to not add support for their agents… https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1404724926 https://github.com/open-telemetry/opentelemetry-collector-co...
- pranay01 3y ago
  I am not sure when you tried OpenTelemetry, but it is decently mature now, esp. for tracing. I am a maintainer at SigNoz (https://github.com/signoz/signoz https://github.com/signoz/signoz) and we have good support for tracing using Otel for most of the common frameworks. I agree it was a bit rapidly evolving in early days, but now its much more mature. You can check out our docs for distributed tracing here - https://signoz.io/docs/instrumentation/ https://signoz.io/docs/instrumentation/
api 3y ago
> You could hire 100+ engineers and still save money compared to a 65M bill I see cloud costs like this a lot and it really puzzles me. It seems like people would rather pay 10X+ more to just not have to think about it than even to hire other people to think about it, because then you have to think about hiring and HR. "Here's a blank check. Just make it go away." Of course corporate consultants run on that, so I guess it's not without abundant precedent elsewhere. I guess if you work for a big company with budget and it's not your money you really have little incentive not to take the easy path.
- cyberpunk 3y ago
  Operational costs can be billed to a project. It's not really that the business doesn't want to save money on this stuff, but it's much easier to wind up in this situation when the loops to hire some engineers are 4x as complicated as just adding another sub-org to the data dog billing.. Does seem like a pretty wild bill, thou.
- ptero 3y ago
  This is indeed the case and one can argue that it makes sense for a small company to focus on the MVP and initial growth. But every such decision needs to be reexamined from time to time as the company scales up. That is unpleasant, requires an expert, so many businesses procrastinate on this and make expensive mistakes. My 2c.
- MisterPea 3y ago
  I would argue it's a much easier blank check to write than for consultants. Software complexity over time is a major headache and new initiatives happen all the time. Take for instance GDPR - in AWS it was a company wide effort to get all the services GDPR compliant and that was basically a non-existent pricing change to consumers. Also the fact that I can call up AWS support and have them look into a bug immediately with real devs on the other end is invaluable when my business needs rely on a certain feature working.
- ethbr0 3y ago
  The real issue is finding the right 100+ engineers and then managing them. That takes calendar time. There's a multiple to what people are willing to pay for SaaS solutions precisely because HR for knowledge work is such a pain.
rocmcd 3y ago
OpenTelemetry (or OTel) is in no way an existential threat to DataDog. Primarily because OTel is simply the substrate/protocol by which data is collected from your apps/systems. DD does _a lot_ more than what OTel provides (RUM, SIEM, synthetics, on-call, dashboarding, anomaly detection, and much much more). If anything it's an existential threat to the more legacy vendors that aren't equipped to provide an OTel ingest layer. OTel _does_ prevent lock-in on the agent side (making it easier to switch vendors) with open source components and consistent schemas, but OTel doesn't enable you to do anything that you couldn't do before with a specific vendor. It just empowers you to take ownership of your observability data, should you want to. Many don't, though. They want to throw money at someone else who can do it relatively well, hence the ridiculous DD pricing. Regarding: > Not sure how companies rationalize these types of services at scale when there are so many open source options to run for a fraction of the cost At Coinbase's scale (which I assume is a lot due to the DD bill, but I haven't looked closely at it) the open source options simply won't cut it. Plus there are no scalable open source options for many of the things that DD does (synthetics, SIEM come to mind - not to mention onerous regulatory requirements). 65M seems like a lot, but it also means their cloud costs are insane - so maybe it lets them put focus elsewhere?
- petersellers 3y ago
  > OTel _does_ prevent lock-in on the agent side (making it easier to switch vendors) with open source components and consistent schemas, but OTel doesn't enable you to do anything that you couldn't do before with a specific vendor This is exactly the reason why we are moving away from NewRelic's SDK to an OpenTelemetry SDK (even though we are still using NewRelic to ingest everything). If (more like when) we decide to switch vendors, it will be much easier to do so.
- jrockway 3y ago
  I don't know the exact history of OTel, but I agree with you. A few years ago, open source applications basically included one type of telemetry: Prometheus metrics. (Maybe Jaeger for tracing if you're really, really lucky.) What OTel is is a way to write an open-source library that emits telemetry, but without dictating that the end user use a particular storage system. Basically, you can use Datadog instead of the open-source stuff, even if the library authors have never heard of Datadog. Datadog specifically, I don't know if they care. They had an army of junior engineers that existed to hack Datadog into every open source project imaginable. If something had monitoring, they would just add their vendor-specific stuff and upstream it. That was probably expensive but probably accounts for a vast amount of their early marketshare. The other vendors wanted in on that racket without having to do too much work; OTel was born.
vasco 3y ago
Presumably these companies have higher ROI projects to do with 100 engineers than reimplementing and maintaining an in-house datadog alternative.
pranay01 3y ago
We are building SigNoz (https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz) - an open source alternative to DataDog. We are natively based on opentelemetry and see lots of our users very interested in that. As mentioned in some other places in the thread, DataDog pricing is very unpredictable and high - and I think more open standards based solutions are the way forward which provides users more predictability and flexibility

ricardobeat 3y ago

I’ve seen an attempted move from DD to OT and it was a nightmare of undocumented features and little compounded issues. Tracing was non functional. It doesn’t seem mature enough yet.

icelancer 3y ago
Were you watching me? Because that's what happened. I am a user of New Relic. Not because I'm happy. But because OpenTelemetry doesn't come close to the same features. Fortunately, at least OT is about 10x harder to set up with worse documentation. Wait a minute...
- VirusNewbie 3y ago
  yup at my last company CTO wanted to switch away from New Relic to save money for Open Telemetry and I can tell you we wasted months of work because OpenTelemetry is dogshit. New Relic is really really good too, so it was even more painful.
  - icelancer 3y ago
    Yeah, New Relic is just an excellent product. No way around it.
hosh 3y ago
We had to use otel with Datadog because Datadog did not support Elixir with an official SDK. There are a lot of subtle issues, but we've been able to work through them to get usable traces. (Metrics and log ingestion already have pretty good existing open-source tooling, like statsd).
rch 3y ago
There's some purposeful friction with DD when it comes to OTel. Incumbents in this space are in for a rough time as more applications provide meaningful telemetry, beyond just logs. Fortunately for them that timeline is 'fuzzy' at best.
SomaticPirate 3y ago
Apparently OT is a threat to DD to the point of them asking contributors to not add support for their agents… https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836#issuecomment-1404724926 https://github.com/open-telemetry/opentelemetry-collector-co...
pranay01 3y ago
I am not sure when you tried OpenTelemetry, but it is decently mature now, esp. for tracing. I am a maintainer at SigNoz (https://github.com/signoz/signoz https://github.com/signoz/signoz) and we have good support for tracing using Otel for most of the common frameworks. I agree it was a bit rapidly evolving in early days, but now its much more mature. You can check out our docs for distributed tracing here - https://signoz.io/docs/instrumentation/ https://signoz.io/docs/instrumentation/

api 3y ago

> You could hire 100+ engineers and still save money compared to a 65M bill I see cloud costs like this a lot and it really puzzles me. It seems like people would rather pay 10X+ more to just not have to think about it than even to hire other people to think about it, because then you have to think about hiring and HR. "Here's a blank check. Just make it go away." Of course corporate consultants run on that, so I guess it's not without abundant precedent elsewhere. I guess if you work for a big company with budget and it's not your money you really have little incentive not to take the easy path.

cyberpunk 3y ago
Operational costs can be billed to a project. It's not really that the business doesn't want to save money on this stuff, but it's much easier to wind up in this situation when the loops to hire some engineers are 4x as complicated as just adding another sub-org to the data dog billing.. Does seem like a pretty wild bill, thou.
ptero 3y ago
This is indeed the case and one can argue that it makes sense for a small company to focus on the MVP and initial growth. But every such decision needs to be reexamined from time to time as the company scales up. That is unpleasant, requires an expert, so many businesses procrastinate on this and make expensive mistakes. My 2c.
MisterPea 3y ago
I would argue it's a much easier blank check to write than for consultants. Software complexity over time is a major headache and new initiatives happen all the time. Take for instance GDPR - in AWS it was a company wide effort to get all the services GDPR compliant and that was basically a non-existent pricing change to consumers. Also the fact that I can call up AWS support and have them look into a bug immediately with real devs on the other end is invaluable when my business needs rely on a certain feature working.
ethbr0 3y ago
The real issue is finding the right 100+ engineers and then managing them. That takes calendar time. There's a multiple to what people are willing to pay for SaaS solutions precisely because HR for knowledge work is such a pain.

rocmcd 3y ago

OpenTelemetry (or OTel) is in no way an existential threat to DataDog. Primarily because OTel is simply the substrate/protocol by which data is collected from your apps/systems. DD does _a lot_ more than what OTel provides (RUM, SIEM, synthetics, on-call, dashboarding, anomaly detection, and much much more). If anything it's an existential threat to the more legacy vendors that aren't equipped to provide an OTel ingest layer. OTel _does_ prevent lock-in on the agent side (making it easier to switch vendors) with open source components and consistent schemas, but OTel doesn't enable you to do anything that you couldn't do before with a specific vendor. It just empowers you to take ownership of your observability data, should you want to. Many don't, though. They want to throw money at someone else who can do it relatively well, hence the ridiculous DD pricing. Regarding: > Not sure how companies rationalize these types of services at scale when there are so many open source options to run for a fraction of the cost At Coinbase's scale (which I assume is a lot due to the DD bill, but I haven't looked closely at it) the open source options simply won't cut it. Plus there are no scalable open source options for many of the things that DD does (synthetics, SIEM come to mind - not to mention onerous regulatory requirements). 65M seems like a lot, but it also means their cloud costs are insane - so maybe it lets them put focus elsewhere?

petersellers 3y ago
> OTel _does_ prevent lock-in on the agent side (making it easier to switch vendors) with open source components and consistent schemas, but OTel doesn't enable you to do anything that you couldn't do before with a specific vendor This is exactly the reason why we are moving away from NewRelic's SDK to an OpenTelemetry SDK (even though we are still using NewRelic to ingest everything). If (more like when) we decide to switch vendors, it will be much easier to do so.
jrockway 3y ago
I don't know the exact history of OTel, but I agree with you. A few years ago, open source applications basically included one type of telemetry: Prometheus metrics. (Maybe Jaeger for tracing if you're really, really lucky.) What OTel is is a way to write an open-source library that emits telemetry, but without dictating that the end user use a particular storage system. Basically, you can use Datadog instead of the open-source stuff, even if the library authors have never heard of Datadog. Datadog specifically, I don't know if they care. They had an army of junior engineers that existed to hack Datadog into every open source project imaginable. If something had monitoring, they would just add their vendor-specific stuff and upstream it. That was probably expensive but probably accounts for a vast amount of their early marketshare. The other vendors wanted in on that racket without having to do too much work; OTel was born.

vasco 3y ago

Presumably these companies have higher ROI projects to do with 100 engineers than reimplementing and maintaining an in-house datadog alternative.

pranay01 3y ago

We are building SigNoz (https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz) - an open source alternative to DataDog. We are natively based on opentelemetry and see lots of our users very interested in that. As mentioned in some other places in the thread, DataDog pricing is very unpredictable and high - and I think more open standards based solutions are the way forward which provides users more predictability and flexibility