5 ms·

Datadog's $65M/year customer mystery solved

152 points by thunderbong 1y ago

hagen1778 1y ago
My understanding is that with Prometheus+Grafana, and the rest of their stack, you can achieve the same functionality as Datadog (or even more) at much lower costs. But, it requires engineering time to set up these tools, monitor them, build dashboards and alerts. Build an observability platform at home, in other words. But what about other open source solutions that already trying very hard to become an out-of-box solution for observability? Things like Netdata, Hyperdx, Coroot, etc. are already platforms for all telemetry signals, with fancy UIs and a lot of presets. Why people don't use them instead of Datadog?
- Cpoll 1y ago
  > or even more Grafana isn't quite as featureful as Datadog, though nothing to keep you from getting the job done. > But, it requires engineering time to set up these tools At some price point, you have to wonder if it doesn't make more sense to hire engineers to get it just right for your use case. I'd bet that price point is less than $65MM. Hell, you could have people full-time on Grafana to add features you want.
mrkramer 1y ago
And who says that SaaS doesn't pay off?! It pays off like hell!
deleted 1y ago
[deleted]
delichon 1y ago
> For observability, Coinbase spun up a dedicated team with the goal of moving off of Datadog, and onto a Grafana/Prometheus/Clickhouse stack. We recently did the same, and our Datadog bill was only five figures. We're finding the new stack to not be a poor man's anything, but more flexible, complete and manageable than yet another SaaS. With just a little extra learning curve observability is a domain where open source trounces proprietary, and not just if you don't have money to set on fire.
- oulipo 1y ago
  Have you tried the ClickStack? https://news.ycombinator.com/item?id=44194082 https://news.ycombinator.com/item?id=44194082
asnyder 1y ago
There's also https://openobserve.ai https://openobserve.ai, while not as stable as Grafana/Prometheus/Clickhouse, feels a bit easier to setup and manage. Though has a bit of ways to go, does the basics and more without issue. Crazy crazy they spent so much on observability. Even with DataDog they could've optimized that spend. DataDog does lots of bad things with billing where by default, especially with on-demand instances you get charged significantly more than you should as they have (had?) pretty deficient counting towards instance hours and instances. For example, rather than run the agent (which counts as an instance regardless of if it's on for a minute), you can send the logs, metrics, etc. directly to their ingestion endpoints and not have those instances counted towards their usage other than log and metric usage. Maybe at that level they don't even get into actual by usage anymore, and they just negotiate arbitrary amounts for some absurd quota of use.
ljm 1y ago
I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture. Coinbase might have been fine blowing 65 mil but take that approach to a new startup and you could trivially eat up a significant amount of runway with it. I won’t single out Datadog on this because the exact same thing happens with cloud spend, and it’s very literally burning money.
- swyx 1y ago
  the visible cost of burning runway on a bill is very often far less than the invisible cost of burning engineer time rebuilding undifferentiated heavy lifting rather than working on product/customer needs
  - 9283409232 1y ago
    People say this but I wonder about this from time to time. I don't think anyone is asking to rebuild datadog from scratch for your company but surely it's worth it to migrate to something not as expensive even if it takes a bit of elbow grease.
    - closeparen 1y ago
      Assuming there's nothing else you could do with that elbow grease that would create more value than the SaaS bill costs.
      - 9283409232 1y ago
        Value is not a hard science. I've seen people shelve tech debt in favor of work on a feature that no one ends up using.
        nemothekid 1y ago
        1. Leadership doesn’t want to burn engineer cycles on undifferentiated features. 2. Management doesn’t get recognized for working on undifferentiated features. 3. Engineers working on undifferentiated features aren’t recognized when looking for new jobs. Saving money “makes” sense but getting people to actually prioritize it is hard.
abxyz 1y ago
(May 2023)
everfrustrated 1y ago
>Originally published on 11 May 2023
cybice 1y ago
An article that's basically an ad for Datadog: Pay us a ton of money - it’s still cheaper in the long run.
decimalenough 1y ago
> Assume that Datadog cuts the number of outages by half, by preventing them with early monitoring. That would mean that without Datadog, we’d look at 24 hours’ worth of downtime, not 12. Let’s also assume that using Datadog results in mitigating outages 50% faster than without - thanks to being able to connect health metrics with logs, debug faster, pinpoint the root cause and mitigate faster. In that case, without Datadog, we could be looking at 36 hours worth of total downtime, versus the 12 hours with Datadog. To put it in numbers: the company would make around $9M in revenue it would otherwise lose, Now that $10M/year fee practically pays for itself! Those are some pretty heroic assumptions. In particular, they assume the only options are Datadog or nothing, when there are far cheaper alternatives like the Prometheus/Grafana/Clickhouse stack mentioned in the article itself.
- passivepinetree 1y ago
  Another assumption that bothers me here is that the $9M in revenue would be completely lost during an outage. I imagine many customers would simply wait until the outage was resolved before performing their intended transactions, meaning far less than $9M would be lost.
  - calt 1y ago
    On the other hand, customers can become frustrated at being unable to trade when they need during an outage to and go to a competitor.
- secondcoming 1y ago
  We are moving from Datadog to Prometheus/Grafana and it's really not all a bed of roses. You'll need monitoring on your monitoring.
  - hagen1778 1y ago
    Ofc you need to monitor your monitoring, because you run it. Datadog runs their own systems and monitors them, that's why they charge you so much. I barely can imagine a criticial piece of software that I need to run and not monitor it in the same time.
cloudking 1y ago
What problems does Datadog solve that you can't solve with cheaper solutions?
- remify 1y ago
  It's honestly easy to use and implements. It's a try it and adopt it product, it's even one of their sales tactics.
- xp84 1y ago
  In my career, I have found that this is not usually the question to ask. I am very confident that I could put together and support a solution that does everything I need Datadog to do and has direct costs far, far lower. However, this would also consume a noticeable fraction of my time, both in predictable ways (maintenance, feature adds that I decide I need), and in unpredictable ways (oh no it’s down). I believe a much more useful question to ask is just “is this the highest and best use of my finite attention and time?” It is much easier to find $100,000 a year of budget than it is to find an additional $50,000 worth of skilled[1] developer time. [1] This skilled part is critical because if you have some flunky create your “SaaS alternative” you are in for an even worse time.
therein 1y ago
I should have known it was Coinbase. I know that Coinbase used to spend $35,000 a month to back up the data directory of ETH nodes.
- mrkramer 1y ago
  >I know that Coinbase used to spend $35,000 a month to back up the data directory of ETH nodes. They paid backups to whom? Who was vendor....I'm interested.
  - therein 1y ago
    Amazon. They put it on an EBS volume and kept it around.
    - mrkramer 1y ago
      Ah ok, I thought it was some startup.
- xp84 1y ago
  Of all the things Coinbase could spend money on, backups may be the smartest! Imagine if they lost a few billion dollars worth of crypto to a faulty SSD or something!
deleted 1y ago
[deleted]
aeyes 1y ago
> we really work with customers to restructure their contracts Does anyone have such an experience with Datadog? A few million wasn't enough to get them to talk about anything, always paid list price and there was no negotiating either when they restructured their pricing.
- arccy 1y ago
  you've got bad negotiators... getting at least 10% off list price should be the baseline, even on less than $1m/year
- evulhotdog 1y ago
  They were completely unwilling to negotiate with us at all, and it forced our hand to go other open source routes so we don’t get locked in again.
GuinansEyebrows 1y ago
> To put it in numbers: the company would make around $9M in revenue it would otherwise lose, Now that $10M/year fee practically pays for itself! am i misunderstanding, or is the author saying it's better to spend $10m than $9m?
- areyourllySorry 1y ago
  you spend that extra million to keep customers satisfied in a competitive industry. they have users trading hundreds of thousands - if there's downtime and they lose money because they weren't able to sell their positions at the right time, they might even try to sue, who knows
gneray 1y ago
This person is like the Gossip Guy of tech. Who cares?
generalpf 1y ago
When did this guy stop writing about engineering and start running a tech gossip rag?
willejs 1y ago
I have run ELK, Grafana + Prom, Grafana + Thanos/Coretex, New relic and all of the more traditional products for monitoring/observability. More recently in the last few years, I have been running full observability stacks via either The Grafana LGTM stack or datadog at a reasonable scale and complexities. Ultimately you want one tool that can alert you off a metric, present you some traces, and drill down into logs, all the way down the stack. I have found Datadog to be, by far hands down the best developer experience from the get go, the way it glues the mostly decent products together is unparalleled in comparison to other products (Grafana cloud/LGTM). I usually say if your at a small to medium scale business just makes sense, IF you understand the product and configure it correctly which is reasonably easy. The seamless integration between tracing, logging and metrics in the platform, which you can then easily combine with alerts is great. However, its easy to misconfigure it and spend a lot of money on seemingly nothing. If you do not implement tracing and structured logs (at the right volume and level) with trace/span ids etc all the way through services its hard to see the value, and seems expensive. It requires some good knowledge, and configuration of the product to make it pay off. The rest of the product features are generally good, for example their security suite is a good entry level to cloud security monitoring and SEIM too. However, when you get to a certain scale, the cost of APM and Infrastructure hosts in Datadog can become become somewhat prohibitive. Also, Datadogs custom metrics pricing is somewhat expensive and its query language cababilities does not quite match the power of promql, and you start to find yourself needed them to debug issues. At that point, the self hosted LGTM stack starts to make sense, however, it involves a lot more education for end users in both integration (a little less now Otel is popular) and querying/building dashboards etc, but also running it yourself. The grafana cloud platform is more attractive though.
- SOLAR_FIELDS 1y ago
  My experience mirrors yours wrt Datadog. It's incredible value at low scale, you get a full robust system with great devex for pennies. Once you hit that tipping point though, you are locked in pretty hardcore. Datadog snakes its way far into your codebase, with all the custom tracing and stuff like that. Migrating off of it is a very expensive endeavor, which is probably one of the reasons why they are such a money printing operation.
wenbin 1y ago
Earlier this year, we at Listen Notes switched to Better Stack [0], replacing both Datadog and PagerDuty, and we couldn’t be happier :) Datadog offers a rich set of features, and as a public company, it makes sense for them to keep expanding their product and pushing larger contracts. But as a small team, we don't have a strong demand for constant new features. By switching to Better Stack, we were able to cut our monitoring and alerting costs by 90%, with basically the same things that we used from Datadog previously. [0] https://www.listennotes.com/blog/use-betterstack-to-replace-pagerduty-datadog-and-112/ https://www.listennotes.com/blog/use-betterstack-to-replace-...
iwovtb 1y ago
Observability spend is super expensive. Realizing this, we built grepr.ai.... reduces 96% spend without change. Check it out... Not just a dumb pipeline either.
iwovtb 1y ago
Observability is expensive. Rip/replace is hard... we built grepr.ai to solve this problem and are seeing 96% reduction in spend from Splunk/Sumo/New Relic/Datadog etc. No change mgt. The result set is: 96% reduction in spend and noise elimination. Pretty compelling.. come check us out. www.grepr.ai
- MehdiHK 1y ago
  No pricing page? Also how does it compare to Cribl?
east4ming 1y ago
Think the other way around, engineers are also a cost, and investment in open-source software is also a cost. If engineers are very cheap, then the company would choose to build it themselves, right?

152 points by thunderbong 1y ago

My understanding is that with Prometheus+Grafana, and the rest of their stack, you can achieve the same functionality as Datadog (or even more) at much lower costs. But, it requires engineering time to set up these tools, monitor them, build dashboards and alerts. Build an observability platform at home, in other words. But what about other open source solutions that already trying very hard to become an out-of-box solution for observability? Things like Netdata, Hyperdx, Coroot, etc. are already platforms for all telemetry signals, with fancy UIs and a lot of presets. Why people don't use them instead of Datadog?

Cpoll 1y ago
> or even more Grafana isn't quite as featureful as Datadog, though nothing to keep you from getting the job done. > But, it requires engineering time to set up these tools At some price point, you have to wonder if it doesn't make more sense to hire engineers to get it just right for your use case. I'd bet that price point is less than $65MM. Hell, you could have people full-time on Grafana to add features you want.

mrkramer 1y ago

And who says that SaaS doesn't pay off?! It pays off like hell!

deleted 1y ago

[deleted]

delichon 1y ago

> For observability, Coinbase spun up a dedicated team with the goal of moving off of Datadog, and onto a Grafana/Prometheus/Clickhouse stack. We recently did the same, and our Datadog bill was only five figures. We're finding the new stack to not be a poor man's anything, but more flexible, complete and manageable than yet another SaaS. With just a little extra learning curve observability is a domain where open source trounces proprietary, and not just if you don't have money to set on fire.

oulipo 1y ago
Have you tried the ClickStack? https://news.ycombinator.com/item?id=44194082 https://news.ycombinator.com/item?id=44194082

asnyder 1y ago

There's also https://openobserve.ai https://openobserve.ai, while not as stable as Grafana/Prometheus/Clickhouse, feels a bit easier to setup and manage. Though has a bit of ways to go, does the basics and more without issue. Crazy crazy they spent so much on observability. Even with DataDog they could've optimized that spend. DataDog does lots of bad things with billing where by default, especially with on-demand instances you get charged significantly more than you should as they have (had?) pretty deficient counting towards instance hours and instances. For example, rather than run the agent (which counts as an instance regardless of if it's on for a minute), you can send the logs, metrics, etc. directly to their ingestion endpoints and not have those instances counted towards their usage other than log and metric usage. Maybe at that level they don't even get into actual by usage anymore, and they just negotiate arbitrary amounts for some absurd quota of use.

ljm 1y ago

I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture. Coinbase might have been fine blowing 65 mil but take that approach to a new startup and you could trivially eat up a significant amount of runway with it. I won’t single out Datadog on this because the exact same thing happens with cloud spend, and it’s very literally burning money.

swyx 1y ago
the visible cost of burning runway on a bill is very often far less than the invisible cost of burning engineer time rebuilding undifferentiated heavy lifting rather than working on product/customer needs
- 9283409232 1y ago
  People say this but I wonder about this from time to time. I don't think anyone is asking to rebuild datadog from scratch for your company but surely it's worth it to migrate to something not as expensive even if it takes a bit of elbow grease.
  - closeparen 1y ago
    Assuming there's nothing else you could do with that elbow grease that would create more value than the SaaS bill costs.
    - 9283409232 1y ago
      Value is not a hard science. I've seen people shelve tech debt in favor of work on a feature that no one ends up using.
      - nemothekid 1y ago
        1. Leadership doesn’t want to burn engineer cycles on undifferentiated features. 2. Management doesn’t get recognized for working on undifferentiated features. 3. Engineers working on undifferentiated features aren’t recognized when looking for new jobs. Saving money “makes” sense but getting people to actually prioritize it is hard.

abxyz 1y ago

(May 2023)

everfrustrated 1y ago

>Originally published on 11 May 2023

cybice 1y ago

An article that's basically an ad for Datadog: Pay us a ton of money - it’s still cheaper in the long run.

decimalenough 1y ago

> Assume that Datadog cuts the number of outages by half, by preventing them with early monitoring. That would mean that without Datadog, we’d look at 24 hours’ worth of downtime, not 12. Let’s also assume that using Datadog results in mitigating outages 50% faster than without - thanks to being able to connect health metrics with logs, debug faster, pinpoint the root cause and mitigate faster. In that case, without Datadog, we could be looking at 36 hours worth of total downtime, versus the 12 hours with Datadog. To put it in numbers: the company would make around $9M in revenue it would otherwise lose, Now that $10M/year fee practically pays for itself! Those are some pretty heroic assumptions. In particular, they assume the only options are Datadog or nothing, when there are far cheaper alternatives like the Prometheus/Grafana/Clickhouse stack mentioned in the article itself.

passivepinetree 1y ago
Another assumption that bothers me here is that the $9M in revenue would be completely lost during an outage. I imagine many customers would simply wait until the outage was resolved before performing their intended transactions, meaning far less than $9M would be lost.
- calt 1y ago
  On the other hand, customers can become frustrated at being unable to trade when they need during an outage to and go to a competitor.
secondcoming 1y ago
We are moving from Datadog to Prometheus/Grafana and it's really not all a bed of roses. You'll need monitoring on your monitoring.
- hagen1778 1y ago
  Ofc you need to monitor your monitoring, because you run it. Datadog runs their own systems and monitors them, that's why they charge you so much. I barely can imagine a criticial piece of software that I need to run and not monitor it in the same time.

cloudking 1y ago

What problems does Datadog solve that you can't solve with cheaper solutions?

remify 1y ago
It's honestly easy to use and implements. It's a try it and adopt it product, it's even one of their sales tactics.
xp84 1y ago
In my career, I have found that this is not usually the question to ask. I am very confident that I could put together and support a solution that does everything I need Datadog to do and has direct costs far, far lower. However, this would also consume a noticeable fraction of my time, both in predictable ways (maintenance, feature adds that I decide I need), and in unpredictable ways (oh no it’s down). I believe a much more useful question to ask is just “is this the highest and best use of my finite attention and time?” It is much easier to find $100,000 a year of budget than it is to find an additional $50,000 worth of skilled[1] developer time. [1] This skilled part is critical because if you have some flunky create your “SaaS alternative” you are in for an even worse time.

therein 1y ago

I should have known it was Coinbase. I know that Coinbase used to spend $35,000 a month to back up the data directory of ETH nodes.

mrkramer 1y ago
>I know that Coinbase used to spend $35,000 a month to back up the data directory of ETH nodes. They paid backups to whom? Who was vendor....I'm interested.
- therein 1y ago
  Amazon. They put it on an EBS volume and kept it around.
  - mrkramer 1y ago
    Ah ok, I thought it was some startup.
xp84 1y ago
Of all the things Coinbase could spend money on, backups may be the smartest! Imagine if they lost a few billion dollars worth of crypto to a faulty SSD or something!

deleted 1y ago

[deleted]

aeyes 1y ago

> we really work with customers to restructure their contracts Does anyone have such an experience with Datadog? A few million wasn't enough to get them to talk about anything, always paid list price and there was no negotiating either when they restructured their pricing.

arccy 1y ago
you've got bad negotiators... getting at least 10% off list price should be the baseline, even on less than $1m/year
evulhotdog 1y ago
They were completely unwilling to negotiate with us at all, and it forced our hand to go other open source routes so we don’t get locked in again.

GuinansEyebrows 1y ago

> To put it in numbers: the company would make around $9M in revenue it would otherwise lose, Now that $10M/year fee practically pays for itself! am i misunderstanding, or is the author saying it's better to spend $10m than $9m?

areyourllySorry 1y ago
you spend that extra million to keep customers satisfied in a competitive industry. they have users trading hundreds of thousands - if there's downtime and they lose money because they weren't able to sell their positions at the right time, they might even try to sue, who knows

gneray 1y ago

This person is like the Gossip Guy of tech. Who cares?

generalpf 1y ago

When did this guy stop writing about engineering and start running a tech gossip rag?

willejs 1y ago

I have run ELK, Grafana + Prom, Grafana + Thanos/Coretex, New relic and all of the more traditional products for monitoring/observability. More recently in the last few years, I have been running full observability stacks via either The Grafana LGTM stack or datadog at a reasonable scale and complexities. Ultimately you want one tool that can alert you off a metric, present you some traces, and drill down into logs, all the way down the stack. I have found Datadog to be, by far hands down the best developer experience from the get go, the way it glues the mostly decent products together is unparalleled in comparison to other products (Grafana cloud/LGTM). I usually say if your at a small to medium scale business just makes sense, IF you understand the product and configure it correctly which is reasonably easy. The seamless integration between tracing, logging and metrics in the platform, which you can then easily combine with alerts is great. However, its easy to misconfigure it and spend a lot of money on seemingly nothing. If you do not implement tracing and structured logs (at the right volume and level) with trace/span ids etc all the way through services its hard to see the value, and seems expensive. It requires some good knowledge, and configuration of the product to make it pay off. The rest of the product features are generally good, for example their security suite is a good entry level to cloud security monitoring and SEIM too. However, when you get to a certain scale, the cost of APM and Infrastructure hosts in Datadog can become become somewhat prohibitive. Also, Datadogs custom metrics pricing is somewhat expensive and its query language cababilities does not quite match the power of promql, and you start to find yourself needed them to debug issues. At that point, the self hosted LGTM stack starts to make sense, however, it involves a lot more education for end users in both integration (a little less now Otel is popular) and querying/building dashboards etc, but also running it yourself. The grafana cloud platform is more attractive though.

SOLAR_FIELDS 1y ago
My experience mirrors yours wrt Datadog. It's incredible value at low scale, you get a full robust system with great devex for pennies. Once you hit that tipping point though, you are locked in pretty hardcore. Datadog snakes its way far into your codebase, with all the custom tracing and stuff like that. Migrating off of it is a very expensive endeavor, which is probably one of the reasons why they are such a money printing operation.

wenbin 1y ago

Earlier this year, we at Listen Notes switched to Better Stack [0], replacing both Datadog and PagerDuty, and we couldn’t be happier :) Datadog offers a rich set of features, and as a public company, it makes sense for them to keep expanding their product and pushing larger contracts. But as a small team, we don't have a strong demand for constant new features. By switching to Better Stack, we were able to cut our monitoring and alerting costs by 90%, with basically the same things that we used from Datadog previously. [0] https://www.listennotes.com/blog/use-betterstack-to-replace-pagerduty-datadog-and-112/ https://www.listennotes.com/blog/use-betterstack-to-replace-...

iwovtb 1y ago

Observability spend is super expensive. Realizing this, we built grepr.ai.... reduces 96% spend without change. Check it out... Not just a dumb pipeline either.

iwovtb 1y ago

Observability is expensive. Rip/replace is hard... we built grepr.ai to solve this problem and are seeing 96% reduction in spend from Splunk/Sumo/New Relic/Datadog etc. No change mgt. The result set is: 96% reduction in spend and noise elimination. Pretty compelling.. come check us out. www.grepr.ai

MehdiHK 1y ago
No pricing page? Also how does it compare to Cribl?

east4ming 1y ago

Think the other way around, engineers are also a cost, and investment in open-source software is also a cost. If engineers are very cheap, then the company would choose to build it themselves, right?