8 ms·

Our renewal bill for Datadog came to –$83,000/year before we canceled

47 points by lopkeny12ko 2y ago

andrewmcwatters 2y ago
Ah, but you did pay it.
- Dylan16807 2y ago
  Once.
commandlinefan 2y ago
I gotta hand it DataDog's sales team - it's the biggest combination of expensive and useless I've ever seen in my life, but they've somehow managed to convince the people with the checkbooks that they can't live without it.
- linotype 2y ago
  I use it every day. I’m curious what you’re building that it’s useless for?
- lijok 2y ago
  My understanding is that DD is widely regarded as one of the best observability platforms if you can stomach the bill. Could you elaborate on the comment that DD is "useless"? Would love to understand where it comes short compared to other platforms.
  - philstephenson 2y ago
    He doesn't know what he's talking about. Don't feed the trolls.
  - HeatrayEnjoyer 2y ago
    What even is an "observability platform" ? A cursory reading has not enlightened me.
    - lijok 2y ago
      Just a suite of tools to aggregate and action telemetry.
    - meiraleal 2y ago
      [flagged]
    - fragmede 2y ago
      the whole pipeline: from an SDK to add metrics collection to your program, to an agent running on machines, to a collector for those stats, to a dashboard with pretty graphs you can use, sliced up in meaningful ways, and to monitor/troubleshoot problems when they arise, to alerting, so you don't have to worry about the system not running when you're not looking at it.
    - commandlinefan 2y ago
      > What even is an "observability platform" ? $83,000/year to store your log files for you (you have to do the work of storing the log files).
  - stackskipton 2y ago
    IMO, I wouldn't call it useless, it's "useless for most teams". I've been at many companies where they roll out Datadog and collect a crap ton of metrics/logs and so forth. Teams act all gung ho about "being data driven team" but quickly learn, almost all the numbers don't matter. As SRE, I attempt to drill down but any observations are met with silence or "Oh yea, that, meh, toss another 2 cores at the problem and let's move on." At the cost of Datadog, their cost rarely exceeds outage cost or higher cloud bill cost. If you think you need it, buy it for a year where you can probably get good discount rate, use the data to rapidly improve most glaring issues you have then cancel the contract until you need it again in 3 years. Obviously, if you are in highly responsive business, none of this applies but for us just cranking out B2B Rest APIs, you will be fine without Datadog.
  - yen223 2y ago
    Having had to use many different logging/observability platforms, I have to say Datadog is one of the better ones I've used. Biggest strength is that it is user-friendly. I had very little trouble finding logs I care about, and to set up monitors + dashboards.
- anbotero 2y ago
  What I’ve seen happen is that people pay Datadog and, for some reason or another, they: 1. Cannot really get the whole team on Datadog onboarding (Development too, not just “Infra”), so there are these really deep gaps that will never, ever be covered. 2. Don’t use all the services they pay for: “Oh, these CI/CD observability pipelines thing that costs us half our infrastructure? Yeah, I suppose they run fine.”, “Oh, these Profile Hosts? Yeah, just in case we use them, although it was 11 months the last time we did use them...” 3. Pay in excess for services they don’t take advantage of because of #1. 4. Cannot allocate time to follow through with their Account Manager and apply the spending suggestions they sometimes make. “Hey, you’re spending tons on Profile Hosts, but I don’t see usage. Have you looked into that? Oh, yeah, some day we will...” I would definitely not call them useless, but gosh they are expensive for the minimum of things, which I agree if you cannot commit to the points above, you might as well setup your own Prometheus + Grafana + Loki + Mimir + Tempo stack and see what you can come up with, or even try one of the other alternatives I’ve seen around, like SigNoz[1]. ----- [1]: https://signoz.io/ https://signoz.io/
  - meiraleal 2y ago
    What you described is the work of a scamming sales team, nothing less. Blame the right people, could you?
    - anbotero 2y ago
      Who’s the scamming Sales team? Datadog’s? I’ve seen it happen first hand where our side promises to be on board, and we start cancelling meetings with them, don’t follow Datadog training, report we don’t have time this week on Datadog meetings, etc. All after having paid for it. I don’t see how that’s Datadog’s fault, but I’m eager to understand it better.
      - meiraleal 2y ago
        You just described that someone/some team sold something that doesn't meet the customer's need. If you can't see what happened, I can try to explain it for you: Someone from Datadog sales team talks to someone from marketing at Corporation X. Corp. X Marketing loves what Datadog salespeople sell and what amazing things they can get from it and, guess what? pay the first bill. Great job, team! Now is when engagement with the technical team of corp X starts but the technical team that wasn't the one that agreed with the bill. Tech-savvy people knows that Datadog isn't worth it (that's why they aren't the ones that salespeople reach to). Now they have to justify a $83.000/year kinda bill they didn't sign up for. Great salespeople, great sales team. Scammers, if you ask me. As someone from marketing, you've just signed the bill and are essentially the one who "colluded" with Datadog to take money out of Corp X's deep pockets, without fully understanding the technical implications or value of the product.
t-writescode 2y ago
“… and Kibana for metrics logging” I love the ELK stack. Very much; but last I worked with it at a well-oiled company 7ish(?) years ago, we had one full time employee’s worth of time dedicated to keeping it stable, not to mention the infra costs. Was this really a savings when you now you have to manage your own SLA and have an on-call rotation for it?
- jeffwask 2y ago
  This right here is why you pay. I've alternatively walked into places where no one was responsible for updating and managing these tools and they degrade into yet another problem. It's still cheaper and more efficient.
lijok 2y ago
DHH you shouldn't complain - you invented this pricing model !
droopyEyelids 2y ago
Its a weird glimpse into DHH’s psychology that he writes such a bombastic take on what is an extremely nuanced decision. If it has any point, it must be to brag about how elite their ops team is to build and maintain obs infrastructure with less than 1/3 of a headcount. THAT would make for interesting reading.
- deleted 2y ago
  [deleted]
mannyv 2y ago
DD was great at the beginning, but over time they moved to value-based pricing. It's sad but understandable that all the good monitoring tools move up the food chain.
- anbotero 2y ago
  I saw the fall of NewRelic when they started copying DD pricing practices, but to be honest, it was all the startup ecosystem at the time that started doing stuff like that.
bosch_mind 2y ago
Why not self host grafana and Loki? It’s not hard and the cost for redundancy and backups running on tons of nodes with fast internet isn’t anything new. Much cheaper
- t-writescode 2y ago
  My experience is with Kibana and Honeycomb; but in my experience, Grafana, even with Tempo, pales in comparison to the power that the ELK stack or a robust, clean UI’d data aggregator, like Honeycomb is capable of.
  - bosch_mind 2y ago
    I guess it’s a cost benefit analysis. If the UI of X is worth $50k, that’s something some companies can say ok to. Moreso speaking for the companies that feel hopeless when the UIs of these open source tools might probably be ok
ctvo 2y ago
There's this cycle of posts that are _When other people do things for you, it's expensive_ Look at these outrageous AWS prices. Look at these outrageous Datadog prices. Look at these outrageous Snowflake prices... And then of course, they lead to _Look how much cheaper my homegrown solution is_ It's a little tiring. You make trade-offs all the time. If the price isn't worth it, don't pay for it.
- Dylan16807 2y ago
  When almost all the money is going to the software and setup, it sure shouldn't cost less to do it yourself. It takes money to make a robust solution, plus healthy profit, but that money is spread across every customer. Renting equipment is an entirely different issue, but I get the impression that the equipment cost is a pretty negligible percent of the number being discussed.
  - techostritch 2y ago
    > it sure shouldn't cost less to do it yourself. This is an interesting point and something I often wonder about. I think there's a number of different reasons for it, obviously markup and the need to make a profit, but one thing I realized is that when you buy something like AWS, you're often paying for the platinum version, with redundant power, networking, etc, and not the bronze version, which a lot of people would settle for. If I setup my one single instance Grafana VM or physical server, it's going to be a lot less expensive than a solution that's both multiple layers of "platinum" and multiple layers of "markup" Datadog probably runs on the cloud, which means you get the cloud markup and datadog's mark up both built in.
    - ctvo 2y ago
      > I realized is that when you buy something like AWS, you're often paying for the platinum version, with redundant power, networking, etc, and not the bronze version, which a lot of people would settle for. This is the difference in most cases. Just taking availability as an example. Datadog has a 99.8% availability SLA. I suspect most companies would be fine paying for a 99% availability SLA, for example, at 1/4th the cost, but in most cases there's no way to scale down SLAs like that and launch a cheaper version. There is a giant leap in cost (complexity, staffing, etc.) between 99% and 99.9% that most of these home grown solutions don't account for. These nuances are lost in posts like these, and it is tiring.
pranabgohain 2y ago
Time to make the move to OTel with an Observability backend like https://www.kloudmate.com https://www.kloudmate.com (or any other APM of choice) and get most of the stuff done for a fraction. The big Dog has dictated terms for far too long.
- ashconnor 2y ago
  You should disclose that this is your company.
  - pranabgohain 2y ago
    Done. Thanks!
- pranabgohain 2y ago
  PS: I am associated with KloudMate
toomanyrichies 2y ago
The title is misleading; it says -$83,000/year when it should say ~$83,000/year, as in DHH's post. Those mean very different things.
- altairprime 2y ago
  Email the mods and they’ll correct it?
bastawhiz 2y ago
Excuse my for rolling my eyes at DHH complaining about the price of software. It's enterprise software. You saw the price when you signed up. DD isn't cheap. Neither are the direct competitors (have you looked at Splunk licenses lately?). Why is the cost so high? Because it's easy to add logging in places it isn't needed or useful. Every place I've ever worked has been able to save costs on these licenses by taking a pass at cleaning up their logs. If you pay by volume and you're logging things that aren't useful and never ever ever get read, you're burning cash. And if you think you're not doing this, I promise you probably are. DD (and others') costs can be reasonable if you don't just spray log data at it indiscriminately. Want it for cheap? Cloudwatch is pennies compared to DD. It's not good, but it works. Want something faster or more featureful? You can stand up ElasticSearch or your favorite log aggregation software, and then you're probably paying a substantial percentage of what you paid for DD in server costs and time required to keep the thing plugging along. And then whatever time you spend dealing with "Grafana started timing out and we don't know why" and similar flavors of incidents. Yes, good software costs about as much as a FTE. Because you'll (hopefully!) save about a FTE's worth of value by using the tool.
UncleGusO11Y 2y ago
I found that "observability platforms" like Datadog, New Relic etc can be really expensive and not that valuable unless you place a centralized "real-time observability pipeline" that's data agonistic between the originating telemetry data sources and the end destination like Datadog, New Relic, Looker, etc. This approach allows you to see the data stream in real-time and then you can reduce, enrich, transform the data and then route it to any end destination like Datadog, New Relic, etc. We did this and saw some pretty significant saving from the observability platform vendors we are using. Not only that the fact we were able to enrich the data made our troubleshooting much faster because we had data ingested now that we can make sense of and act on. We used datable.io who is a startup out of SF accomplish this. We talked with them about a month ago and their service was still in beta but they are allowing us to use it free of charge for now (not sure if they are still doing that or not?). I think the founder is an Ex New Relic guy?
- starkparker 2y ago
  There's a class of control plane startups that specialize in shaping/dropping observability data points (Honeycomb, Signoz, ChronoSphere) to both focus the results and reduce ingest/storage costs. OTel can do this independent of providers at the collector, for better (all downstream tooling benefits) and worse (with considerable fussiness and bespoke config).
PeterZaitsev 2y ago
Datadog is New Oracle and Observability is ripe for Open Source Disruption. Great to see projects like SigNoz, NetData, Coroot appearing and Open Telemetry ecosystem getting more and more usable Opens Source backends and visualization tools not just agents.
- pranay01 2y ago
  Thanks for mentioning SigNoz. Totally agree that OpenTelemetry and open source should be the way forward here. If anyone wants to check out SigNoz repo - https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz PS: I am one of the maintainers at SigNoz
lebesguem 2y ago
"The man who needs a new machine tool, and hasn't bought it, is already paying for it." Agree though ballooning cost will prove to be Datadog's Achilles heel. But $83k/year is a bargain no? Where do you even find an engineer working less than $150k/year who could provide the same level of service as Datadog?

47 points by lopkeny12ko 2y ago

andrewmcwatters 2y ago

Ah, but you did pay it.

Dylan16807 2y ago
Once.

commandlinefan 2y ago

I gotta hand it DataDog's sales team - it's the biggest combination of expensive and useless I've ever seen in my life, but they've somehow managed to convince the people with the checkbooks that they can't live without it.

linotype 2y ago
I use it every day. I’m curious what you’re building that it’s useless for?
lijok 2y ago
My understanding is that DD is widely regarded as one of the best observability platforms if you can stomach the bill. Could you elaborate on the comment that DD is "useless"? Would love to understand where it comes short compared to other platforms.
- philstephenson 2y ago
  He doesn't know what he's talking about. Don't feed the trolls.
- HeatrayEnjoyer 2y ago
  What even is an "observability platform" ? A cursory reading has not enlightened me.
  - lijok 2y ago
    Just a suite of tools to aggregate and action telemetry.
  - meiraleal 2y ago
    [flagged]
  - fragmede 2y ago
    the whole pipeline: from an SDK to add metrics collection to your program, to an agent running on machines, to a collector for those stats, to a dashboard with pretty graphs you can use, sliced up in meaningful ways, and to monitor/troubleshoot problems when they arise, to alerting, so you don't have to worry about the system not running when you're not looking at it.
  - commandlinefan 2y ago
    > What even is an "observability platform" ? $83,000/year to store your log files for you (you have to do the work of storing the log files).
- stackskipton 2y ago
  IMO, I wouldn't call it useless, it's "useless for most teams". I've been at many companies where they roll out Datadog and collect a crap ton of metrics/logs and so forth. Teams act all gung ho about "being data driven team" but quickly learn, almost all the numbers don't matter. As SRE, I attempt to drill down but any observations are met with silence or "Oh yea, that, meh, toss another 2 cores at the problem and let's move on." At the cost of Datadog, their cost rarely exceeds outage cost or higher cloud bill cost. If you think you need it, buy it for a year where you can probably get good discount rate, use the data to rapidly improve most glaring issues you have then cancel the contract until you need it again in 3 years. Obviously, if you are in highly responsive business, none of this applies but for us just cranking out B2B Rest APIs, you will be fine without Datadog.
- yen223 2y ago
  Having had to use many different logging/observability platforms, I have to say Datadog is one of the better ones I've used. Biggest strength is that it is user-friendly. I had very little trouble finding logs I care about, and to set up monitors + dashboards.
anbotero 2y ago
What I’ve seen happen is that people pay Datadog and, for some reason or another, they: 1. Cannot really get the whole team on Datadog onboarding (Development too, not just “Infra”), so there are these really deep gaps that will never, ever be covered. 2. Don’t use all the services they pay for: “Oh, these CI/CD observability pipelines thing that costs us half our infrastructure? Yeah, I suppose they run fine.”, “Oh, these Profile Hosts? Yeah, just in case we use them, although it was 11 months the last time we did use them...” 3. Pay in excess for services they don’t take advantage of because of #1. 4. Cannot allocate time to follow through with their Account Manager and apply the spending suggestions they sometimes make. “Hey, you’re spending tons on Profile Hosts, but I don’t see usage. Have you looked into that? Oh, yeah, some day we will...” I would definitely not call them useless, but gosh they are expensive for the minimum of things, which I agree if you cannot commit to the points above, you might as well setup your own Prometheus + Grafana + Loki + Mimir + Tempo stack and see what you can come up with, or even try one of the other alternatives I’ve seen around, like SigNoz[1]. ----- [1]: https://signoz.io/ https://signoz.io/
- meiraleal 2y ago
  What you described is the work of a scamming sales team, nothing less. Blame the right people, could you?
  - anbotero 2y ago
    Who’s the scamming Sales team? Datadog’s? I’ve seen it happen first hand where our side promises to be on board, and we start cancelling meetings with them, don’t follow Datadog training, report we don’t have time this week on Datadog meetings, etc. All after having paid for it. I don’t see how that’s Datadog’s fault, but I’m eager to understand it better.
    - meiraleal 2y ago
      You just described that someone/some team sold something that doesn't meet the customer's need. If you can't see what happened, I can try to explain it for you: Someone from Datadog sales team talks to someone from marketing at Corporation X. Corp. X Marketing loves what Datadog salespeople sell and what amazing things they can get from it and, guess what? pay the first bill. Great job, team! Now is when engagement with the technical team of corp X starts but the technical team that wasn't the one that agreed with the bill. Tech-savvy people knows that Datadog isn't worth it (that's why they aren't the ones that salespeople reach to). Now they have to justify a $83.000/year kinda bill they didn't sign up for. Great salespeople, great sales team. Scammers, if you ask me. As someone from marketing, you've just signed the bill and are essentially the one who "colluded" with Datadog to take money out of Corp X's deep pockets, without fully understanding the technical implications or value of the product.

t-writescode 2y ago

“… and Kibana for metrics logging” I love the ELK stack. Very much; but last I worked with it at a well-oiled company 7ish(?) years ago, we had one full time employee’s worth of time dedicated to keeping it stable, not to mention the infra costs. Was this really a savings when you now you have to manage your own SLA and have an on-call rotation for it?

jeffwask 2y ago
This right here is why you pay. I've alternatively walked into places where no one was responsible for updating and managing these tools and they degrade into yet another problem. It's still cheaper and more efficient.

lijok 2y ago

DHH you shouldn't complain - you invented this pricing model !

droopyEyelids 2y ago

Its a weird glimpse into DHH’s psychology that he writes such a bombastic take on what is an extremely nuanced decision. If it has any point, it must be to brag about how elite their ops team is to build and maintain obs infrastructure with less than 1/3 of a headcount. THAT would make for interesting reading.

deleted 2y ago
[deleted]

mannyv 2y ago

DD was great at the beginning, but over time they moved to value-based pricing. It's sad but understandable that all the good monitoring tools move up the food chain.

anbotero 2y ago
I saw the fall of NewRelic when they started copying DD pricing practices, but to be honest, it was all the startup ecosystem at the time that started doing stuff like that.

bosch_mind 2y ago

Why not self host grafana and Loki? It’s not hard and the cost for redundancy and backups running on tons of nodes with fast internet isn’t anything new. Much cheaper

t-writescode 2y ago
My experience is with Kibana and Honeycomb; but in my experience, Grafana, even with Tempo, pales in comparison to the power that the ELK stack or a robust, clean UI’d data aggregator, like Honeycomb is capable of.
- bosch_mind 2y ago
  I guess it’s a cost benefit analysis. If the UI of X is worth $50k, that’s something some companies can say ok to. Moreso speaking for the companies that feel hopeless when the UIs of these open source tools might probably be ok

ctvo 2y ago

There's this cycle of posts that are _When other people do things for you, it's expensive_ Look at these outrageous AWS prices. Look at these outrageous Datadog prices. Look at these outrageous Snowflake prices... And then of course, they lead to _Look how much cheaper my homegrown solution is_ It's a little tiring. You make trade-offs all the time. If the price isn't worth it, don't pay for it.

Dylan16807 2y ago
When almost all the money is going to the software and setup, it sure shouldn't cost less to do it yourself. It takes money to make a robust solution, plus healthy profit, but that money is spread across every customer. Renting equipment is an entirely different issue, but I get the impression that the equipment cost is a pretty negligible percent of the number being discussed.
- techostritch 2y ago
  > it sure shouldn't cost less to do it yourself. This is an interesting point and something I often wonder about. I think there's a number of different reasons for it, obviously markup and the need to make a profit, but one thing I realized is that when you buy something like AWS, you're often paying for the platinum version, with redundant power, networking, etc, and not the bronze version, which a lot of people would settle for. If I setup my one single instance Grafana VM or physical server, it's going to be a lot less expensive than a solution that's both multiple layers of "platinum" and multiple layers of "markup" Datadog probably runs on the cloud, which means you get the cloud markup and datadog's mark up both built in.
  - ctvo 2y ago
    > I realized is that when you buy something like AWS, you're often paying for the platinum version, with redundant power, networking, etc, and not the bronze version, which a lot of people would settle for. This is the difference in most cases. Just taking availability as an example. Datadog has a 99.8% availability SLA. I suspect most companies would be fine paying for a 99% availability SLA, for example, at 1/4th the cost, but in most cases there's no way to scale down SLAs like that and launch a cheaper version. There is a giant leap in cost (complexity, staffing, etc.) between 99% and 99.9% that most of these home grown solutions don't account for. These nuances are lost in posts like these, and it is tiring.

pranabgohain 2y ago

Time to make the move to OTel with an Observability backend like https://www.kloudmate.com https://www.kloudmate.com (or any other APM of choice) and get most of the stuff done for a fraction. The big Dog has dictated terms for far too long.

ashconnor 2y ago
You should disclose that this is your company.
- pranabgohain 2y ago
  Done. Thanks!
pranabgohain 2y ago
PS: I am associated with KloudMate

toomanyrichies 2y ago

The title is misleading; it says -$83,000/year when it should say ~$83,000/year, as in DHH's post. Those mean very different things.

altairprime 2y ago
Email the mods and they’ll correct it?

bastawhiz 2y ago

Excuse my for rolling my eyes at DHH complaining about the price of software. It's enterprise software. You saw the price when you signed up. DD isn't cheap. Neither are the direct competitors (have you looked at Splunk licenses lately?). Why is the cost so high? Because it's easy to add logging in places it isn't needed or useful. Every place I've ever worked has been able to save costs on these licenses by taking a pass at cleaning up their logs. If you pay by volume and you're logging things that aren't useful and never ever ever get read, you're burning cash. And if you think you're not doing this, I promise you probably are. DD (and others') costs can be reasonable if you don't just spray log data at it indiscriminately. Want it for cheap? Cloudwatch is pennies compared to DD. It's not good, but it works. Want something faster or more featureful? You can stand up ElasticSearch or your favorite log aggregation software, and then you're probably paying a substantial percentage of what you paid for DD in server costs and time required to keep the thing plugging along. And then whatever time you spend dealing with "Grafana started timing out and we don't know why" and similar flavors of incidents. Yes, good software costs about as much as a FTE. Because you'll (hopefully!) save about a FTE's worth of value by using the tool.

UncleGusO11Y 2y ago

I found that "observability platforms" like Datadog, New Relic etc can be really expensive and not that valuable unless you place a centralized "real-time observability pipeline" that's data agonistic between the originating telemetry data sources and the end destination like Datadog, New Relic, Looker, etc. This approach allows you to see the data stream in real-time and then you can reduce, enrich, transform the data and then route it to any end destination like Datadog, New Relic, etc. We did this and saw some pretty significant saving from the observability platform vendors we are using. Not only that the fact we were able to enrich the data made our troubleshooting much faster because we had data ingested now that we can make sense of and act on. We used datable.io who is a startup out of SF accomplish this. We talked with them about a month ago and their service was still in beta but they are allowing us to use it free of charge for now (not sure if they are still doing that or not?). I think the founder is an Ex New Relic guy?

starkparker 2y ago
There's a class of control plane startups that specialize in shaping/dropping observability data points (Honeycomb, Signoz, ChronoSphere) to both focus the results and reduce ingest/storage costs. OTel can do this independent of providers at the collector, for better (all downstream tooling benefits) and worse (with considerable fussiness and bespoke config).

PeterZaitsev 2y ago

Datadog is New Oracle and Observability is ripe for Open Source Disruption. Great to see projects like SigNoz, NetData, Coroot appearing and Open Telemetry ecosystem getting more and more usable Opens Source backends and visualization tools not just agents.

pranay01 2y ago
Thanks for mentioning SigNoz. Totally agree that OpenTelemetry and open source should be the way forward here. If anyone wants to check out SigNoz repo - https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz PS: I am one of the maintainers at SigNoz

lebesguem 2y ago

"The man who needs a new machine tool, and hasn't bought it, is already paying for it." Agree though ballooning cost will prove to be Datadog's Achilles heel. But $83k/year is a bargain no? Where do you even find an engineer working less than $150k/year who could provide the same level of service as Datadog?