13 ms·
A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call
- panda888888 3y agoWhy does the tweet author think this is wild? Sure, 65 million is a lot, but plenty of companies pay large bills for their major cloud services (especially AWS/Azure and the like). I'm not an expert on monitoring/observability/telemetry etc, nor an expert on Datadog pricing/billing, but paying a lot of money for major infrastructure components doesn't surprise me.
- Rapzid 3y agoImagine you had 65 great engineers to build out your companies observability infra. They can use and contribute to OSS tools like Loki and Prometheus. They can split off small teams to build brand new infra and tools. This is THIRTEEN teams of 5 engineers! You give them 1m each. You never see them again.
- paigerduty 3y ago(Disclaimer: I work at Chronosphere) Our company helps avoid these kinds of observability bills and issues like scaling for fast-growing cloud deployments. Generally speaking, many vendors let you fall into the cardinality trap b/c they have an economic incentive to let you do so. One of our biggest selling points is that we provide an observability control plane that helps drill down into wasted queries, shows how metrics can be aggregated, and other ways of avoiding wasted cost/effort. tbh no one should have to pay more to observe a service than to operate it. Where’s the ROI in that? Another plus is that we're all in on open source instrumentation with OpenTelemetry & Prometheus so none of that annoying vendor lock-in.
- jiggawatts 3y agoNone of this should be even remotely necessary. It’s like being frugal with table salt. “We’ll show you how to make sure you don’t have even one crystal fall off the plate.” My personal pet peeve is Azure Application Insights which uses Log Analytics under the hood… at a rate of $2.75 per ingested GB of logs stored for one month. That’s highway robbery. Let that sink in: They charge $2,800 to store a TB of text that takes a few hundred dollars of overpriced cloud disk and maybe $10 of CPU time for the actual processing. That’s the cost of a serviceable used car or a brand new gaming PC! But wait! There’s more. In reality that 1 TB is column compressed down to maybe 100 MB, making it about $30K charged per terabyte stored on disk. It doesn’t stop there! Thanks to misaligned incentives, the ingested data format is fantastically inefficient JSON that re-sends static values for every metric sample collected. Why would anyone ever bother to optimise their only revenue?! They won’t. The reality is that a numeric metric collected once a second (not minute!) is just 21 MB if stored as a simple array. Most metrics are highly compressible, and that would easily pack to 100 KB per metric per month. A typical Windows server has about 15,000 performance metrics. We could be collecting these once a second and use a grand total of… 1.5 GB per month. That’s every metric for every process, every device, every error counter, everything. Modern server monitoring is inefficient and overpriced by 5 orders of magnitude. It’s that simple. That fact that your company can exist at all is a testament to that.
- paigerduty 3y ago"They charge $2,800 to store a TB" !!! good god Totally agree about the compressibility of metrics and toying with the scraping interval. I started out working for an enterprise monitoring vendor that had a proprietary agent that already decided sane intervals to emit metrics, when I learned that Prometheus let users configure that to me...just sounds like an expensive foot gun. My real beef with metrics is at least for app layer insights is the waste. I'd so much rather have a span/event configured with tail sampling so you can derive metrics from traces and tie them to logs in a native contextualized way vs having to do that correlation on the backend and within different systems and query langs. Seems much more efficient and cost-effective that way, I'm scarred from seeing a zillion "service_name.http_response.p95.average" metrics that are imo useless
- jiggawatts 3y ago> for app layer insights is the waste. I’m starting to come to the same conclusion, but the point I’m making is a general one: efficient formats would allow finer grained telemetry to be collected without having to be tuned and carefully monitored. What’s the point of a monitoring system that itself needs baby sitting?!
- pxeger1 3y ago"it" = Datadog
- legohead 3y agoDatadog prices are out of this world. I assumed their front facing prices were for the lone dev, so when I reached out as a company looking to integrate multiple of their products I expected a deal, and got very little compromise. I told them up front they were going to have to cut prices by 90% for us to consider them -- no budge. And they are pretty belligerent salesman, not wanting to leave me alone. After a while I just blocked their domain to my email.
- bunnyfoofoo 3y agoTheir sales team is a nightmare, I had to block them on personal and company email.
- bena 3y ago[flagged]
- nicenewtemp84 3y agoIve worked in sales for over 10 years. Incel is a weird insult for salespeople, as I've never met a group of people that sleeps around more than salespeople. I spent years selling cars even, and if you bring a gf or wife you're having trouble with to go car shopping... Odds of her later sleeping with the car salesman are not zero to put it mildly.
- torbTurret 3y agoNowadays, incel is synonymous with the Andrew Tate “top g”/alpha sex obsession. Backwards? Definitely, but more symbolic of the arrested development/lack of maturity than the lack of sex. Sleeping with others’ significant others, or actively trying to/bragging about it is a perfect example. Low-class behavior that will get you props from low-class folks.
- TOMDM 3y agoI've certainly heard that fans of Andrew Tate are incels, or that their world view and advice is built for incels, but I've not heard of many people calling Andrew Tate himself an incel. An incel maker though for sure.
- aftbit 3y agoWhy do we think this was Coinbase?
- swyx 3y agovery few other crypto companies capable of paying that much. god knows ftx wasnt investing in observability
- droopyEyelids 3y agoNot on purpose, but I could imagine someone at FTX signing up for datadog, configuring it to ingest their logs without doing any estimation or setting up any guardrails and then not checking on it because things were probably crazy over there.
- bmurray7jhu 3y agoDatadog's SEC filings indicated otherwise. Since 2022 Q1, the allowance for doubtful accounts has remained under $6 million. Bankruptcy generally triggers a writedown of any associated receivables, so Datadog appears to not have had any material exposure to FTX or any other bankrupt customer.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- Scoundreller 3y agoHaving never heard of datadog, Wikipedia’s summary is: > Datadog is an observability service for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform. So it checks if your servers have crashed or slowed down with a nice dashboard? Any better summaries or descriptions of what it does and how coinbase would have used it?
- eli 3y agoit's a New Relic competitor
- bdcravens 3y agoThey also do client side monitoring, log aggregation, and provide metrics for your overall operations (not just when there are performance issues). They also have quite an unethical sales operation: https://news.ycombinator.com/item?id=35837965 https://news.ycombinator.com/item?id=35837965
- kingsloi 3y agoThat's funny, DD is the only company to email my work email, add me as a connection on LinkedIn, txt me over WhatApp, and call my personal phone number multiple times. Amber flag after the LinkedIn connection/message, but red/purple after the WhatsApp/call on my personal phone.
- jabroni_salad 3y agoit ingests and can help you sort through logs, and also does performance monitoring. I guess I would describe it as prometheus + logstash + powerBI in a single unit. And with a healthy dose of blue cross bolted on for the surprise bills and difficult bureaucracy.
- hdjjhhvvhga 3y agoSuppose you have a bunch of k8s clusters, an AWS Organization etc. You just follow a simple setup and see a nice Dashboard with practically every aspect of your infrastructure, from accounts to nodes to pods.
- draw_down 3y agoA previous workplace switched from this to a competitor, which had much worse graphing as far as I could tell. Seems like an important function for these tools! I wondered why they did that, I guess this is the answer.
- ren_engineer 3y agoOpenTelemetry is going to be an existential threat to DataDog and other companies that effectively rely on vendor lock-in to exploit customers. Not sure how companies rationalize these types of services at scale when there are so many open source options to run for a fraction of the cost. You could hire 100+ engineers and still save money compared to a 65M bill https://news.ycombinator.com/item?id=34540419 https://news.ycombinator.com/item?id=34540419
- ricardobeat 3y agoI’ve seen an attempted move from DD to OT and it was a nightmare of undocumented features and little compounded issues. Tracing was non functional. It doesn’t seem mature enough yet.
- icelancer 3y agoWere you watching me? Because that's what happened. I am a user of New Relic. Not because I'm happy. But because OpenTelemetry doesn't come close to the same features. Fortunately, at least OT is about 10x harder to set up with worse documentation. Wait a minute...
- VirusNewbie 3y agoyup at my last company CTO wanted to switch away from New Relic to save money for Open Telemetry and I can tell you we wasted months of work because OpenTelemetry is dogshit. New Relic is really really good too, so it was even more painful.
- icelancer 3y agoYeah, New Relic is just an excellent product. No way around it.
- hosh 3y agoWe had to use otel with Datadog because Datadog did not support Elixir with an official SDK. There are a lot of subtle issues, but we've been able to work through them to get usable traces. (Metrics and log ingestion already have pretty good existing open-source tooling, like statsd).
- n0us 3y agoHaving recently dealt with a surprise runaway DataDog bill and personally completed lots of work to reduce our spend internally all I have to say is that their pricing is outrageously high and there is almost no way to put controls in place to prevent overspend. If you're considering using them keep that in mind and tbh I would strongly recommend considering if CloudWatch or some other cheaper alternative is suitable for your needs.
- jgalt212 3y agoThe internet is mostly machines talking other machines or monitoring what other machines are monitoring. But then you can also pay for machines to watch the machines that watch the machines.
- nix0n 3y agoWe need one more layer, for machines that watch how much you're spending on the machines to watch the machines that watch the machines.
- victorbjorklund 3y agoDamn. Why not self-host something on that scale?!
- ericmcer 3y agoYeah seriously, normally the argument is: 'it will require N engineers to run our own and they cost N * 250k/yr...' but for 65million you could fund 5 Datadog competitors and still come out ahead.
- coinbasethrow 3y agoThey did build a self hosted alternative based on grafana, Loki, Prometheus Had a whole team of 10+ engineers working on it for 2 quarters, then scrapped it because it performed terribly The only thing that came of it was negotiation leverage with datadog ("give us X% off or we go self hosted")
- Arbortheus 3y agoWhat kind of scale of logs are we talking here? The company I work for run a self-hosted Grafana LGTM stack ingesting about 1TB of logs per day, it’s pretty snappy and works well enough, and only costs a few thousand dollars per month in GKE costs for the entire observability stack. How much logging are we talking here?
- grepfru_it 3y agoGitHub has over 21TB of source code. Applications consistently pour through this data and emit logs and events. 1TB of data by breakfast maybe? In reality, we're not pushing logs to datadog, just metrics and event tags. Our level of cardinality, however, requires a lot of horsepower on the backend. Our attempted Prometheus transition was just not cost effective when attempting to view large sets of data over a large-ish period of time. Combined with the heavy lift of integration (we depended heavily on dogstatsd) it just didn't seem efficient to move to Prometheus, support the infrastructure required, all while migrating to microsoft's inhouse product.
- deleted 3y ago
- karaterobot 3y agoIs the speculation that this was Coinbase just based on Coinbase being a big crypto company? I see nothing in those messages that implies who the customer is (rightly so) and I am wondering if there's some other information I'm missing.
- coinbasethrow 3y agoI worked at Coinbase until very recently and can confirm this is Coinbase They paid upfront for 3 years of usage, and yes they were burning > $20m/year on datadog
- skippyboxedhero 3y agoDear Lord...the commission on that hog. The sales team eating good.
- austhrow743 3y agoI was previously in sales and SaaS is considered the pinnacle of industries to be in for sales people. Medical device sales is the only real competitor when it comes to earning potential and my understanding is that's US specific and also comes with atrocious work life balance.
- thinkmassive 3y agoSo you’re saying it’s $65 million over three years? The headline seems misleading, like it’s for a single month.
- KaiserPro 3y agoyes but 20 million dollars a year. 20 fucking million getting splunk with 1pb a month ingest isnt that expensive.
- rednerrus 3y agoThey'll probably throw in SignalFX for free if you ask them nicely.
- quickthrower2 3y agoPlease tell me I am allowed to say "you could just use grep and rsync". I must be allowed!
- christkv 3y agoWe use sentry and seems to work well and the way the pricing is structured I don’t expect any runaway costs to happen.
- rocmcd 3y agoSentry is strictly code-first APM, which is only a part of what DD provides. What "APM" _is_ can get kind of blurry, but they are not direct competitors in a meaningful way.
- b33j0r 3y agoUse the crypto to fund the AI that can actually process those logs, then the log analysis can determine how to buy more crypto. It’s the simple mathematics of perpetual motion, as observed in all stable systems.
- polskibus 3y agoHow does Datadog compare against Azure Application Insights?
- pragmatic 3y agoI have limited experience in both. We use Datadog for VM and database monitoring. When I worked at a place that was all in on Azure, application insights was so we needed because we had no dedicated VMs just all built in Azure services (Cosmos, queues, blob/table storage and functions etc)
- firstSpeaker 3y agoHow does this compares with Grafana Cloud prices? Anyone knows any estimate and comparison?
- re-thc 3y agoGrafana Cloud is definitely cheaper especially if you have some volume.
- BryantD 3y agoI’m not gonna claim that Datadog is cheap, but that screams “we didn’t bother to optimize our usage.” Lots of logs, long retention maybe? Really heavy RUM with the replay feature turned on?
- davidw 3y agoI feel nostalgic for the days of grepping logs in some ways. At a previous company, we had DD, and I got asked to find some problem. I was able to sift through the data in it and zero in on the instance and find the bad data that came in. Then it got expensive and so they turned on 'sampling' and the next time I was asked to look for a particular problem, we had no idea if it was even logged.
- ibejoeb 3y agoThat reminds me of the quip in gnu parallel about log search: https://www.gnu.org/software/parallel/parallel_examples.html#bigger-problem https://www.gnu.org/software/parallel/parallel_examples.html...
- willio58 3y agoDatadog prices are insane. We're leaving them now at my work because they've gotten to be more expensive than our entire AWS bill.
- willejs 3y agoDatadog is a pretty amazing product, and if you are careful and use it in the right way, it is very powerful, and cheaper than rolling your own LGTM Grafana stack (or similar). If you are not careful, or at a decent scale, you can easily spend obscene amounts of money. The metrics pricing is completely insane for example, and its easy for people to emit high cardinality metrics from apps and explode your bill. I think by this point you need to run an internal solution, and that is when it makes sense to double down on a combo of elastic, and grafanas stack for logging, tracing and metrics.
- therealmarv 3y agoStupid question: What open source solution can give me an easy Datadog(ish) experience and is simple to implement?
- candiddevmike 3y agoGrafana, Prometheus, Loki, tempo
- grepfru_it 3y agoSooo nothing yet :D We attempted to migrate from datadog to prometheus at GitHub and that stack did not cover our use case at all. So much tooling had to be recreated. I took a lot of flak when I pointed out numbers made sense to stay on DataDog and migrate to a Microsoft product instead, but the cost savings spoke for itself
- candiddevmike 3y agoGrafana isn't incentived to simplify self hosting IMO, so this is the best you can do without paying them.
- paulddraper 3y agoThat's a bit like asking for an "open source alternative to AWS." It depends what you're looking for: metrics, logs, APM, tracing, synthetics, web analytics, etc.
- therealmarv 3y agometrics, APM mostly
- RhodesianHunter 3y agoThe list of targets Micrometer posts to is a good starting point for options: https://micrometer.io/docs https://micrometer.io/docs
- ACV001 3y agothat's not possible
- acapybara 3y ago[dead]
- skrtskrt 3y agoIs anyone using the newer Grafana Cloud Mimir/Tempo/Loki stack that can compare pricing for real usage? I know Datadog probably has a hell of a lot more add-on features, I'm more interested in head to head of comparable produts
- mgillett54 3y agoIt depends a lot on your usage pattern, but we are switching to Granfana cloud from Datadog and are looking at about 1/3 the cost per year. This is using logs, metrics, and traces with OpenTelemetry instrumentation. The Grafana pricing is more cleanly volume based that is disconnected from number of “hosts” which is where datadog really squeezes you in a kubernetes setup with many pods.
- eatmyshorts 3y agoThis doesn't surprise me much. From what I've seen consulting/contracting, SaaS-based observability tends to cost 30-50% of cloud spend--EC2, storage, S3, RDS, maybe k8s, and other cloud services, or whatever the equivalent is on GCP/Azure. I wouldn't be surprised to see Coinbase with a >$150M quarterly cloud spend, so $65M on observability would make sense. That said, managing observability yourself should result in <5% of cloud spend. So I'm figuring someone at Coinbase said "WTF" to this bill and migrated to Grafana/Loki or Kibana/OpenSearch or Kibana/Elastic. Well, that, and Coinbase's business also dropped off a cliff. Combined, I could easily see a one-time influx of $65M from one customer, gone the next quarter.
- pranay01 3y agoFolks on this thread might want to check out SigNoz (https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz). It's an open source alternative to Datadog. I am one of the maintainers at SigNoz. We have come across many more horror stories around Datadog billing while interacting with our users. We recently did a deep dive on pricing, and found some interesting insights on how it is priced compared to other products. Datadog's billing has two key issues: - Very complex SKU based pricing which makes it impossible to predict how much it would cost - Custom metrics billing ($0.05 per custom metric) - we found that custom metrics can account for up to 52% of the total billing which just does not make sense More details in the blog here with a complete spreadsheet for detailed calculation https://signoz.io/blog/pricing-comparison-signoz-vs-datadog-vs-newrelic-vs-grafana/ https://signoz.io/blog/pricing-comparison-signoz-vs-datadog-...
- hnbear 3y agoThis seems like such a failing in both sides. DD for letting an obviously huge and important client run up a bill so crazy they have to quit DD out of shame and governance. CB for demonstrating a complete failure of financial management, vendor management and any sort of ability to track expenses. Doesn’t help either side to get to a point where you have to quit to demonstrate you’re not totally incompetent and crazy.
- davidguetta 3y agoOh that's why they have insane salaries
- abnerg 3y agoGergely Orosz dug into this a bit more and surfaced quite a build your own vs renew vs renew with highly negotiated terms. (https://blog.pragmaticengineer.com/datadog-65m-year-customer-mystery/ https://blog.pragmaticengineer.com/datadog-65m-year-customer...) A lot of this discussion reminds me of this talk:"Netflix built its own monitoring system - and why you probably shouldn't" (https://www.infoq.com/presentations/netflix-monitoring-system/ https://www.infoq.com/presentations/netflix-monitoring-syste...) where Roy Rappport describes Netflix as a "monitoring system that happens to stream movies" As someone who spent a few years at New Relic and Lacework, I can also say that pricing observability fairly is crazy hard when you account for different architectures, usage pricing, and the humans experience the value.
- monero-xmr 3y agoWhat is best cost effective and decent alternative to DD in 2023? I also feel like they are robbing me blind. Great product though.
- stock_toaster 3y agoSelf hosted signoz for opentelemetry? with grafana, prometheus, loki for metrics and logs?
- nikolay_sivko 3y agoTake a look at Coroot: it's open-source, ebpf-powered, and can be integrated in minutes: https://github.com/coroot/coroot https://github.com/coroot/coroot
- m-o11y 3y agoPlug: If you're looking for something a bit more "few-clicks-and-you-are-up-and-running", check out OpsVerse ObserveNow: https://opsverse.io/observenow-observability/ https://opsverse.io/observenow-observability/ .. Entirely powered by OSS tools, ingestion-driven pricing, and without the hassle of managing the stack and scaling up. Best of all, can also be run entirely within your own AWS/GCP/Azure so you only pay OpsVerse for maintaining the stack based on your ingestion (and we also monitor the monitoring system for you ;))
- ankitnayan 3y agoAn OpenSource alternative to DataDog built on ClickHouse database which is also native to OpenTelemetry https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz
- scottfingerhut 3y ago(disclaimer, I'm with Grafana). We added a lot to our free Grafana Cloud so you can kick it pretty hard (and harder during the first 14 days when everything is beefed up). Free tier comes with 3 Grafana front end users fully managed, backend (with storage) 50gb Loki logs, 50gb Tempo traces, 10k monthly active series prom metrics, IRM/on call, k6 user testing hours and other stuff too. And for the quick solution integrations we made a K8s monitoring solution with out of the box dashboards, KPIs and alerts. Same thing we did with many others. We absolutely have more work to do in simplifying the user experience too.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- justrealist 3y agoSounds like some kind of upfront payment discount.
- xeromal 3y agoLooks like it but what cartoon world do we live in where a bill for logging is 65 million.
- ReflectedImage 3y agoCloud
- JumpCrisscross 3y ago> what cartoon world do we live in where a bill for logging is 65 million Perhaps it unlocked insurance?
- selcuka 3y agoIt could be a 65-year upfront payment. /s
- csomar 3y agoThis is crypto. Literally, the cartoon world.
- tayo42 3y agoSeems like there's opportunity for compition in SaaS monitoring then. I'd imagine a few small efficient teams could beat the price on the core part of what datadog offers. At 5000 employees Your probably paying for corporate bloat
- zamnos 3y agoThere are plenty of competitors in the observability space, what's another one? The real issue is once your company is on a platform, it's very costly to move off. The biggest consideration is that it won't be a drop in change for all employees, so the retraining needed is substantial across all teams. Far easier to train employees in cardinality, and the cost implied by it, and to expose the cost for their particular monitoring to their teams. This may come as a surprise but when giving money to a for-profit company, not only are you paying for corporate bloat, but you're also paying for the CEO's lavish compensation package, free lunches, and very costly health insurance for their employees. You're even paying for employee salaries while they're not doing work while on vacation! If that's a big problem for you, Graphana may be the better product for you.
- tayo42 3y agoI don't consider that corporate bloat. That's just life in the software business if you want good talent. Corporate bloat is like the mini empires people build, headcount for the sake of headcount, that guy who's been here forever passion project that doesn't make money. Process because it helped someone's resume. Those kinds of inefficiencies. This stuff is different then treating employees nicely.
- thinkingkong 3y agoThats completely wild. Did coinbase go into trace mode and log every blockchain append to datadog? I cant even figure out how anything could cost that much.
- tornato7 3y agoLikely logging something for every order placed/removed/modified on every order book, which can be thousands per second per order book.
- bdcravens 3y agoWe tried Datadog, and due to how they bill, our DD bill was more than the services being monitored. (At the time, we used a lot of short-term spot instances, which billed by the second, but DD billed by the hour per instance) We reached out to DD to try to work something out, and they agreed to cut the bill in half, but only IF we signed up for additional services. I will never use their services again, and will always share this story when their name is brought up.
- anotherhue 3y agoAfter a sales call that I was unimpressed with, they began calling my engineers in order to put pressure on me to re-engage. Harassing engineers is a hard no.
- brink 3y agoYup, they did that to our team too. Calling my coworker during the evening, emailing me and messaging me on LinkedIn at the same time. When I said "no thanks" they replied with a message to set up a meeting. Went from a maybe someday to a never real quick. They really need to rethink their marketing.
- bdcravens 3y agoIt feels to me like they have a culture that enables out-of-control sales staff. Probably juicy commissions with a tendency to look the other way.
- himlion 3y agoI wonder if that kind of marketing is actually quite successful. You'd think they wouldn't risk their reputation otherwise right?
- MikeDelta 3y agoCould be salespeople that are paid by contracts closed, with zero regards for the consequences to the company's reputation.