4 ms·

I've lived that life with Datadog. I was the dev who was the most into logging so I got voluntold to be in charge of the Datadog rollout. Told everyone it was

I've lived that life with Datadog. I was the dev who was the most into logging so I got voluntold to be in charge of the Datadog rollout. Told everyone it was not a good idea and presented cost estimates to why, we're better off just keeping our own logs and using one of many open source projects that let you pull in logs and analyze them. But all the cool kids were on Datadog, so onto it we went. What killed us was the combo of the desire to really aggressive log on the staging-master and release-candidate environments (five envs total) and keep the logs for more than 7 days so the QA team could compare the logs from the current release candidate to the previous release candidates. Between the volume of having all the services set at the info level and the desire to keep it more than 7 days, Datadog was >30k a month. Which at least finally got me permission to set up an open source log manager and switch everyone over to that. Once the initial panic over the Datadog bill died down.

dmlittle 6y ago
I haven't used DD logs personally but from what I've been told if you send the majority of your logs straight to archive to an S3 bucket and only rehydrate the logs when you need to access them their pricing is pretty reasonable. Because you're storing the logs in your S3 bucket you can keep them however long you want and rehydrate them when you need them (say 30 days later). The bill will still be in the low thousands but paying 30-40k per year for a logging solution is probably worth the cost if it means you don't have to maintain it. Having logs and metrics (and the APM if you're using it) all in one place is _really_ nice.
- cinquemb 6y ago
  You have to survive as company long enough where paying 30k/month sounds reasonable for logging… esp if your revenue is nowhere near it, and are not VC funded… and you will always have to maintain your integrations at the very least.
  - dmlittle 6y ago
    If you're using log rehydration with DD it's hard to hit the $30k/mo fee. Before I left my last company we were evaluating DD logging and it came out to ~$5-6k/mo for about 25B logs per month. When you start you'll likely pay a few hundred dollars per month tops. Your personal time alone is probably worth more than that.
    - cinquemb 6y ago
      Before I left my last company, we were spending about a couple hundred dollars per month just on media storage/bandwidth alone for serving academic papers/docs/data (as well as our logs)… in top 5k in alexa, top 100 in indonesia with about maybe 60-70 customers paying 150-200 dollars per year coming from mostly EM countries… the customers can barely afford/justify spending what we were offering over using some open source alternatives, and there would be no way in hell we'd spend that on just logging. I at least helped 10x the customers and cut costs from when I joined to before I left (stayed a little over a year)… but man, talk about constraints…
dosethree 6y ago
Whats the point of even having it in data dog? Typically we pay a ton for services like Newrelic for monitoring, the APM, and insights. Like you, we have tons of logs, so I wouldn't want to pay Newrelic or Newrelic-similar prices for a service you can just use ELK for yourself much cheaper. So basically I don't really see why you signed on to data dog at all

dmlittle 6y ago

I haven't used DD logs personally but from what I've been told if you send the majority of your logs straight to archive to an S3 bucket and only rehydrate the logs when you need to access them their pricing is pretty reasonable. Because you're storing the logs in your S3 bucket you can keep them however long you want and rehydrate them when you need them (say 30 days later). The bill will still be in the low thousands but paying 30-40k per year for a logging solution is probably worth the cost if it means you don't have to maintain it. Having logs and metrics (and the APM if you're using it) all in one place is _really_ nice.

cinquemb 6y ago
You have to survive as company long enough where paying 30k/month sounds reasonable for logging… esp if your revenue is nowhere near it, and are not VC funded… and you will always have to maintain your integrations at the very least.
- dmlittle 6y ago
  If you're using log rehydration with DD it's hard to hit the $30k/mo fee. Before I left my last company we were evaluating DD logging and it came out to ~$5-6k/mo for about 25B logs per month. When you start you'll likely pay a few hundred dollars per month tops. Your personal time alone is probably worth more than that.
  - cinquemb 6y ago
    Before I left my last company, we were spending about a couple hundred dollars per month just on media storage/bandwidth alone for serving academic papers/docs/data (as well as our logs)… in top 5k in alexa, top 100 in indonesia with about maybe 60-70 customers paying 150-200 dollars per year coming from mostly EM countries… the customers can barely afford/justify spending what we were offering over using some open source alternatives, and there would be no way in hell we'd spend that on just logging. I at least helped 10x the customers and cut costs from when I joined to before I left (stayed a little over a year)… but man, talk about constraints…

dosethree 6y ago

Whats the point of even having it in data dog? Typically we pay a ton for services like Newrelic for monitoring, the APM, and insights. Like you, we have tons of logs, so I wouldn't want to pay Newrelic or Newrelic-similar prices for a service you can just use ELK for yourself much cheaper. So basically I don't really see why you signed on to data dog at all