6 ms·

We use [insert very large application performance monitoring tool here] for workloads running in [insert very, very large cloud provider here] and after examini

We use [insert very large application performance monitoring tool here] for workloads running in [insert very, very large cloud provider here] and after examining our deployments, concluded that we were spending nearly $13k/mo for data transfer out expenditures because the monitoring agents have crazy aggressive defaults. Seems like running our own (which may be worthwhile) would alleviate anything like that.

spahl 6y ago
Yes that is frustrating indeed. On top of paying your external vendor, you are punished by the egress cost you have to pay to your infrastructure cloud provider. This is one of the problems we wanted to solve. Feel free to contact me seb@opstrace.com.
nrmitchi 6y ago
Tip, if you happen to be using datadog, make sure datadog agent logs are disabled from being ingested into datadog. If you can disable them at the agent level and avoid the data out that would be even better. At a previous employer the defaults were quite literally half of our log volume, that we were paying for. I was doing a sanity check before renewing our datadog contract and was very not-pleased to discover that.
- fat-apple 6y ago
  We’re about to release a Datadog compatible API so you can point your Datadog agent at Opstrace instead (stay tuned for the blog post). Our goal is to be able to tell you exactly how much data the agent is sending and how much that is costing you (and for example what services/containers are responsible for the bulk of the cost). Here’s a list of the PRs: https://github.com/opstrace/opstrace/pulls?q=is%3Apr+is%3Aclosed+author%3Ajgehrcke+DD+API https://github.com/opstrace/opstrace/pulls?q=is%3Apr+is%3Acl...
- mdaniel 6y ago
  I even opened a support ticket for their stupid python agent logging its connection refused tracebacks on every metrics poll and was told "too bad" They really don't give one whit about log discipline or allowing the user to influence the agent's log levels
  - englambert 6y ago
    Perhaps on a related note, see this discussion about the power of incentives here: https://news.ycombinator.com/item?id=25994653 https://news.ycombinator.com/item?id=25994653
- brodouevencode 6y ago
  Lol no, the other really large one. Five minute Cloudwatch polling defaults are just overkill even in production.
alexchamberlain 6y ago
It feels like the large monitoring applications should run aggregators in large cloud providers to reduce traffic for everyone.
- jgehrcke 6y ago
  Haha, sure. I suppose that for example AWS has little incentive for allowing for example Datadog to to offer a special per-AZ endpoint. But hey. Here we come into play :).
cyberpunk 6y ago
Can hurt yourself that way too -- happened to us, but with not a lot of data, and all down to Thanos aggregating/reducing/whatever-ing meeeeeeeeeelions of metrics inside a s3 bucket to the tune of about 7k a month :/
- valyala 6y ago
  Probably you need monitoring solution that stores data on plain old EBS disks (or GCE persistent disks) such as VictoriaMetrics? Then you don't need to pay for IO operations. And such disks usually have an order of magnitude lower latencies for IO operations than S3, so they may return results faster.

spahl 6y ago

Yes that is frustrating indeed. On top of paying your external vendor, you are punished by the egress cost you have to pay to your infrastructure cloud provider. This is one of the problems we wanted to solve. Feel free to contact me seb@opstrace.com.

nrmitchi 6y ago

Tip, if you happen to be using datadog, make sure datadog agent logs are disabled from being ingested into datadog. If you can disable them at the agent level and avoid the data out that would be even better. At a previous employer the defaults were quite literally half of our log volume, that we were paying for. I was doing a sanity check before renewing our datadog contract and was very not-pleased to discover that.

fat-apple 6y ago
We’re about to release a Datadog compatible API so you can point your Datadog agent at Opstrace instead (stay tuned for the blog post). Our goal is to be able to tell you exactly how much data the agent is sending and how much that is costing you (and for example what services/containers are responsible for the bulk of the cost). Here’s a list of the PRs: https://github.com/opstrace/opstrace/pulls?q=is%3Apr+is%3Aclosed+author%3Ajgehrcke+DD+API https://github.com/opstrace/opstrace/pulls?q=is%3Apr+is%3Acl...
mdaniel 6y ago
I even opened a support ticket for their stupid python agent logging its connection refused tracebacks on every metrics poll and was told "too bad" They really don't give one whit about log discipline or allowing the user to influence the agent's log levels
- englambert 6y ago
  Perhaps on a related note, see this discussion about the power of incentives here: https://news.ycombinator.com/item?id=25994653 https://news.ycombinator.com/item?id=25994653
brodouevencode 6y ago
Lol no, the other really large one. Five minute Cloudwatch polling defaults are just overkill even in production.

alexchamberlain 6y ago

It feels like the large monitoring applications should run aggregators in large cloud providers to reduce traffic for everyone.

jgehrcke 6y ago
Haha, sure. I suppose that for example AWS has little incentive for allowing for example Datadog to to offer a special per-AZ endpoint. But hey. Here we come into play :).

cyberpunk 6y ago

Can hurt yourself that way too -- happened to us, but with not a lot of data, and all down to Thanos aggregating/reducing/whatever-ing meeeeeeeeeelions of metrics inside a s3 bucket to the tune of about 7k a month :/

valyala 6y ago
Probably you need monitoring solution that stores data on plain old EBS disks (or GCE persistent disks) such as VictoriaMetrics? Then you don't need to pay for IO operations. And such disks usually have an order of magnitude lower latencies for IO operations than S3, so they may return results faster.