3 ms·

Having never heard of datadog, Wikipedia’s summary is: > Datadog is an observability service for cloud-scale applications, providing monitoring of servers, dat

Having never heard of datadog, Wikipedia’s summary is: > Datadog is an observability service for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform. So it checks if your servers have crashed or slowed down with a nice dashboard? Any better summaries or descriptions of what it does and how coinbase would have used it?

eli 4y ago
it's a New Relic competitor
bdcravens 4y ago
They also do client side monitoring, log aggregation, and provide metrics for your overall operations (not just when there are performance issues). They also have quite an unethical sales operation: https://news.ycombinator.com/item?id=35837965 https://news.ycombinator.com/item?id=35837965
- kingsloi 4y ago
  That's funny, DD is the only company to email my work email, add me as a connection on LinkedIn, txt me over WhatApp, and call my personal phone number multiple times. Amber flag after the LinkedIn connection/message, but red/purple after the WhatsApp/call on my personal phone.
jabroni_salad 4y ago
it ingests and can help you sort through logs, and also does performance monitoring. I guess I would describe it as prometheus + logstash + powerBI in a single unit. And with a healthy dose of blue cross bolted on for the surprise bills and difficult bureaucracy.
hdjjhhvvhga 4y ago
Suppose you have a bunch of k8s clusters, an AWS Organization etc. You just follow a simple setup and see a nice Dashboard with practically every aspect of your infrastructure, from accounts to nodes to pods.
yuck39 4y ago
I would wager you could build something pretty close to what data dog is selling for $65M
deleted 4y ago
[deleted]
KaiserPro 4y ago
Datadog from when I was looking at them (2017ish) appeared to be an automatic version of nagios with a nice user interface and super simple client side installer. But it was super tied to VMs at that point, and we were running a bunch of lambdas, herokus and docker instances, along with a shit tonne of AWS services, and java lumps from the 90s
Arbortheus 4y ago
Observability is about more than crashes or slowdowns, serious investment in observability is a must-have for any SaaS/cloud product to have reliability, auto scaling and velocity. It’s more than just crashes/slowdowns. My team use Grafana’s open source LGTM stack. We use Prometheus metrics to track anything from JVM/Go runtime stats, K8S metrics, saturation of CPU/memory, scalability issues, crashes/OOMs, custom metrics for business insights, debugging. We use USE/RED metrics (see: Google’s SRE handbook) to track our production services performance in an objective way. We track SLAs and SLOs so we know when it’s time to focus on features and business impact, and when it’s time to put that aside to focus on stability and maintenance before our customers notice reduced reliability. As a developer it’s really helpful for testing changes. For example, I added a new database index in dev, then run some load tests and check our dashboards before and after. I look at Q95 latency of APIs and database load to see if it has the desired effect, then when I roll out to production I can monitor those same dashboards and make sure the same desired improvement can be seen for real-word usage. I used traces recently to discover that something that should have been happening in parallel was instead happening sequentially leading to very long/timing out requests. Adding visualisations via traces helps get your head around how something is working. I added annotations to our dashboards that shows when our K8S pods restart alongside the metrics. This made me realise that some requests were failing exactly around deployments because we were not cleanly handling SIGTERM in some services. We have started adding horizontal auto scaling based on metrics for the number of queued messages on a specific Kafka queue. If a large number of messages are waiting we spin up more K8S replicas, and then once this reduces, we reduce the replicas to keep costs down. I optimise the resource allocations on our services by looking at historical CPU/memory usage so we make the best use of our K8S cluster and avoid OOMs as we scale. We use Loki for log querying and parsing, you can create really advanced/domain-specialised log querying dashboards and provide that to your support team, and integrate those logs with traces to debug different stages of a request as it traverses your microservices or different processing stages. You can even build dashboards from logs, which is helpful when debugging a particular type of error over time that you were not specifically monitoring with metrics, or determine which customer(s) are affected by this error. Alternatively if you have a legacy system that does not have effective metrics, you can build metrics from its logs. We use our metrics for alerting and paging in a way that provides a better signal-to-noise ratio than old-school alerts like “high memory usage” so people don’t get woken up as much (we’ve had zero pages since my product launched 6 months ago!). It’s better to alert only when we have a measurable impact on customer experience, like when a smoke test has failed more than 80% of the time, or HTTP requests 5xx rate is elevated to abnormal levels. It’s also really reassuring when you do a prod rollout to easily see that stuff is still working without digging into logs, so you can spend more time coding and less time babying prod. Overall I think having good observability is definitely a worthwhile investment. There are cheaper ways to do it than datadog. I expect much of the trouble is that switching providers is a huge job, we have invested so much time building our observability stack, the challenge of moving seems massive. Thankfully we picked Grafana’s open source LGTM stack and self-hosted it. Even if you picked their SaaS offering, switching to open-source self-hosted is an option so you are less tied in.

eli 4y ago

it's a New Relic competitor

bdcravens 4y ago

They also do client side monitoring, log aggregation, and provide metrics for your overall operations (not just when there are performance issues). They also have quite an unethical sales operation: https://news.ycombinator.com/item?id=35837965 https://news.ycombinator.com/item?id=35837965

kingsloi 4y ago
That's funny, DD is the only company to email my work email, add me as a connection on LinkedIn, txt me over WhatApp, and call my personal phone number multiple times. Amber flag after the LinkedIn connection/message, but red/purple after the WhatsApp/call on my personal phone.

jabroni_salad 4y ago

it ingests and can help you sort through logs, and also does performance monitoring. I guess I would describe it as prometheus + logstash + powerBI in a single unit. And with a healthy dose of blue cross bolted on for the surprise bills and difficult bureaucracy.

hdjjhhvvhga 4y ago

Suppose you have a bunch of k8s clusters, an AWS Organization etc. You just follow a simple setup and see a nice Dashboard with practically every aspect of your infrastructure, from accounts to nodes to pods.

yuck39 4y ago

I would wager you could build something pretty close to what data dog is selling for $65M

deleted 4y ago

[deleted]

KaiserPro 4y ago

Datadog from when I was looking at them (2017ish) appeared to be an automatic version of nagios with a nice user interface and super simple client side installer. But it was super tied to VMs at that point, and we were running a bunch of lambdas, herokus and docker instances, along with a shit tonne of AWS services, and java lumps from the 90s

Arbortheus 4y ago

Observability is about more than crashes or slowdowns, serious investment in observability is a must-have for any SaaS/cloud product to have reliability, auto scaling and velocity. It’s more than just crashes/slowdowns. My team use Grafana’s open source LGTM stack. We use Prometheus metrics to track anything from JVM/Go runtime stats, K8S metrics, saturation of CPU/memory, scalability issues, crashes/OOMs, custom metrics for business insights, debugging. We use USE/RED metrics (see: Google’s SRE handbook) to track our production services performance in an objective way. We track SLAs and SLOs so we know when it’s time to focus on features and business impact, and when it’s time to put that aside to focus on stability and maintenance before our customers notice reduced reliability. As a developer it’s really helpful for testing changes. For example, I added a new database index in dev, then run some load tests and check our dashboards before and after. I look at Q95 latency of APIs and database load to see if it has the desired effect, then when I roll out to production I can monitor those same dashboards and make sure the same desired improvement can be seen for real-word usage. I used traces recently to discover that something that should have been happening in parallel was instead happening sequentially leading to very long/timing out requests. Adding visualisations via traces helps get your head around how something is working. I added annotations to our dashboards that shows when our K8S pods restart alongside the metrics. This made me realise that some requests were failing exactly around deployments because we were not cleanly handling SIGTERM in some services. We have started adding horizontal auto scaling based on metrics for the number of queued messages on a specific Kafka queue. If a large number of messages are waiting we spin up more K8S replicas, and then once this reduces, we reduce the replicas to keep costs down. I optimise the resource allocations on our services by looking at historical CPU/memory usage so we make the best use of our K8S cluster and avoid OOMs as we scale. We use Loki for log querying and parsing, you can create really advanced/domain-specialised log querying dashboards and provide that to your support team, and integrate those logs with traces to debug different stages of a request as it traverses your microservices or different processing stages. You can even build dashboards from logs, which is helpful when debugging a particular type of error over time that you were not specifically monitoring with metrics, or determine which customer(s) are affected by this error. Alternatively if you have a legacy system that does not have effective metrics, you can build metrics from its logs. We use our metrics for alerting and paging in a way that provides a better signal-to-noise ratio than old-school alerts like “high memory usage” so people don’t get woken up as much (we’ve had zero pages since my product launched 6 months ago!). It’s better to alert only when we have a measurable impact on customer experience, like when a smoke test has failed more than 80% of the time, or HTTP requests 5xx rate is elevated to abnormal levels. It’s also really reassuring when you do a prod rollout to easily see that stuff is still working without digging into logs, so you can spend more time coding and less time babying prod. Overall I think having good observability is definitely a worthwhile investment. There are cheaper ways to do it than datadog. I expect much of the trouble is that switching providers is a huge job, we have invested so much time building our observability stack, the challenge of moving seems massive. Thankfully we picked Grafana’s open source LGTM stack and self-hosted it. Even if you picked their SaaS offering, switching to open-source self-hosted is an option so you are less tied in.