4 ms·

This effected a production deploy today - our team did not know datadog was down prior to deployment. Horrible.

palijer 4y ago
We have "datadog outage" as an abort condition that is checked before any deploys or operations. I highly suggest implementing something like that for things you depend on. If y'all are slinging code you are aware that nothing is 100% available, a consumer failing to anticipate that isn't really the providers fault.
- temp_praneshp 4y ago
  What actual metric do you monitor for "datadog outage"? Simply have the deploy tooling make an api request/something else? Our playbook asks the deployer to take a look at dashboard X, but automating this would be nicer for some of our CD pipelines
  - palijer 4y ago
    We don't have a metric for checking and automatically locking deploys. We just integrate their StatusPage notifications into our channels. Datadog doesn't go down often enough for me invest time in automating locking deploys based on it. For anything bigger than a regular code deploys we typically have a runbook ahead of time, and in our template we have a manual check for "make sure datadog is operational" that needs to be checked off on the call. Same with with Github, circleCI, AWS, etc all because we got burned once and and in the postmortem identified that a simple "preflight checklist" would have prevented the issue from lasting so long. It's a good sanity check, reading The Checklist Manifesto influenced me here for these. We work in complex systems, gotta make sure all the stuff is in working order before takeoff.
amanj41 4y ago
Out of curiosity, what datadog services do you use that cause deployments to be interrupted when they are having an outage?
- mjcl 4y ago
  Different person, but during a deploy the pipeline checks some the APM metrics (error rate, new errors, latency) to determine if it should roll back the deploy. A previous company would do a canary deployment (like 2 out of 60 instances), wait 20 minutes and only proceed to a full deploy if the canary instances had similar error & latency rates.
- justinzollars 4y ago
  Professionally it is a very bad idea to deploy to production without log monitoring and alerts to see the effects of a deployment.

We have "datadog outage" as an abort condition that is checked before any deploys or operations. I highly suggest implementing something like that for things you depend on. If y'all are slinging code you are aware that nothing is 100% available, a consumer failing to anticipate that isn't really the providers fault.

temp_praneshp 4y ago
What actual metric do you monitor for "datadog outage"? Simply have the deploy tooling make an api request/something else? Our playbook asks the deployer to take a look at dashboard X, but automating this would be nicer for some of our CD pipelines
- palijer 4y ago
  We don't have a metric for checking and automatically locking deploys. We just integrate their StatusPage notifications into our channels. Datadog doesn't go down often enough for me invest time in automating locking deploys based on it. For anything bigger than a regular code deploys we typically have a runbook ahead of time, and in our template we have a manual check for "make sure datadog is operational" that needs to be checked off on the call. Same with with Github, circleCI, AWS, etc all because we got burned once and and in the postmortem identified that a simple "preflight checklist" would have prevented the issue from lasting so long. It's a good sanity check, reading The Checklist Manifesto influenced me here for these. We work in complex systems, gotta make sure all the stuff is in working order before takeoff.

amanj41 4y ago

Out of curiosity, what datadog services do you use that cause deployments to be interrupted when they are having an outage?

mjcl 4y ago
Different person, but during a deploy the pipeline checks some the APM metrics (error rate, new errors, latency) to determine if it should roll back the deploy. A previous company would do a canary deployment (like 2 out of 60 instances), wait 20 minutes and only proceed to a full deploy if the canary instances had similar error & latency rates.
justinzollars 4y ago
Professionally it is a very bad idea to deploy to production without log monitoring and alerts to see the effects of a deployment.