4 ms·
Latest scoop: At 06:00 UTC on March 8th, 2023 the Datadog platform started experiencing widespread issues across multiple products. The web application was una
by donutshop 4y ago
Latest scoop:
At 06:00 UTC on March 8th, 2023 the Datadog platform started experiencing widespread issues across multiple products. The web application was unavailable or intermittently loading, and data ingestion & monitor evaluation were delayed.
We have identified and remedied the issue that caused this outage. We will prepare and share a detailed root cause analysis as soon as possible after our incident response is complete, but we can share a preliminary analysis now. A critical software update applied to a broad set of hosts in our infrastructure caused a subset of these hosts to lose network connectivity.
The primary impact of this was that several of our regional Kubernetes clusters became unhealthy, affecting the control plane that keeps our workloads running smoothly. At this point, we believe we have repaired all the affected Kubernetes clusters, and our recovery efforts are now focused on the application layer above this.
The web application is now generally available, although data and monitor evaluation remains delayed in some cases (refer to the Status Page in your region for the latest information). We have made substantial progress on restoring the various core services that were impacted by the incident, and have now moved on to getting our data processing pipelines for metrics, logs, traces, and other data into a healthy state.
It is difficult to give a precise ETA on our full recovery and we are focusing our efforts on restoring real-time data and alerts within a matter of hours (not minutes, but also not days). The recovery of historical data (between the start of the outage and 15 minutes in the past) has been deprioritized.
We understand the impact an outage can have, and are sorry for the disruption.