5 ms·

Datadog’s $5M Outage

38 points by PreInternet01 3y ago

PreInternet01 3y ago
The style of the article is overly verbose to me, but the underlying incident is fascinating. Most of my workloads run on Windows Server, and having a monthly requirement to reboot due to updates is definitely annoying, but also a great way to force you to carefully orchestrate distributed system cold-starts from day one...
- SamuelGalbraith 3y ago
  I agree!
nubinetwork 3y ago
https://news.ycombinator.com/item?id=35971320 https://news.ycombinator.com/item?id=35971320 https://news.ycombinator.com/item?id=35963879 https://news.ycombinator.com/item?id=35963879
remram 3y ago
> This incident was costly, at $5M in lost revenue directly as a result of the outage, as revealed in a later earnings call. This was due to usage-based billing No SLA whatsoever? Send Datadog your data, they lose it all, no discount or penalty but at least they don't charge you for storing it?
- iruoy 3y ago
  They didn't lose any data according to this article. > For most customers, Datadog’s live functionality was restored within a day and a half, and all data was backfilled within two days, with no reported data loss.
edgyquant 3y ago
We’ve recently signed up with datadog. I think their platform is good, but recently I’ve seen posts on a few platforms stating that they are known for up charging significantly and without warning. Wondering if anyone has experience with this and if we should have went with someone else
terom 3y ago
TLDR: Ubuntu unattended-upgrades and the systemd-networkd `ManageForeignRoutes=true` [1] default broke all Kubernetes networking globally, across multiple regions and cloud providers simultaneously, at 06:xx UTC. Recovery was difficult and slow due to the scale, particularly with cloud auto-scaling behavior kinda making things worse when all nodes failed simultaneously. Please test how `systemctl restart systemd-networkd` affects your custom networking setup. It is quite likely to break. DataDog also published their own post-mortem: [2] [1] https://www.freedesktop.org/software/systemd/man/networkd.conf.html#ManageForeignRoutes= https://www.freedesktop.org/software/systemd/man/networkd.co... [2] https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/ https://www.datadoghq.com/blog/2023-03-08-multiregion-infras...
- throwaway67743 3y ago
  I find it odd they hadn't seen this before, even with a moderate amount of route updates networkd will consume an excessive amount of CPU (even with foreign routes disabled it still processes netlink messages, annoyingly) - this is a pretty rookie thing to miss for a metrics company to be honest
deleted 3y ago
[deleted]
guitarbill 3y ago
> I’ll use Central European Summer Time (CEST) for time stamps, as the majority of Datadog’s engineering team is in this timezone. I’m also indicating US Eastern time (EST) for reference. I can convert from UTC to my timezone (TZ) in my head, since logs are usually UTC. I know other engineers can do this, too. What I cannot do is convert from a random TZ to my TZ without looking it up. Clearly, the author knew this, which is why EST was added for an American audience. Note the Datadog post-mortem is UTC [0], but the status page isn't [1]; even the EU one is ET [2]. So why are time stamps not being in UTC a thing? Reasons I can think of: * Status pages and post-mortems are not for engineers, but for management? * Certain US-based companies use ET (i.e. EST or EDT) or PT (i.e. PST or PDT), and everybody else has cargo-culted it. [0] https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/ https://www.datadoghq.com/blog/2023-03-08-multiregion-infras... [1] https://status.datadoghq.com/history https://status.datadoghq.com/history [2] https://status.datadoghq.eu/history https://status.datadoghq.eu/history

38 points by PreInternet01 3y ago

The style of the article is overly verbose to me, but the underlying incident is fascinating. Most of my workloads run on Windows Server, and having a monthly requirement to reboot due to updates is definitely annoying, but also a great way to force you to carefully orchestrate distributed system cold-starts from day one...

SamuelGalbraith 3y ago
I agree!

nubinetwork 3y ago

https://news.ycombinator.com/item?id=35971320 https://news.ycombinator.com/item?id=35971320 https://news.ycombinator.com/item?id=35963879 https://news.ycombinator.com/item?id=35963879

remram 3y ago

> This incident was costly, at $5M in lost revenue directly as a result of the outage, as revealed in a later earnings call. This was due to usage-based billing No SLA whatsoever? Send Datadog your data, they lose it all, no discount or penalty but at least they don't charge you for storing it?

iruoy 3y ago
They didn't lose any data according to this article. > For most customers, Datadog’s live functionality was restored within a day and a half, and all data was backfilled within two days, with no reported data loss.

edgyquant 3y ago

We’ve recently signed up with datadog. I think their platform is good, but recently I’ve seen posts on a few platforms stating that they are known for up charging significantly and without warning. Wondering if anyone has experience with this and if we should have went with someone else

terom 3y ago

TLDR: Ubuntu unattended-upgrades and the systemd-networkd `ManageForeignRoutes=true` [1] default broke all Kubernetes networking globally, across multiple regions and cloud providers simultaneously, at 06:xx UTC. Recovery was difficult and slow due to the scale, particularly with cloud auto-scaling behavior kinda making things worse when all nodes failed simultaneously. Please test how `systemctl restart systemd-networkd` affects your custom networking setup. It is quite likely to break. DataDog also published their own post-mortem: [2] [1] https://www.freedesktop.org/software/systemd/man/networkd.conf.html#ManageForeignRoutes= https://www.freedesktop.org/software/systemd/man/networkd.co... [2] https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/ https://www.datadoghq.com/blog/2023-03-08-multiregion-infras...

throwaway67743 3y ago
I find it odd they hadn't seen this before, even with a moderate amount of route updates networkd will consume an excessive amount of CPU (even with foreign routes disabled it still processes netlink messages, annoyingly) - this is a pretty rookie thing to miss for a metrics company to be honest

deleted 3y ago

[deleted]

guitarbill 3y ago

> I’ll use Central European Summer Time (CEST) for time stamps, as the majority of Datadog’s engineering team is in this timezone. I’m also indicating US Eastern time (EST) for reference. I can convert from UTC to my timezone (TZ) in my head, since logs are usually UTC. I know other engineers can do this, too. What I cannot do is convert from a random TZ to my TZ without looking it up. Clearly, the author knew this, which is why EST was added for an American audience. Note the Datadog post-mortem is UTC [0], but the status page isn't [1]; even the EU one is ET [2]. So why are time stamps not being in UTC a thing? Reasons I can think of: * Status pages and post-mortems are not for engineers, but for management? * Certain US-based companies use ET (i.e. EST or EDT) or PT (i.e. PST or PDT), and everybody else has cargo-culted it. [0] https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/ https://www.datadoghq.com/blog/2023-03-08-multiregion-infras... [1] https://status.datadoghq.com/history https://status.datadoghq.com/history [2] https://status.datadoghq.eu/history https://status.datadoghq.eu/history