TLDR: Ubuntu unattended-upgrades and the systemd-networkd `ManageForeignRoutes=true` [1] default broke all Kubernetes networking globally, across multiple regio

TLDR: Ubuntu unattended-upgrades and the systemd-networkd `ManageForeignRoutes=true` [1] default broke all Kubernetes networking globally, across multiple regions and cloud providers simultaneously, at 06:xx UTC. Recovery was difficult and slow due to the scale, particularly with cloud auto-scaling behavior kinda making things worse when all nodes failed simultaneously. Please test how `systemctl restart systemd-networkd` affects your custom networking setup. It is quite likely to break. DataDog also published their own post-mortem: [2] [1] https://www.freedesktop.org/software/systemd/man/networkd.conf.html#ManageForeignRoutes= https://www.freedesktop.org/software/systemd/man/networkd.co... [2] https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/ https://www.datadoghq.com/blog/2023-03-08-multiregion-infras...