Reminds me a bit of this: https://discord.statuspage.io/incidents/62gt9cgjwdgf https://discord.statuspage.io/incidents/62gt9cgjwdgf In particular: "14:55 - Eng

Reminds me a bit of this: https://discord.statuspage.io/incidents/62gt9cgjwdgf https://discord.statuspage.io/incidents/62gt9cgjwdgf In particular: "14:55 - Engineers pinpoint the issue to be strongly correlated to a spike in errors in originating from our service discovery modules. It is determined that the service discovery processes of our API service had gotten into a crash loop due to an unexpected deserialization error. This triggered an event called "max restart intensity" where, the process's supervisor determined it was crashing too frequently, and decided to trigger a full restart of the node. This event occurred instantaneously across approximately 50% of the nodes that were watching for API nodes, across multiple clusters. We believed it to be related to us hitting a cap in the number of watchers in etcd (the key-value store we use for service discovery.) We attempt to increase this using runtime configuration. Engineers continue to remediate any failed nodes, and restore service to our users."