3 ms·
Additionally, this was a code release based outage(?). It seems like a graph annotated with releases (or a/b testing changes) would have made this outage somewh
by hayst4ck 4y ago
Additionally, this was a code release based outage(?). It seems like a graph annotated with releases (or a/b testing changes) would have made this outage somewhat trivial to identify.
2022-10-25 14:39: Multiple teams become involved in the investigation as more customers start reporting increases in errors.
2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed.
This is what I find worrying.
How does cloudflare track their code releases and does cloudflare annotate their 4 major graphs (errors, rps, utilization, latency) with lines showing when things were pushed to prod?
- yjftsjthsd-h 4y agoI dunno; they say > At first glance, the errors looked like they were caused by a different system that had started a release some time before. And with somewhat noisy error rates I'm not sure even annotating graphs with releases would have made it unambiguous.