4 ms·

87 sounds large 2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed. 2022-10-25 17:28: Peak impact with approximately 5%

87 sounds large 2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed. 2022-10-25 17:28: Peak impact with approximately 5% of all HTTP requests resulting in an error with status code 530. 2022-10-25 17:38: An accelerated rollback continues with large data centers acting as Upper tier for many customers. 2022-10-25 18:04: Rollback is complete in all Upper Tiers. 2022-10-25 18:30: Rollback is complete. But this seems like the 87 minutes referred to. The 503 graph appears to show a steep drop between 17:38 and and 17:45, which looks like the "rollback" speed to me. I think they likely drained many of the lower tier caches/pops/edges after which roll back speed doesn't matter very much since the machines being rolled back likely weren't servicing traffic. > large data centers acting as Upper tier for many customers. To me this sounds like a very awkwardly worded way to say "we shifted traffic". Clarification on whether traffic was shifted or tiers were drained would be nice.

hayst4ck 4y ago
Additionally, this was a code release based outage(?). It seems like a graph annotated with releases (or a/b testing changes) would have made this outage somewhat trivial to identify. 2022-10-25 14:39: Multiple teams become involved in the investigation as more customers start reporting increases in errors. 2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed. This is what I find worrying. How does cloudflare track their code releases and does cloudflare annotate their 4 major graphs (errors, rps, utilization, latency) with lines showing when things were pushed to prod?
- yjftsjthsd-h 4y ago
  I dunno; they say > At first glance, the errors looked like they were caused by a different system that had started a release some time before. And with somewhat noisy error rates I'm not sure even annotating graphs with releases would have made it unambiguous.

hayst4ck 4y ago

Additionally, this was a code release based outage(?). It seems like a graph annotated with releases (or a/b testing changes) would have made this outage somewhat trivial to identify. 2022-10-25 14:39: Multiple teams become involved in the investigation as more customers start reporting increases in errors. 2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed. This is what I find worrying. How does cloudflare track their code releases and does cloudflare annotate their 4 major graphs (errors, rps, utilization, latency) with lines showing when things were pushed to prod?

yjftsjthsd-h 4y ago
I dunno; they say > At first glance, the errors looked like they were caused by a different system that had started a release some time before. And with somewhat noisy error rates I'm not sure even annotating graphs with releases would have made it unambiguous.