4 ms·

When deciding whether to do an incident investigation for a near miss, one aspect to consider is whether it was caught by a safety system as designed, or caught

by wrs 2y ago

When deciding whether to do an incident investigation for a near miss, one aspect to consider is whether it was caught by a safety system as designed, or caught by a lucky accident. The latter should be top of the priority list. E.g., Bad package deployed to production. Stopped because it didn’t have the “CI tested” flag: low pri. Stopped because someone happened to notice a high CPU alert before the load balancer switched over: high pri.

fambalamboni 2y ago
> Stopped because it didn’t have the “CI tested” flag: low pri. Agree, certainly lower priority than the "caught by chance" case, but I wouldn't say "low priority". That system is the last line of defence and the fact that something is hitting it is definitely a large concern. In the CrowdStrike's fiasco last year, their last line of defence, "Content Validator" failed. "On July 19, 2024, two additional IPC Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data." https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio...
- wrs 2y ago
  Oh absolutely, if you have the budget then do them all. I just meant “lower pri” than the other. My assumption is in most orgs you’ll find it hard to take time to investigate any of them, so you have to place your bets.
tetha 2y ago
It can be a good idea to count and record these safety net activations for reporting purposes. If you do so, the discussion of e.g. "Why do we run replicas for our databases" becomes a very simple discussion: It has simplified us maintenance X times this year (by avoiding downtime to to switch overs), and it has avoided Y outages (again due to automated switch overs) - recovery of the databases after the outage took Z hours, which would have been downtime. Suddenly you've moved the vague "We need it for safety and availability reasons" into concrete admin-hours and downtime-minutes. Or, "Can't we skip all these tests/other long-running quality process?"
- wrs 2y ago
  Definitely, counting the ones you have is great data for justifying their existence, and can help identify places you should perhaps put a check farther upstream.

by wrs 2y ago

fambalamboni 2y ago

> Stopped because it didn’t have the “CI tested” flag: low pri. Agree, certainly lower priority than the "caught by chance" case, but I wouldn't say "low priority". That system is the last line of defence and the fact that something is hitting it is definitely a large concern. In the CrowdStrike's fiasco last year, their last line of defence, "Content Validator" failed. "On July 19, 2024, two additional IPC Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data." https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio...

wrs 2y ago
Oh absolutely, if you have the budget then do them all. I just meant “lower pri” than the other. My assumption is in most orgs you’ll find it hard to take time to investigate any of them, so you have to place your bets.

tetha 2y ago

It can be a good idea to count and record these safety net activations for reporting purposes. If you do so, the discussion of e.g. "Why do we run replicas for our databases" becomes a very simple discussion: It has simplified us maintenance X times this year (by avoiding downtime to to switch overs), and it has avoided Y outages (again due to automated switch overs) - recovery of the databases after the outage took Z hours, which would have been downtime. Suddenly you've moved the vague "We need it for safety and availability reasons" into concrete admin-hours and downtime-minutes. Or, "Can't we skip all these tests/other long-running quality process?"

wrs 2y ago
Definitely, counting the ones you have is great data for justifying their existence, and can help identify places you should perhaps put a check farther upstream.