4 ms·
What commonly happens in these organizations is they have a software delivery path that has a lot of these best practices but soon people figure out that it is
by siliconc0w 2y ago
What commonly happens in these organizations is they have a software delivery path that has a lot of these best practices but soon people figure out that it is too slow so they invent a new, faster, path. From what I can tell Crowdstrike had a lot of the usual best practices like canary rollouts on their binary but they didn't on this configuration file despite it having the same consequences of a bad binary push. This wasn't even an edge case, it reliably BSOD'd every windows machine that got this update.
One strategy Google SRE uses is that the team ensuring reliability has a different reporting path than the product team - so there is always a check and balance when things like rollout policies get worked around by clever product teams.
It's a shame because I hear it's actually a pretty good product.