3 ms·
I don’t have much sympathy for CrowdStrike but deploying slowly seems mutually exclusive to protecting against emerging threats. They have to strike a balance.
by taspeotis 2y ago
I don’t have much sympathy for CrowdStrike but deploying slowly seems mutually exclusive to protecting against emerging threats. They have to strike a balance.
- yardstick 2y agoIn CrowdStrikes case, they could have rolled out to even 1 million endpoints first and done an automated sanity/wellness check before unleashing the content update on everyone. In the past when I have designed update mechanisms I’ve included basic failsafes such as automated checking for a % failed updates over a sliding 24-hour window and stopping any more if there’s too many failures.
- zavec 2y agoEven a staged rollout over a few hours would have made a huge difference here. "Slow" in the context of a rollout can still be pretty fast.
- bboygravity 2y agoBut it can also still be way too slow in the context of an exploit that is being abused globally.
- taspeotis 2y agoSure but GP is praising "deploy so slowly that people complain."
- getcrunk 2y agoSeriously like rolling out on some exponential scale even over the course of 10 minutes would have stopped this dead in its tracks
- goalieca 2y agoThey need a lab full of canaries.
- raffraffraff 2y agoyeah, I don't get the "we couldn't have tested it" crap, because "something happens to the payload after we tested it". Create a fake downstream company and put a bunch of machines in it. That's your final test before releasing to the rest of the world.