3 ms·
My speculation is they probably did stage the rollout to some extent, but didn't have a viable or fast enough feedback mechanism to let them know there was a ke
by dataflow 2y ago
My speculation is they probably did stage the rollout to some extent, but didn't have a viable or fast enough feedback mechanism to let them know there was a kernel crash. That seems much more plausible than the engineers being incompetent enough to not have staged rollouts at all. Or they might have had it, but only for code rather than data.
- aeyes 2y agoCrowdstrike in their own RCA wrote that they had no staged rollout for these types of updates: https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf https://www.crowdstrike.com/wp-content/uploads/2024/08/Chann... And I believe it because for administrators there is no configuration to delay the rollout of these "content updates", you can only delay the sensor updates.
- ibiza 2y agoI think that is being too charitable. The problem is 100% reproducible. The machine blue-screens at boot up. What kind of staging environment allows this to go through?
- ethbr1 2y agoStaged rollout in this case would be releasing it to a small portion of your customer base, then additional customers, etc.