3 ms·

Ok. But the good old push code to staging / canary it before mainstream updates was a simpler way of solving the same problem. Crowdstrike knows the computers

Ok. But the good old push code to staging / canary it before mainstream updates was a simpler way of solving the same problem. Crowdstrike knows the computers they're running on, it is trivial to implement a system where only few designated computers download and install the update and report metrics before the update controller decides to push it to next set.

phartenfeller 2y ago
Why trust somebody else not messing up? With that in place for windows and crowdstrike billions of dollars would be saved and many lives not negatively impacted ...
rldjbpin 2y ago
with the way they handled the debian crashing a little while ago, frankly they are happy to still go ahead with testing this way. still much better way to handle things than pushing to everybody at the same time.
Archelaos 2y ago
It would mitigate the problem, but not solve it. You can still imagine a condition that only occurs after the update has been rolled out everywhere. Furthermore, such a bug would still be extremely problematic for the concerned customers, even if not all of them were affected. In addition, it would be necessary to react very quickly in the case of zero-day vulnerabilities.
- tantalor 2y ago
  (semantic argument warning) "Mitigation" is dealing with an outage/breakage after it occurs, to reduce the impact or get system healthy again. You're talking about "prevention" which keeps it from happening at all. Canarying is generic approach to prevention, and should not be skipped. Avoiding the risk entirely (eBPF) would also help prevent outage, but I think we're deluding ourselves to say it "solves" the problem once and for all; systems will still go down due to bad deploys.
- blinkingled 2y ago
  Yes, I am not arguing against having the ability to deal with it quickly - I am saying canary/ staging helps you do exactly that. Because as we see in the case of Intel CPUs and Crowdstrike some problems or scale of some problems is best prevented.

phartenfeller 2y ago

Why trust somebody else not messing up? With that in place for windows and crowdstrike billions of dollars would be saved and many lives not negatively impacted ...

rldjbpin 2y ago

with the way they handled the debian crashing a little while ago, frankly they are happy to still go ahead with testing this way. still much better way to handle things than pushing to everybody at the same time.

Archelaos 2y ago

It would mitigate the problem, but not solve it. You can still imagine a condition that only occurs after the update has been rolled out everywhere. Furthermore, such a bug would still be extremely problematic for the concerned customers, even if not all of them were affected. In addition, it would be necessary to react very quickly in the case of zero-day vulnerabilities.

tantalor 2y ago
(semantic argument warning) "Mitigation" is dealing with an outage/breakage after it occurs, to reduce the impact or get system healthy again. You're talking about "prevention" which keeps it from happening at all. Canarying is generic approach to prevention, and should not be skipped. Avoiding the risk entirely (eBPF) would also help prevent outage, but I think we're deluding ourselves to say it "solves" the problem once and for all; systems will still go down due to bad deploys.
blinkingled 2y ago
Yes, I am not arguing against having the ability to deal with it quickly - I am saying canary/ staging helps you do exactly that. Because as we see in the case of Intel CPUs and Crowdstrike some problems or scale of some problems is best prevented.