3 ms·

Would any of these, or even a collection of these, resolving in some direction make it highly improbable that it'll never happen again? Seems to me 3rd party

Would any of these, or even a collection of these, resolving in some direction make it highly improbable that it'll never happen again? Seems to me 3rd party code, running in the kernel, on parsed inputs, that can be remotely updated is enough to be disaster waiting to happen gestures breezily at Friday That's, in the Taleb parlance, a Fat Tony argument, but barring it being a cosmic ray causing a uncorrected bit flop during deploy, I don't think there's room to call it anything but "a disaster waiting to happen"

slt2021 2y ago
kernel driver could have data check on the channel file and fail gracefully/ignore wrong file instead of BSOD. this code is executed only once during the driver initialization, so shouldn't be much overhead, but will greatly improve reliability against broken channel file
- refulgentis 2y ago
  This is going to code as radical, but I always assumed it was derivable from bog-standard first principles that would fit in any economics class I sat in for my 40 credits: the natural cost of these bits we sell is zero, so in the long run, if the bar is "just write a good & tested kernel driver", there will always be one more subsequent market entrant who will go too cheap on engineering. Then, they touch the hot wire and burn down the establishment. That doesn't mean capitalism bad, but it does mean I expect only Microsoft is capable of writing and maintaining this type of software in the long run. Ex. The dentist and dental hygienist were asking me who was attacking Microsoft on Friday, and they were not going to get through to the the subtleties of 3rd kernel driver release gating strategy. MS has a very strong incentive to fix this. I don't know how they will. But I love when incentives align and assume they always will, in the long run.
deleted 2y ago
[deleted]
nickm12 2y ago
Yes, if CrowdStrike was following industry best practices and this happened, it would teach us something novel about industry practices that we could learn from and use to reduce the risk of a similar scale outage happening again. If they weren't following these practices, this is kind of a boring incident with not much to be learned, despite how dramatic the scale is. Practices like staged rollout of changes exist precisely because we've learned these lessons before.
YZF 2y ago
Well, kernel code is kernel code, and kernel code in general takes input from outside the kernel. An audio driver takes audio data, a video driver might take drawing instructions, a file system interacts with files, etc. Microsoft, and others, have been releasing kernel code since forever and for the most part, not crashlooping their entire install base. My Tesla remote updates ... hmph. It doesn't feel like this is inherently impossible. It feels more like not enough design/process to mitigate the risks.

slt2021 2y ago

kernel driver could have data check on the channel file and fail gracefully/ignore wrong file instead of BSOD. this code is executed only once during the driver initialization, so shouldn't be much overhead, but will greatly improve reliability against broken channel file

refulgentis 2y ago
This is going to code as radical, but I always assumed it was derivable from bog-standard first principles that would fit in any economics class I sat in for my 40 credits: the natural cost of these bits we sell is zero, so in the long run, if the bar is "just write a good & tested kernel driver", there will always be one more subsequent market entrant who will go too cheap on engineering. Then, they touch the hot wire and burn down the establishment. That doesn't mean capitalism bad, but it does mean I expect only Microsoft is capable of writing and maintaining this type of software in the long run. Ex. The dentist and dental hygienist were asking me who was attacking Microsoft on Friday, and they were not going to get through to the the subtleties of 3rd kernel driver release gating strategy. MS has a very strong incentive to fix this. I don't know how they will. But I love when incentives align and assume they always will, in the long run.

deleted 2y ago

[deleted]

nickm12 2y ago

Yes, if CrowdStrike was following industry best practices and this happened, it would teach us something novel about industry practices that we could learn from and use to reduce the risk of a similar scale outage happening again. If they weren't following these practices, this is kind of a boring incident with not much to be learned, despite how dramatic the scale is. Practices like staged rollout of changes exist precisely because we've learned these lessons before.

YZF 2y ago

Well, kernel code is kernel code, and kernel code in general takes input from outside the kernel. An audio driver takes audio data, a video driver might take drawing instructions, a file system interacts with files, etc. Microsoft, and others, have been releasing kernel code since forever and for the most part, not crashlooping their entire install base. My Tesla remote updates ... hmph. It doesn't feel like this is inherently impossible. It feels more like not enough design/process to mitigate the risks.