4 ms·

Ultimately this would have been almost a non-issue if there had been better deployment strategies in place for also the data file updates. If by changing the s

by _flux 2y ago

Ultimately this would have been almost a non-issue if there had been better deployment strategies in place for also the data file updates. If by changing the system you mean adding some kind of in-kernel isolation to it, then I don't think it would be worth the effort to make that kind of major change to the way operating systems work just to give arguably a minor risk reduction to systems—in particular if CrowdStrike and other vendors take some learnings from this event. Microsoft might improve their system rollback mechanism to also include files that are not strictly integrated to the system, merely used by the parts that are (the channel files loaded by the driver). Actually I think we can just be happy that the incident was a mistake, not an attack. Had this kind of "first ever" situation been an attack, it could be extremely difficult to recover from it. I wonder how well EDRs deal with "attacks from within".. CrowdStrike pulled off the update within 1.5 hours. I wonder if they actually use Falcon themselves? But then somehow missed the problem? Doesn't seem like they eat their own dog food :). (Or at least their own channel files.)

mbreese 2y ago
If many things had gone differently, this could have been avoided. But I’m looking at this from the Microsoft perspective. No matter how much people scream high and loud that it was a CloudStrike issue and not Microsoft’s fault, Microsoft is still getting blamed. It’s a Windows BSOD. I talked to my dad (retired enterprise operations/IT) this weekend and he was telling me that the next computer he buys will probably be a Mac, largely because he doesn’t want to deal with the possibility of a crash like this. Does he run CloudStrike? Not at all. Does he know who they are? Nope. (He’s been retired for a while) What he does know (well, thinks) is that Windows now has an unstable kernel. And Microsoft has no control over distribution policies for other vendors. How those vendors distribute updates is up to them. Even if a sane deployment strategy could have avoided the larger global problems, Microsoft can’t control that. So, if you have Microsoft dealing with negative publicity and public sentiment, with no way to control errors like this in the future, what can you do? To me, the best they can do is kneecap CloudStrike, put the full blame on them, and use this as an excuse to change the kernel/driver model to one where they can have more control over the stability of the OS.
- hello_moto 2y ago
  They will kneecap Security industry and open up another can of worm: Windows insecure back on the menu.
  - mbreese 2y ago
    There are other vendors. Microsoft could even reinstate CloudStrike at some point, but only after an extensive review process. And then probably require similar process reviews/checks for any other vendor that requires the same kernel access. Or just remove the need for kernel access at all and migrate to a better driver architecture at the sacrifice of backwards compatibility. Security software doesn’t need to run in kernel space… there are other ways.
    - hello_moto 2y ago
      That could potentially be a lawsuit against MSFT since their own MSFT Defender is in this space and potentially doing the same thing or else they have way less potency of catching attacks no?
roblabla 2y ago
There's a simple thing microsoft could do to avoid this, that doesn't require anything too crazy. EDRs work in kernel-land because that's the only place you can place yourself to block certain things, like process creation, driver loading, etc... macOS has a userland API for this, called EndpointSecurity, which allows doing all the things an EDR needs, without ever touching kernelland. Microsoft could introduce a similar API, and EDRs would no longer need a driver.
- _flux 2y ago
  I suppose that's what CrowdStrike's system on Mac uses as well, then. Apparently on Linux they use EBPF and Microsoft is researching that for Windows as well: https://github.com/microsoft/ebpf-for-windows https://github.com/microsoft/ebpf-for-windows . So maybe that's actually the solution they'll go with? It would certainly help solving this particular problem, even if not the kernel-integration in general.
- mbreese 2y ago
  This is exactly what I’d advocate for. There are many things that run in kernel space that don’t need to. The Mac model with user-land hooks is one model. EBPF from Linux (and Windows?) is another. I’m sure the reason why Apple migrated was because of all of the bugs/crashes security companies kept introducing into the kernel with kexts. Apple had the ability to change their architecture on a whim because of they aren’t quite a beholden to backwards compatibility as Windows. Microsoft could take this as an opportunity to make some major changes that would be more readily accepted by the market.

by _flux 2y ago

mbreese 2y ago

If many things had gone differently, this could have been avoided. But I’m looking at this from the Microsoft perspective. No matter how much people scream high and loud that it was a CloudStrike issue and not Microsoft’s fault, Microsoft is still getting blamed. It’s a Windows BSOD. I talked to my dad (retired enterprise operations/IT) this weekend and he was telling me that the next computer he buys will probably be a Mac, largely because he doesn’t want to deal with the possibility of a crash like this. Does he run CloudStrike? Not at all. Does he know who they are? Nope. (He’s been retired for a while) What he does know (well, thinks) is that Windows now has an unstable kernel. And Microsoft has no control over distribution policies for other vendors. How those vendors distribute updates is up to them. Even if a sane deployment strategy could have avoided the larger global problems, Microsoft can’t control that. So, if you have Microsoft dealing with negative publicity and public sentiment, with no way to control errors like this in the future, what can you do? To me, the best they can do is kneecap CloudStrike, put the full blame on them, and use this as an excuse to change the kernel/driver model to one where they can have more control over the stability of the OS.

hello_moto 2y ago
They will kneecap Security industry and open up another can of worm: Windows insecure back on the menu.
- mbreese 2y ago
  There are other vendors. Microsoft could even reinstate CloudStrike at some point, but only after an extensive review process. And then probably require similar process reviews/checks for any other vendor that requires the same kernel access. Or just remove the need for kernel access at all and migrate to a better driver architecture at the sacrifice of backwards compatibility. Security software doesn’t need to run in kernel space… there are other ways.
  - hello_moto 2y ago
    That could potentially be a lawsuit against MSFT since their own MSFT Defender is in this space and potentially doing the same thing or else they have way less potency of catching attacks no?

roblabla 2y ago

There's a simple thing microsoft could do to avoid this, that doesn't require anything too crazy. EDRs work in kernel-land because that's the only place you can place yourself to block certain things, like process creation, driver loading, etc... macOS has a userland API for this, called EndpointSecurity, which allows doing all the things an EDR needs, without ever touching kernelland. Microsoft could introduce a similar API, and EDRs would no longer need a driver.

_flux 2y ago
I suppose that's what CrowdStrike's system on Mac uses as well, then. Apparently on Linux they use EBPF and Microsoft is researching that for Windows as well: https://github.com/microsoft/ebpf-for-windows https://github.com/microsoft/ebpf-for-windows . So maybe that's actually the solution they'll go with? It would certainly help solving this particular problem, even if not the kernel-integration in general.
mbreese 2y ago
This is exactly what I’d advocate for. There are many things that run in kernel space that don’t need to. The Mac model with user-land hooks is one model. EBPF from Linux (and Windows?) is another. I’m sure the reason why Apple migrated was because of all of the bugs/crashes security companies kept introducing into the kernel with kexts. Apple had the ability to change their architecture on a whim because of they aren’t quite a beholden to backwards compatibility as Windows. Microsoft could take this as an opportunity to make some major changes that would be more readily accepted by the market.