4 ms·

There is a frustrating amount of nuance being lost in this discussion, and as usual it's devolving into tribalism. However, I'll say that, while this clearly i

by rewgs 2y ago

There is a frustrating amount of nuance being lost in this discussion, and as usual it's devolving into tribalism. However, I'll say that, while this clearly is not Microsoft's fault, the realization of just how much critical infrastructure is running on Windows -- let alone Windows that's connected to the internet and has automatic updates enabled -- was sobering. Are kernel mode drives maybe a bad idea? Yes! Should Windows be able to automatically roll back if a kernel mode driver fails? Yes! Should CrowdStrike have tested before pushing out the update? Yes! Should hospitals, police stations, airlines, etc be testing any and all updates that come their way prior to releasing them onto the rest of their fleets? Yes! And so on and so on. The failures here are legion. I can at least say for myself that all of that (and more) is what I'm grappling with today, rather than the direct, root cause and content of the issue itself. I say this entirely with as little tribalism or bias as I can muster: Windows should not be the foundation upon which critical infrastructure is built. It is a bad OS. It is a rickety, insecure mess; this observation (note that I did not say opinion) has only ever increased with time. The NT kernel, I hear, is great. I generally trust those who say this, as they tend to be demonstrably much smarter than I am, especially with regards to kernel design, which is something I know next to nothing about. But Windows? Nah man. Maybe there's a good kernel running the show, but the OS as a whole is a shitshow. Yes, this exact same problem could have (and has) happened on Linux; the OS is indeed not the culprit. But the understanding that this one, single, no good very bad OS is responsible for this much critical infrastructure going from an abstract worry to a real, concrete horror show has, I think, hit a lot of people, leading to a poorly-pivoted change of subject.

gruez 2y ago
>There is a frustrating amount of nuance being lost in this discussion, and as usual it's devolving into tribalism. It's ironic that you're saying this given the points you're making below. Let's go through them: >Are kernel mode drives maybe a bad idea? Yes! You can't have a EDR product that isn't kernel mode. Otherwise it's trivial for malware to evade (eg. by being kernel mode themselves). >Should Windows be able to automatically roll back if a kernel mode driver fails? Yes! see: https://news.ycombinator.com/item?id=41019743 https://news.ycombinator.com/item?id=41019743 >Should hospitals, police stations, airlines, etc be testing any and all updates that come their way prior to releasing them onto the rest of their fleets? Yes! Getting companies to test updates is already like pulling teeth. Besides, crowdstrike said the update they pushed was "designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks". Is this something you really want to sit on for testing, which might take weeks or months? >Windows should not be the foundation upon which critical infrastructure is built. It is a bad OS. It is a rickety, insecure mess; this observation (note that I did not say opinion) has only ever increased with time. >The NT kernel, I hear, is great. I generally trust those who say this, as they tend to be demonstrably much smarter than I am, especially with regards to kernel design, which is something I know next to nothing about. >But Windows? Nah man. Maybe there's a good kernel running the show, but the OS as a whole is a shitshow. What specific security issues do you think windows/NT kernel has? Moreover, how is windows being a "shitshow" relevant to the question of resiliency or dependence? Don't get me wrong, windows spying on you or using dark patterns to get you to use Edge or whatever isn't great, but it's a weird thing to bring up in a discussion about how airports run on windows, and reeks of "don't let a disaster go to waste" on the part of the anti-windows tribe.
- Hikikomori 2y ago
  Mac doesn't allow 3rd part kernel drivers and on Linux they use ebpf. Is their product useless there?
  - gruez 2y ago
    >Is their product useless there? Probably? For instance I doubt an EDR product can detect malware being executed on iOS/Android, because all the apps there are heavily sandboxed and provide no mechanism to do invasive monitoring of everything's that's being run. >Linux they use ebpf According to wikipedia it's been ported to windows and on linux you can still load kernel modules which are crashable.
  - jkrejcha 2y ago
    > and on Linux On Linux, they apparently had done the same thing and caused a bunch of Linux systems to crash[1]. [1]: https://www.neowin.net/news/crowdstrike-broke-debian-and-rocky-linux-months-ago-but-no-one-noticed/ https://www.neowin.net/news/crowdstrike-broke-debian-and-roc...
- rewgs 2y ago
  > It's ironic that you're saying this given the points you're making below. The nuance I was referring to was with regards to the actual facts of the situation, such as who is responsible, and that many, many people are just using it as an excuse to dunk on Windows. As I said, Windows/Microsoft are not at fault in this precise situation. However, is it possible that I can state that a thing is bad without it being "tribalism?" I'm invoking tribalism and setting myself apart from it in an attempt to make it very clear that my criticism of Windows is not simply tribalism. Stating that Boeing airplanes are prone to critical faults due to bad engineering is a fact, not tribalism; that is still true even if I personally dislike Boeing airplanes/the company. Perhaps I'm even critical of them precisely because of the bad engineering I've observed! Weird how that works. The same can be said for Windows. > Getting companies to test updates is already like pulling teeth. Besides, crowdstrike said the update they pushed was "designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks". Is this something you really want to sit on for testing, which might take weeks or months? gestures broadly I mean...given the current situation, obviously I'm going to answer with an emphatic "yes!" You know we have these things called computers, right? They're really good at automating stuff. Like testing. I know that "getting companies to test updates is like pulling teeth," but that doesn't mean it shouldn't be done. Companies do all sorts of stupid and negligent bullshit, are happy to spend money in the name of shifting blame, but are cheap as hell with regards to actually avoiding problems. That's not a good thing, and it should change. Is that really such a controversial statement? Apologies for potentially engaging in strawmanning, but in the even that your response is something along the lines of "they should test, but it's not realistic to expect that," yeah, I agree, but perhaps this precise event is the kick in the pants those who are against testing need to stop being cheap morons. And in case you're not clean on who I'm referring to: the decision makers at the top, not the engineers caring out their irresponsible and negligent agendas. I was in the ER quite literally the day before this hit with a slash to my popliteal artery (long story, freak accident), and I shudder to think how it would have gone a day later -- I honestly could have bled out and died. The fact that so many places running absolutely critical infrastructure aren't routinely testing every change they push out to their devices is insane. Utterly, bat shit insane. > What specific security issues do you think windows/NT kernel has? Sorry, not taking the bait on this one. Windows is a piece of shit, and it's absolutely self-evident to anyone even kind of exposed to the alternatives. Expanding on this to you is a waste of time, as either you've never used Windows before (highly unlikely) or you're unwilling to see what I'm talking about for whatever reason. > Moreover, how is windows being a "shitshow" relevant to the question of resiliency or dependence? Do I really have to explain to you why a shitshow of an OS isn't resilient? I've tried to make it extremely clear that 1) I don't blame Windows or Microsoft in this incident, but 2) this incident revealed just how much critical infrastructure relies on an OS that has no business being used as such. That's not "don't let a disaster go to waste," it's one disaster revealing a situation that is ripe for many, many more. I'm not "anti-Windows," I'm "anti-Windows-as-a-server" and "anti-horrible-system-administration-practices."
averageRoyalty 2y ago
But as you've outlined multiple times, the flaws of the OS are not the problem here. Changing OS, the same poor decisions can and have been made. Windows can be locked down or configured to a very stable level, and Linux can be configured to a shitshow. I don't like Windows, but embedded edition has been the most common operating system you encounter in the physical world for decades, and broadly it works so well you don't know. I don't see a good argument that something on the Windows end needs fixing here.
fredski42 2y ago
Totally agree here. There should be a mechanism to go back to ‘last known good’ regardless of kernel level issues. Innovations like Fedora Silverblue with ostree and greenboot tech should be adopted by Windows.
- gruez 2y ago
  >There should be a mechanism to go back to ‘last known good’ regardless of kernel level issues. Innovations like Fedora Silverblue with ostree and greenboot tech should be adopted by Windows. Can you explain how they work? AFAIK the issue is that they pushed a bad config file, and that's the thing that caused the crash, not a new driver. Are those systems going to roll back every file ever to try to recover themselves?
  - josephcsible 2y ago
    > Are those systems going to roll back every file ever to try to recover themselves? Isn't that totally feasible with things like Btrfs snapshots?
    - AstralStorm 2y ago
      It is, assuming your disk driver or related drivers aren't the ones failing. Then again, nothing is preventing an update loop here.
AstralStorm 2y ago
Windows specifically does not allow automated rollback in case of boot drivers because: a) that generally does not work, because the driver is required to boot into recovery environment b) even if it did, for security like this, downgrade attacks are a consideration and most importantly c) in this case, Windows does not have a full backup of previous external configuration because this was not a driver update

by rewgs 2y ago

gruez 2y ago

>There is a frustrating amount of nuance being lost in this discussion, and as usual it's devolving into tribalism. It's ironic that you're saying this given the points you're making below. Let's go through them: >Are kernel mode drives maybe a bad idea? Yes! You can't have a EDR product that isn't kernel mode. Otherwise it's trivial for malware to evade (eg. by being kernel mode themselves). >Should Windows be able to automatically roll back if a kernel mode driver fails? Yes! see: https://news.ycombinator.com/item?id=41019743 https://news.ycombinator.com/item?id=41019743 >Should hospitals, police stations, airlines, etc be testing any and all updates that come their way prior to releasing them onto the rest of their fleets? Yes! Getting companies to test updates is already like pulling teeth. Besides, crowdstrike said the update they pushed was "designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks". Is this something you really want to sit on for testing, which might take weeks or months? >Windows should not be the foundation upon which critical infrastructure is built. It is a bad OS. It is a rickety, insecure mess; this observation (note that I did not say opinion) has only ever increased with time. >The NT kernel, I hear, is great. I generally trust those who say this, as they tend to be demonstrably much smarter than I am, especially with regards to kernel design, which is something I know next to nothing about. >But Windows? Nah man. Maybe there's a good kernel running the show, but the OS as a whole is a shitshow. What specific security issues do you think windows/NT kernel has? Moreover, how is windows being a "shitshow" relevant to the question of resiliency or dependence? Don't get me wrong, windows spying on you or using dark patterns to get you to use Edge or whatever isn't great, but it's a weird thing to bring up in a discussion about how airports run on windows, and reeks of "don't let a disaster go to waste" on the part of the anti-windows tribe.

Hikikomori 2y ago
Mac doesn't allow 3rd part kernel drivers and on Linux they use ebpf. Is their product useless there?
- gruez 2y ago
  >Is their product useless there? Probably? For instance I doubt an EDR product can detect malware being executed on iOS/Android, because all the apps there are heavily sandboxed and provide no mechanism to do invasive monitoring of everything's that's being run. >Linux they use ebpf According to wikipedia it's been ported to windows and on linux you can still load kernel modules which are crashable.
- jkrejcha 2y ago
  > and on Linux On Linux, they apparently had done the same thing and caused a bunch of Linux systems to crash[1]. [1]: https://www.neowin.net/news/crowdstrike-broke-debian-and-rocky-linux-months-ago-but-no-one-noticed/ https://www.neowin.net/news/crowdstrike-broke-debian-and-roc...
rewgs 2y ago
> It's ironic that you're saying this given the points you're making below. The nuance I was referring to was with regards to the actual facts of the situation, such as who is responsible, and that many, many people are just using it as an excuse to dunk on Windows. As I said, Windows/Microsoft are not at fault in this precise situation. However, is it possible that I can state that a thing is bad without it being "tribalism?" I'm invoking tribalism and setting myself apart from it in an attempt to make it very clear that my criticism of Windows is not simply tribalism. Stating that Boeing airplanes are prone to critical faults due to bad engineering is a fact, not tribalism; that is still true even if I personally dislike Boeing airplanes/the company. Perhaps I'm even critical of them precisely because of the bad engineering I've observed! Weird how that works. The same can be said for Windows. > Getting companies to test updates is already like pulling teeth. Besides, crowdstrike said the update they pushed was "designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks". Is this something you really want to sit on for testing, which might take weeks or months? gestures broadly I mean...given the current situation, obviously I'm going to answer with an emphatic "yes!" You know we have these things called computers, right? They're really good at automating stuff. Like testing. I know that "getting companies to test updates is like pulling teeth," but that doesn't mean it shouldn't be done. Companies do all sorts of stupid and negligent bullshit, are happy to spend money in the name of shifting blame, but are cheap as hell with regards to actually avoiding problems. That's not a good thing, and it should change. Is that really such a controversial statement? Apologies for potentially engaging in strawmanning, but in the even that your response is something along the lines of "they should test, but it's not realistic to expect that," yeah, I agree, but perhaps this precise event is the kick in the pants those who are against testing need to stop being cheap morons. And in case you're not clean on who I'm referring to: the decision makers at the top, not the engineers caring out their irresponsible and negligent agendas. I was in the ER quite literally the day before this hit with a slash to my popliteal artery (long story, freak accident), and I shudder to think how it would have gone a day later -- I honestly could have bled out and died. The fact that so many places running absolutely critical infrastructure aren't routinely testing every change they push out to their devices is insane. Utterly, bat shit insane. > What specific security issues do you think windows/NT kernel has? Sorry, not taking the bait on this one. Windows is a piece of shit, and it's absolutely self-evident to anyone even kind of exposed to the alternatives. Expanding on this to you is a waste of time, as either you've never used Windows before (highly unlikely) or you're unwilling to see what I'm talking about for whatever reason. > Moreover, how is windows being a "shitshow" relevant to the question of resiliency or dependence? Do I really have to explain to you why a shitshow of an OS isn't resilient? I've tried to make it extremely clear that 1) I don't blame Windows or Microsoft in this incident, but 2) this incident revealed just how much critical infrastructure relies on an OS that has no business being used as such. That's not "don't let a disaster go to waste," it's one disaster revealing a situation that is ripe for many, many more. I'm not "anti-Windows," I'm "anti-Windows-as-a-server" and "anti-horrible-system-administration-practices."

averageRoyalty 2y ago

But as you've outlined multiple times, the flaws of the OS are not the problem here. Changing OS, the same poor decisions can and have been made. Windows can be locked down or configured to a very stable level, and Linux can be configured to a shitshow. I don't like Windows, but embedded edition has been the most common operating system you encounter in the physical world for decades, and broadly it works so well you don't know. I don't see a good argument that something on the Windows end needs fixing here.

fredski42 2y ago

Totally agree here. There should be a mechanism to go back to ‘last known good’ regardless of kernel level issues. Innovations like Fedora Silverblue with ostree and greenboot tech should be adopted by Windows.

gruez 2y ago
>There should be a mechanism to go back to ‘last known good’ regardless of kernel level issues. Innovations like Fedora Silverblue with ostree and greenboot tech should be adopted by Windows. Can you explain how they work? AFAIK the issue is that they pushed a bad config file, and that's the thing that caused the crash, not a new driver. Are those systems going to roll back every file ever to try to recover themselves?
- josephcsible 2y ago
  > Are those systems going to roll back every file ever to try to recover themselves? Isn't that totally feasible with things like Btrfs snapshots?
  - AstralStorm 2y ago
    It is, assuming your disk driver or related drivers aren't the ones failing. Then again, nothing is preventing an update loop here.

AstralStorm 2y ago

Windows specifically does not allow automated rollback in case of boot drivers because: a) that generally does not work, because the driver is required to boot into recovery environment b) even if it did, for security like this, downgrade attacks are a consideration and most importantly c) in this case, Windows does not have a full backup of previous external configuration because this was not a driver update