6 ms·

Microsoft blames EU rules for allowing biggest IT outage to happen

21 points by riv991 2y ago

bnchrch 2y ago
non-paywall: https://archive.ph/20240722121551/https://www.telegraph.co.uk/business/2024/07/22/microsoft-blames-eu-rules-worlds-biggest-tech-outage/ https://archive.ph/20240722121551/https://www.telegraph.co.u...
dijit 2y ago
Yeah, of course they would. "Just regulate us less, seriously!". Their solution to this situation is quite literally monopoly, which is hilarious.
janice1999 2y ago
Is anyone really gullible enough to blame the EU for CrowdStrike rolling out a faulty update? What is Microsoft PR thinking? I also find it amusing that George Kurtz (CrowdStrike CEO) was perfectly capable of causing a global IT disaster in 2010 as McAfee CTO without kernel access (the update just deleted critical Windows XP system files).
_heimdall 2y ago
That's cute, but I don't know of any EU rule that requires a Windows to give the blue screen of death when a third party kernel module fails.
- sunflowerfly 2y ago
  It is by design. If the system is allowed to continue even worse things might happen. Such as data corruption.
  - hilbert42 2y ago
    That's if allowed to continue booting, the correct procure would be on fault, reload without the patch. There are additional measures too that Microsoft could have adopted but this would have required the rewriting of code that goes back to Windows 2000 or earlier. But that would cost and thus eat into MS's profits. Oh sorry, I forgot doing that would be the ultimate sin.
    - _heimdall 2y ago
      This 100%. I could see a fault tolerance threshold where a module has to fail say three times in a row before bring disbled, but the OS should be able to handle failing third party kernel modules.
      - hilbert42 2y ago
        Absolutely. Here's a rehash in a similar story: https://news.ycombinator.com/item?id=41097794 https://news.ycombinator.com/item?id=41097794
fabian2k 2y ago
I don't find that particularly credible. If Microsoft would boot all security products (including their own) out of the kernel and force them to use specific interfaces for this there is no anti-trust concern at all there. It would only be a problem if they wanted Windows Defender to remain having privileged access while taking it away from everyone else.
vfclists 2y ago
The true reason is a regulatory environment where snake-oil peddling corporations lobby for governments to mandate the use of their crapware in the name of security and reliability. This causes senior management to push for the installation of this crapware/malware on their systems, overriding the objection of their system administrators who know better. Senior management want to cover their asses, and the administrators shrug their shoulders and respond "if you say so". In sum it is mostly a regulatory racket that is profitable for the peddlers of this crapware and for management who can claim that they did what they were required to do to stop shit from happening. Everybody wins except shareholders, customers and IT staff who have to fix this mess without getting the righteously justified overtime and bonuses. Why do you think insurance companies have backing out of the business of insuring against these kind of disasters? They've clearly learned better. Why do you think I stopped using Norton, MacAfee, etc, etc, and etc and opted for just decent backups and Windows own built-in stuff?
- orev 2y ago
  Are you saying that all anti-malware software is junk? Sure there are some that are bad, but also some that are fine, just like any other marketplace.
- MattPalmer1086 2y ago
  What government regulations mandate the use of EDR in private corporations?
anotherhue 2y ago
Never waste a crisis.
mwhitfield 2y ago
The headline, as always, is disingenuous. They were asked why they couldn't lock third parties out of this level of unprotected system access, and said that the 2009 ruling prevented them from doing so. Which is simply factually correct.
- redprince 2y ago
  While that is correct they could at any point in the last 15 years have implemented an API for security software which does not require to load executable code into kernel space. Kind of like Apple did: https://developer.apple.com/documentation/endpointsecurity https://developer.apple.com/documentation/endpointsecurity Then Microsoft would have proceeded to only use this documented user land API themselves for their own Defender product and thus have no undocumented API or access advantage over other security software. The EU ruling only cares about a level playing field and not about the implementation details.
josefritzishere 2y ago
I'm sure Microsoft would prefer to be less regulated, but I dont see how that would have changed the outcome here.
- JohnTHaller 2y ago
  CrowdStrike could have been locked out of kernel space and possibly not been able to screw up all those machines that badly.
  - ClumsyPilot 2y ago
    I am reasonably confident that if Microsoft publically communicated that CrowdStrike breaks their recommendations and decreases security of their systems, CrowdStrike would have way fewer customers
nullfrigid 2y ago
Microsoft has a very solid point here. MS has wanted to kick AV vendors out of kernel space for a long time because it isn't necessary, and can lead to the type of incident we are talking about here. MS provides a userspace interface[0] for AV vendors to do what they need to do, but they can't be forced to use it. So yes, due to EU regulations, AV vendors can still play in kernel space, and can bring much of the world to a halt when they make a mistake as a result. [0] https://learn.microsoft.com/en-us/windows/win32/amsi/antimalware-scan-interface-portal https://learn.microsoft.com/en-us/windows/win32/amsi/antimal...
- piva00 2y ago
  If they provide a userspace interface but use their own set of kernel accesses for Windows Defender then they are not competing fairly, don't you think? They just need to make Windows Defender use those userspace interfaces as well and all is good :)
  - nullfrigid 2y ago
    [dead]
- trueismywork 2y ago
  The real cause of mistake was a single point of failure where there was no A/B testing, no gradual deployment nothing. To say that banning AV from kernel would prevent this is not just hilarious but disingenuous and shows complete lack of knowledge of operations and deployment. There's no golden rule which says that Windows cannot make these errors.
  - cdumler 2y ago
    That is a solution, but not the cause. The cause is not having a culture that evaluates failure scenarios. From what I have read: * Updates are not vetted or sanity checked. * Updates are not slow-rolled to production. * Updates are not signed to prevent corruption or alteration. * Updater does not sanitize or validate inputs. * Updater does not have a reversion process to previously known good position on faulty boot. * Updater should mark itself as Unnecessary For Boot on faulty boot at some point. Finally, its high adoption means it creates a mono-culture. There should be another version built independently where one is running on a machine and another sits in a ready state. If there is a fault in one, it becomes disabled and the second takes over. Good ol' NASA style redundancy.
hilbert42 2y ago
Oh yeah, if Microsoft was so worried about other companies tinkering with its kernel then why didn't it introduce routines that would ensure that a reboot would actually occur on a boot load error? (Upon error, a reload would then omit the faulty code as well as tell users there was a problem with the update.) I'd suggest that there is no reason a BSOD—Blue Screen of Death—should ever occur on a system that was already working as the OS should be constructed in such a way that it can undo a faulty patch. As you'd know, there is already such a thing as Volume Shadow Copy, VSS, in MS Windows. Microsoft could have adopted this and similar techniques to ensure that the system either stayed up or rebooted. Yes, I can hear Microsoft's retort now that doing that would make Windows more vulnerable to viruses, infiltration, etc. To that I'd say utter bullshit, the real problem—as it has always been with Microsoft—is that it doesn't properly finish or bootstrap its code against errors before it releases it to the public. Microsoft is thus doing cheapskate engineering as it's much more profitable. Hopefully, eventually regulators will require hardening of such software together with guarantees against such faults—guarantees that if not honored would result in enforceable financial penalties. Only loss of income/profit is likely to fix this problem. EU, for everyone's sake quickly debunk that deliberately misleading PR crap from Microsoft before it takes hold.

21 points by riv991 2y ago

non-paywall: https://archive.ph/20240722121551/https://www.telegraph.co.uk/business/2024/07/22/microsoft-blames-eu-rules-worlds-biggest-tech-outage/ https://archive.ph/20240722121551/https://www.telegraph.co.u...

dijit 2y ago

Yeah, of course they would. "Just regulate us less, seriously!". Their solution to this situation is quite literally monopoly, which is hilarious.

janice1999 2y ago

Is anyone really gullible enough to blame the EU for CrowdStrike rolling out a faulty update? What is Microsoft PR thinking? I also find it amusing that George Kurtz (CrowdStrike CEO) was perfectly capable of causing a global IT disaster in 2010 as McAfee CTO without kernel access (the update just deleted critical Windows XP system files).

_heimdall 2y ago

That's cute, but I don't know of any EU rule that requires a Windows to give the blue screen of death when a third party kernel module fails.

sunflowerfly 2y ago
It is by design. If the system is allowed to continue even worse things might happen. Such as data corruption.
- hilbert42 2y ago
  That's if allowed to continue booting, the correct procure would be on fault, reload without the patch. There are additional measures too that Microsoft could have adopted but this would have required the rewriting of code that goes back to Windows 2000 or earlier. But that would cost and thus eat into MS's profits. Oh sorry, I forgot doing that would be the ultimate sin.
  - _heimdall 2y ago
    This 100%. I could see a fault tolerance threshold where a module has to fail say three times in a row before bring disbled, but the OS should be able to handle failing third party kernel modules.
    - hilbert42 2y ago
      Absolutely. Here's a rehash in a similar story: https://news.ycombinator.com/item?id=41097794 https://news.ycombinator.com/item?id=41097794

fabian2k 2y ago

I don't find that particularly credible. If Microsoft would boot all security products (including their own) out of the kernel and force them to use specific interfaces for this there is no anti-trust concern at all there. It would only be a problem if they wanted Windows Defender to remain having privileged access while taking it away from everyone else.

vfclists 2y ago

The true reason is a regulatory environment where snake-oil peddling corporations lobby for governments to mandate the use of their crapware in the name of security and reliability. This causes senior management to push for the installation of this crapware/malware on their systems, overriding the objection of their system administrators who know better. Senior management want to cover their asses, and the administrators shrug their shoulders and respond "if you say so". In sum it is mostly a regulatory racket that is profitable for the peddlers of this crapware and for management who can claim that they did what they were required to do to stop shit from happening. Everybody wins except shareholders, customers and IT staff who have to fix this mess without getting the righteously justified overtime and bonuses. Why do you think insurance companies have backing out of the business of insuring against these kind of disasters? They've clearly learned better. Why do you think I stopped using Norton, MacAfee, etc, etc, and etc and opted for just decent backups and Windows own built-in stuff?

orev 2y ago
Are you saying that all anti-malware software is junk? Sure there are some that are bad, but also some that are fine, just like any other marketplace.
MattPalmer1086 2y ago
What government regulations mandate the use of EDR in private corporations?

anotherhue 2y ago

Never waste a crisis.

mwhitfield 2y ago

The headline, as always, is disingenuous. They were asked why they couldn't lock third parties out of this level of unprotected system access, and said that the 2009 ruling prevented them from doing so. Which is simply factually correct.

redprince 2y ago
While that is correct they could at any point in the last 15 years have implemented an API for security software which does not require to load executable code into kernel space. Kind of like Apple did: https://developer.apple.com/documentation/endpointsecurity https://developer.apple.com/documentation/endpointsecurity Then Microsoft would have proceeded to only use this documented user land API themselves for their own Defender product and thus have no undocumented API or access advantage over other security software. The EU ruling only cares about a level playing field and not about the implementation details.

josefritzishere 2y ago

I'm sure Microsoft would prefer to be less regulated, but I dont see how that would have changed the outcome here.

JohnTHaller 2y ago
CrowdStrike could have been locked out of kernel space and possibly not been able to screw up all those machines that badly.
- ClumsyPilot 2y ago
  I am reasonably confident that if Microsoft publically communicated that CrowdStrike breaks their recommendations and decreases security of their systems, CrowdStrike would have way fewer customers

nullfrigid 2y ago

Microsoft has a very solid point here. MS has wanted to kick AV vendors out of kernel space for a long time because it isn't necessary, and can lead to the type of incident we are talking about here. MS provides a userspace interface[0] for AV vendors to do what they need to do, but they can't be forced to use it. So yes, due to EU regulations, AV vendors can still play in kernel space, and can bring much of the world to a halt when they make a mistake as a result. [0] https://learn.microsoft.com/en-us/windows/win32/amsi/antimalware-scan-interface-portal https://learn.microsoft.com/en-us/windows/win32/amsi/antimal...

piva00 2y ago
If they provide a userspace interface but use their own set of kernel accesses for Windows Defender then they are not competing fairly, don't you think? They just need to make Windows Defender use those userspace interfaces as well and all is good :)
- nullfrigid 2y ago
  [dead]
trueismywork 2y ago
The real cause of mistake was a single point of failure where there was no A/B testing, no gradual deployment nothing. To say that banning AV from kernel would prevent this is not just hilarious but disingenuous and shows complete lack of knowledge of operations and deployment. There's no golden rule which says that Windows cannot make these errors.
- cdumler 2y ago
  That is a solution, but not the cause. The cause is not having a culture that evaluates failure scenarios. From what I have read: * Updates are not vetted or sanity checked. * Updates are not slow-rolled to production. * Updates are not signed to prevent corruption or alteration. * Updater does not sanitize or validate inputs. * Updater does not have a reversion process to previously known good position on faulty boot. * Updater should mark itself as Unnecessary For Boot on faulty boot at some point. Finally, its high adoption means it creates a mono-culture. There should be another version built independently where one is running on a machine and another sits in a ready state. If there is a fault in one, it becomes disabled and the second takes over. Good ol' NASA style redundancy.

hilbert42 2y ago

Oh yeah, if Microsoft was so worried about other companies tinkering with its kernel then why didn't it introduce routines that would ensure that a reboot would actually occur on a boot load error? (Upon error, a reload would then omit the faulty code as well as tell users there was a problem with the update.) I'd suggest that there is no reason a BSOD—Blue Screen of Death—should ever occur on a system that was already working as the OS should be constructed in such a way that it can undo a faulty patch. As you'd know, there is already such a thing as Volume Shadow Copy, VSS, in MS Windows. Microsoft could have adopted this and similar techniques to ensure that the system either stayed up or rebooted. Yes, I can hear Microsoft's retort now that doing that would make Windows more vulnerable to viruses, infiltration, etc. To that I'd say utter bullshit, the real problem—as it has always been with Microsoft—is that it doesn't properly finish or bootstrap its code against errors before it releases it to the public. Microsoft is thus doing cheapskate engineering as it's much more profitable. Hopefully, eventually regulators will require hardening of such software together with guarantees against such faults—guarantees that if not honored would result in enforceable financial penalties. Only loss of income/profit is likely to fix this problem. EU, for everyone's sake quickly debunk that deliberately misleading PR crap from Microsoft before it takes hold.