7 ms·

Microsoft says EU to blame for the worst IT outage

43 points by matthewmorgan 2y ago

sandworm101 2y ago
Never waste a crisis. MS is already on the attack, seeking to leverage this crisis for its own ends. Machiavelli would be proud.
bloopernova 2y ago
> Microsoft says 2009 law made cloudstrife outage happen Or, an alternative interpretation: Microsoft had 15 years to fix any issues.
- geitir 2y ago
  You can’t “fix issues” if the software is running at the kernel level
- alephnerd 2y ago
  The fix is preventing kernel mods, but MS legally cannot.
  - Sayrus 2y ago
    Apple did, Microsoft is working on eBPF for Windows[1] but I doubt they'll sunset their kernel modules support. At the very least, it means there are safer ways to load third-party code in the kernel without allowing them to crash your entire system by mistake. Even if kernel modules are still supported, a compliance framework may introduce a "No kernel module" requirement, just like they require a CrowdStrike-like software to be installed. However, doing so is no easy feat. The first version of eBPF was released over 10 years ago. [1] https://github.com/microsoft/ebpf-for-windows https://github.com/microsoft/ebpf-for-windows
    - alephnerd 2y ago
      > Apple did Apple did not have to sign a settlement with the EU, which Microsoft did in 2009. The terms of the settlement state: "Microsoft shall make available to interested undertakings Interoperability Information that enables non-Microsoft server Software Products to interoperate with Windows Server Operating System on an equal footing with other Microsoft Server Software Products." [0] "Microsoft shall ensure on an ongoing basis and in a Timely Manner that the APIs in the Windows Client PC Operating System and the Windows Server Operating System that are called on by Microsoft Security Software Products are documented and available for use by third-party security software products that run on the Windows Client PC Operating System and/or the Windows Server Operating System. These APIs will be documented on the Microsoft Developer Network, unless open publication would create security risks. In such circumstances, Microsoft will provide third-party security vendors with access to such APIs pursuant to a royalty-free license and on fair, reasonable and non-discriminatory terms." [0] This means that by offering Microsoft Defender for Endpoint, Microsoft needs to give similar access to the underlying kernel to competing vendors like CRWD and S1. > At the very least, it means there are safer ways to load third-party code in the kernel without allowing them to crash your entire system by mistake An eBPF or a similar technology wouldn't necessarily enhance stability when probing kernel-land from user-land, as any interaction with a kernel can cause kernel panics. Plenty of endpoint vendors have had issues with PTrace (MacOS), kprobes (Linux), eBPF (Linux, K8s), etc. The reality is that $))&#( happens, and a lot of comments on HN about this bug are really dumb. [0] - https://news.microsoft.com/download/archived/presskits/eu-msft/docs/MicrosoftInteroperabilityUndertaking16Dec2009.doc https://news.microsoft.com/download/archived/presskits/eu-ms...
      - Sayrus 2y ago
        > Plenty of endpoint vendors have had issues with PTrace (MacOS), kprobes (Linux), eBPF (Linux, K8s), etc. Nothing is perfect, latest generation of Intel CPUs are unstable, bugs are part of all kernels, and so on and so forth. But eBPF does allow for less crashes than a kernel module and these crashes are usually sandboxing error rather than an error made by the third-party provider. As a matter of fact, the eBPF Linux version of Crowdstrike did create kernel panics on some distros. > This means that by offering Microsoft Defender for Endpoint, Microsoft needs to give similar access to the underlying kernel to competing vendors like CRWD and S1. Absolutely, and that's a good thing. This doesn't seem to prevent them from moving Defender to another set of security APIs or from creating another set of security APIs for anyone to hook on.
jcfrei 2y ago
What a disingenuous argument by Microsoft - I really hope nobody buys it. Lots of the most mission critical software runs on Linux. They don't have these security issues because they were open from the start.
- gjsman-1000 2y ago
  Meh - Microsoft can point to the fact that they saved the day by preventing every Linux device from getting backdoored with the `xz-utils` debacle. The fact that it was caught was an exceedingly lucky break, and had nothing to do with quality engineering, or even that the code was open source. (Edit for the downvoters - nothing I said above was an opinion. https://news.ycombinator.com/item?id=41049312 https://news.ycombinator.com/item?id=41049312)
  - vectorhacker 2y ago
    I think the fact that it was caught is a testament to the power of open source and the quality of engineering. It speaks volumes about the promise of OSS.
    - gjsman-1000 2y ago
      > I think the fact that it was caught is a testament to the power of open source and the quality of engineering. Ha, no. It was caught because an engineer noticed his SSH was taking slightly longer than normal, a few hundred milliseconds of difference. There was nothing in the code to suggest an anomaly; so he began fully reverse-engineering the binary as though it was proprietary software. The open-source communities meanwhile had approved and were even distributing that code on testing branches. And to top it all off, that engineer worked for Microsoft; so it's pretty ironic to complain about Microsoft's behavior with Windows, considering they just saved Linux from catastrophe.
      - bryanrasmussen 2y ago
        wait, did the open source community have access to Microsoft's code and then fail to find the Cloudstrike vulnerability even though they had the same amount of time with that code that MS had with theirs? I mean open source idea is that even MS engineer can find their problems. And then MS engineer found their problems. So yeah, does seem like a win for open source ideas which is not even something I particular care about...
lionradio 2y ago
I laughed
Timber-6539 2y ago
A ridiculous excuse. They could have provided an non-EU Windows version if that's the case or better yet, create a robust solution as a software vendor to the said security companies.
- deleted 2y ago
  [deleted]
- Tabular-Iceberg 2y ago
  Then they wouldn't be able to use CrowdStrike. People interested in running CrowdStrike would be forced to use the EU version anyway, and people uninterested wouldn't be affected by the Crowdstrike bug whether they used the EU version or not, so I don't see exactly how having two versions of Windows would help here.
  - Timber-6539 2y ago
    My assumption here is that only the EU version of Windows would crash as Crowdstrike bugged only those versions. The rest of the world would have come to a different solution not based on kernel-level access. I.e Taking Microsoft's argument to it's literal conclusion.
ragebol 2y ago
LOL, sure
deleted 2y ago
[deleted]
offmycloud 2y ago
Would that be the same Microsoft that approved and digitally signed the buggy CrowdStrike Falcon Agent kernel mode driver for Windows?
- jeroenhd 2y ago
  From what I can tell, Microsoft signed DigiCert's certificate, and DigiCert signed CrowdStrike's certificate, and CrowdStrike signed the driver file. Windows kernel signing does not work like Apple's Big Brother approach, it uses a set of certificate authorities. Microsoft does have a program for verifying drivers: WHQL, which you may recognise from the slower driver that Windows Update installs for your GPU before you download the faster one that didn't pass Microsoft's verification from the manufacturer's website. CrowdStrike doesn't seem to be WHQL-certified.
asmnzxklopqw 2y ago
TLDR; Microsoft says EU to blame for letting competition to exist.
wordofx 2y ago
MS is damned if they do damned if they don’t. You can already see it in the comments. “They had 15 years to fix this”, “this na an excuse”, “they are already on the attack” etc. If MS had blocked these type of things people would be in here complaining about antitrust and MS is evil.
- drawfloat 2y ago
  They had 15 years to fix this.
- throwawayFanta 2y ago
  Microsoft is stuck because anything they'd do would reduce the "freedom" of end users. I work in big tech, and unfortunately we frequently need to have conversations about the smallest features because we have evidence about us giving users an inch and they taking a mile.
- Alupis 2y ago
  Every time something like this comes up - people are unwilling and incapable of comprehending regulation has consequences. This is one of the consequences... Our EU friends really enjoy having all the regs on everything... but then demand to be treated as-if the regs don't exist. It's amazing to see...
- shufflerofrocks 2y ago
  HN and in general the software community has a hate-boner for Microsoft, it is almost tradition at this poin. While the hate is valid in many cases, I've observed that the cribbing about it has also been unwarranted or unjustified a lot of the time (also no other corp is held to the same standard) - and this is a prime example. MS cannot legally restrict third party kernel. Apple can, bc they didn't get struck down like MS did. MS has an option to not bundle Defender with their OS, which would let them lock the kernel to avoid the anti-trust restrictions, but that would be an insane decision to make. Damned if they do, Damned if they don't indeed
justin66 2y ago
I didn’t notice any actual quotations, and the article ends: Euronews Next has contacted Microsoft for comment High quality stuff.
throwaway5752 2y ago
What an incredibly embarrassing response. They didn't even have to say anything.
jmclnx 2y ago
Looks like a third dup of the same plot: https://news.ycombinator.com/item?id=41038520 https://news.ycombinator.com/item?id=41038520 https://news.ycombinator.com/item?id=41029590 https://news.ycombinator.com/item?id=41029590
poikroequ 2y ago
Why would Microsoft even bother making this comment? Is the outage in some part their fault? I was under the impression it had everything to do with the botched croudstrike update, and nothing to do with Windows itself. This could have just as well happened with some widely deployed antivirus running in the Linux kernel.
- dagmx 2y ago
  The headline is clickbait. Microsoft is saying why they couldn’t secure the kernel against such an attack, and are right in saying that the EU prevents them from closing it off to third parties. They are not saying the EU is the root cause of the failure, just that they cannot close the hole currently due to the EU. What they leave out is that they could choose to integrate Defender into the OS for free, thereby removing it as a product to compete against. They could also move Defender to not require kernel hooks either. Neither are options they want to consider currently.
  - zogrodea 2y ago
    Integrating Defender sounds like it would create an antitrust issue? If I remember correctly, MS was in the past taken to court and forced to sell some product or other separately, when they previously provided it for free. No comment about being able to move Defender to not require kernel hooks (I don't know).
    - dagmx 2y ago
      It’s unclear imho if it would be an antitrust behaviour. Many regions have security exceptions as long as it’s a core feature. It also wouldn’t prevent competitors working at a higher level either.
- gumby 2y ago
  > Why would Microsoft even bother making this comment? Is the outage in some part their fault? Two reasons: 1 - Few people understand anything about how their computer (/car/stove/phone/medicine/...) works -- they spend their time on other things. Without any model of how their device works its easy to misassign responsibility (see how many people think that Safari is Google or vice versa). So it's in MS's interest to try to get the message out. Of course people do this when they are at fault as well. 2 - EU is in a wave of beating up* on certain large companies. This can also be an opportunistic way to push back. * I am not implying whether I think the EU is correct or not.
heisig 2y ago
Now Microsoft just sounds like pre-Brexit Britain. Why reflect on your own shortcomings when you can blame the EU instead :) I suggest Microsoft follows Britain's example and leaves. The main difference is that we Europeans actually miss the Brits, whereas nobody would miss Microsoft and its shoddy products and business practices. On a more serious note, I fully understand that the Digital Markets Act is causing Microsoft headaches. But I think this headache is well deserved. Big Tech has been building moats where they should have built bridges, and now our computing landscape resembles medieval Germany where everything was at the mercy of a few feudal lords. It is time to drive out those lords and reshape software in a way that empowers, not enslaves.
- lolokwhatever 2y ago
  [flagged]
- deleted 2y ago
  [deleted]
ChrisArchitect 2y ago
[dupe] More discussion: https://news.ycombinator.com/item?id=41029590 https://news.ycombinator.com/item?id=41029590
amai 2y ago
The EU prays for a MSexit from the EU. The efficiency gains by that would be enormous. If we could just get also a SAPexit, Europe would become unbeatable.

43 points by matthewmorgan 2y ago

Never waste a crisis. MS is already on the attack, seeking to leverage this crisis for its own ends. Machiavelli would be proud.

bloopernova 2y ago

> Microsoft says 2009 law made cloudstrife outage happen Or, an alternative interpretation: Microsoft had 15 years to fix any issues.

geitir 2y ago
You can’t “fix issues” if the software is running at the kernel level
alephnerd 2y ago
The fix is preventing kernel mods, but MS legally cannot.
- Sayrus 2y ago
  Apple did, Microsoft is working on eBPF for Windows[1] but I doubt they'll sunset their kernel modules support. At the very least, it means there are safer ways to load third-party code in the kernel without allowing them to crash your entire system by mistake. Even if kernel modules are still supported, a compliance framework may introduce a "No kernel module" requirement, just like they require a CrowdStrike-like software to be installed. However, doing so is no easy feat. The first version of eBPF was released over 10 years ago. [1] https://github.com/microsoft/ebpf-for-windows https://github.com/microsoft/ebpf-for-windows
  - alephnerd 2y ago
    > Apple did Apple did not have to sign a settlement with the EU, which Microsoft did in 2009. The terms of the settlement state: "Microsoft shall make available to interested undertakings Interoperability Information that enables non-Microsoft server Software Products to interoperate with Windows Server Operating System on an equal footing with other Microsoft Server Software Products." [0] "Microsoft shall ensure on an ongoing basis and in a Timely Manner that the APIs in the Windows Client PC Operating System and the Windows Server Operating System that are called on by Microsoft Security Software Products are documented and available for use by third-party security software products that run on the Windows Client PC Operating System and/or the Windows Server Operating System. These APIs will be documented on the Microsoft Developer Network, unless open publication would create security risks. In such circumstances, Microsoft will provide third-party security vendors with access to such APIs pursuant to a royalty-free license and on fair, reasonable and non-discriminatory terms." [0] This means that by offering Microsoft Defender for Endpoint, Microsoft needs to give similar access to the underlying kernel to competing vendors like CRWD and S1. > At the very least, it means there are safer ways to load third-party code in the kernel without allowing them to crash your entire system by mistake An eBPF or a similar technology wouldn't necessarily enhance stability when probing kernel-land from user-land, as any interaction with a kernel can cause kernel panics. Plenty of endpoint vendors have had issues with PTrace (MacOS), kprobes (Linux), eBPF (Linux, K8s), etc. The reality is that $))&#( happens, and a lot of comments on HN about this bug are really dumb. [0] - https://news.microsoft.com/download/archived/presskits/eu-msft/docs/MicrosoftInteroperabilityUndertaking16Dec2009.doc https://news.microsoft.com/download/archived/presskits/eu-ms...
    - Sayrus 2y ago
      > Plenty of endpoint vendors have had issues with PTrace (MacOS), kprobes (Linux), eBPF (Linux, K8s), etc. Nothing is perfect, latest generation of Intel CPUs are unstable, bugs are part of all kernels, and so on and so forth. But eBPF does allow for less crashes than a kernel module and these crashes are usually sandboxing error rather than an error made by the third-party provider. As a matter of fact, the eBPF Linux version of Crowdstrike did create kernel panics on some distros. > This means that by offering Microsoft Defender for Endpoint, Microsoft needs to give similar access to the underlying kernel to competing vendors like CRWD and S1. Absolutely, and that's a good thing. This doesn't seem to prevent them from moving Defender to another set of security APIs or from creating another set of security APIs for anyone to hook on.

jcfrei 2y ago

What a disingenuous argument by Microsoft - I really hope nobody buys it. Lots of the most mission critical software runs on Linux. They don't have these security issues because they were open from the start.

gjsman-1000 2y ago
Meh - Microsoft can point to the fact that they saved the day by preventing every Linux device from getting backdoored with the `xz-utils` debacle. The fact that it was caught was an exceedingly lucky break, and had nothing to do with quality engineering, or even that the code was open source. (Edit for the downvoters - nothing I said above was an opinion. https://news.ycombinator.com/item?id=41049312 https://news.ycombinator.com/item?id=41049312)
- vectorhacker 2y ago
  I think the fact that it was caught is a testament to the power of open source and the quality of engineering. It speaks volumes about the promise of OSS.
  - gjsman-1000 2y ago
    > I think the fact that it was caught is a testament to the power of open source and the quality of engineering. Ha, no. It was caught because an engineer noticed his SSH was taking slightly longer than normal, a few hundred milliseconds of difference. There was nothing in the code to suggest an anomaly; so he began fully reverse-engineering the binary as though it was proprietary software. The open-source communities meanwhile had approved and were even distributing that code on testing branches. And to top it all off, that engineer worked for Microsoft; so it's pretty ironic to complain about Microsoft's behavior with Windows, considering they just saved Linux from catastrophe.
    - bryanrasmussen 2y ago
      wait, did the open source community have access to Microsoft's code and then fail to find the Cloudstrike vulnerability even though they had the same amount of time with that code that MS had with theirs? I mean open source idea is that even MS engineer can find their problems. And then MS engineer found their problems. So yeah, does seem like a win for open source ideas which is not even something I particular care about...

lionradio 2y ago

I laughed

Timber-6539 2y ago

A ridiculous excuse. They could have provided an non-EU Windows version if that's the case or better yet, create a robust solution as a software vendor to the said security companies.

deleted 2y ago
[deleted]
Tabular-Iceberg 2y ago
Then they wouldn't be able to use CrowdStrike. People interested in running CrowdStrike would be forced to use the EU version anyway, and people uninterested wouldn't be affected by the Crowdstrike bug whether they used the EU version or not, so I don't see exactly how having two versions of Windows would help here.
- Timber-6539 2y ago
  My assumption here is that only the EU version of Windows would crash as Crowdstrike bugged only those versions. The rest of the world would have come to a different solution not based on kernel-level access. I.e Taking Microsoft's argument to it's literal conclusion.

ragebol 2y ago

LOL, sure

deleted 2y ago

[deleted]

offmycloud 2y ago

Would that be the same Microsoft that approved and digitally signed the buggy CrowdStrike Falcon Agent kernel mode driver for Windows?

jeroenhd 2y ago
From what I can tell, Microsoft signed DigiCert's certificate, and DigiCert signed CrowdStrike's certificate, and CrowdStrike signed the driver file. Windows kernel signing does not work like Apple's Big Brother approach, it uses a set of certificate authorities. Microsoft does have a program for verifying drivers: WHQL, which you may recognise from the slower driver that Windows Update installs for your GPU before you download the faster one that didn't pass Microsoft's verification from the manufacturer's website. CrowdStrike doesn't seem to be WHQL-certified.

asmnzxklopqw 2y ago

TLDR; Microsoft says EU to blame for letting competition to exist.

wordofx 2y ago

MS is damned if they do damned if they don’t. You can already see it in the comments. “They had 15 years to fix this”, “this na an excuse”, “they are already on the attack” etc. If MS had blocked these type of things people would be in here complaining about antitrust and MS is evil.

drawfloat 2y ago
They had 15 years to fix this.
throwawayFanta 2y ago
Microsoft is stuck because anything they'd do would reduce the "freedom" of end users. I work in big tech, and unfortunately we frequently need to have conversations about the smallest features because we have evidence about us giving users an inch and they taking a mile.
Alupis 2y ago
Every time something like this comes up - people are unwilling and incapable of comprehending regulation has consequences. This is one of the consequences... Our EU friends really enjoy having all the regs on everything... but then demand to be treated as-if the regs don't exist. It's amazing to see...
shufflerofrocks 2y ago
HN and in general the software community has a hate-boner for Microsoft, it is almost tradition at this poin. While the hate is valid in many cases, I've observed that the cribbing about it has also been unwarranted or unjustified a lot of the time (also no other corp is held to the same standard) - and this is a prime example. MS cannot legally restrict third party kernel. Apple can, bc they didn't get struck down like MS did. MS has an option to not bundle Defender with their OS, which would let them lock the kernel to avoid the anti-trust restrictions, but that would be an insane decision to make. Damned if they do, Damned if they don't indeed

justin66 2y ago

I didn’t notice any actual quotations, and the article ends: Euronews Next has contacted Microsoft for comment High quality stuff.

throwaway5752 2y ago

What an incredibly embarrassing response. They didn't even have to say anything.

jmclnx 2y ago

Looks like a third dup of the same plot: https://news.ycombinator.com/item?id=41038520 https://news.ycombinator.com/item?id=41038520 https://news.ycombinator.com/item?id=41029590 https://news.ycombinator.com/item?id=41029590

poikroequ 2y ago

Why would Microsoft even bother making this comment? Is the outage in some part their fault? I was under the impression it had everything to do with the botched croudstrike update, and nothing to do with Windows itself. This could have just as well happened with some widely deployed antivirus running in the Linux kernel.

dagmx 2y ago
The headline is clickbait. Microsoft is saying why they couldn’t secure the kernel against such an attack, and are right in saying that the EU prevents them from closing it off to third parties. They are not saying the EU is the root cause of the failure, just that they cannot close the hole currently due to the EU. What they leave out is that they could choose to integrate Defender into the OS for free, thereby removing it as a product to compete against. They could also move Defender to not require kernel hooks either. Neither are options they want to consider currently.
- zogrodea 2y ago
  Integrating Defender sounds like it would create an antitrust issue? If I remember correctly, MS was in the past taken to court and forced to sell some product or other separately, when they previously provided it for free. No comment about being able to move Defender to not require kernel hooks (I don't know).
  - dagmx 2y ago
    It’s unclear imho if it would be an antitrust behaviour. Many regions have security exceptions as long as it’s a core feature. It also wouldn’t prevent competitors working at a higher level either.
gumby 2y ago
> Why would Microsoft even bother making this comment? Is the outage in some part their fault? Two reasons: 1 - Few people understand anything about how their computer (/car/stove/phone/medicine/...) works -- they spend their time on other things. Without any model of how their device works its easy to misassign responsibility (see how many people think that Safari is Google or vice versa). So it's in MS's interest to try to get the message out. Of course people do this when they are at fault as well. 2 - EU is in a wave of beating up* on certain large companies. This can also be an opportunistic way to push back. * I am not implying whether I think the EU is correct or not.

heisig 2y ago

Now Microsoft just sounds like pre-Brexit Britain. Why reflect on your own shortcomings when you can blame the EU instead :) I suggest Microsoft follows Britain's example and leaves. The main difference is that we Europeans actually miss the Brits, whereas nobody would miss Microsoft and its shoddy products and business practices. On a more serious note, I fully understand that the Digital Markets Act is causing Microsoft headaches. But I think this headache is well deserved. Big Tech has been building moats where they should have built bridges, and now our computing landscape resembles medieval Germany where everything was at the mercy of a few feudal lords. It is time to drive out those lords and reshape software in a way that empowers, not enslaves.

lolokwhatever 2y ago
[flagged]
deleted 2y ago
[deleted]

ChrisArchitect 2y ago

[dupe] More discussion: https://news.ycombinator.com/item?id=41029590 https://news.ycombinator.com/item?id=41029590

amai 2y ago

The EU prays for a MSexit from the EU. The efficiency gains by that would be enormous. If we could just get also a SAPexit, Europe would become unbeatable.