7 ms·

The CrowdStrike Failure Was a Warning

112 points by Kaibeezy 2y ago

gssa 2y ago
[dead]
Kaibeezy 2y ago
https://archive.md/pyW91 https://archive.md/pyW91
mylastattempt 2y ago
I found the article unbearable and just a convoluted way to say: this incident would have had a lot less impact if CrowdStrike had less customers or more competitors. A real page filler without any insight or solutions, just look at this paragraph, completely void of anything useful. > This time, the digital cataclysm was caused by well-intentioned people who made a mistake. That meant the fix came relatively quickly; CrowdStrike knew what had gone wrong. But we may not be so lucky next time. If a malicious actor had attacked CrowdStrike or a similarly essential bit of digital infrastructure, the disaster could have been much worse. Gee, the damage from an honest mistake (what does the author even base that on) is most likely easier to fix than the damage done by a malicious actor with bad intent. I feel so enlightened!
- hsbauauvhabzb 2y ago
  > mistake Funny way of spelling ‘complete failure in functional testing before going to prod causing billions of dollars worth of damage likely due to criminal level negligence’.
  - tuatoru 2y ago
    Billions is probably an underestimate. When you factor on the time spent in meetings discussing endlessly what to do about this in corporations around the world, besides the cancelled flights and banking and hospital and 911 mayhem, a trillion isn't indefensible. Who needs North Korean hackers?
    - hsbauauvhabzb 2y ago
      If crowdstrikes internal competency failed to detect a consistent bsod, I doubt they’d notice North Korea dropping malware into their product solarwinds style.
  - Galanwe 2y ago
    I mean, we've seen impacts of these unstaged updates in the deeper layers of banks, airports and hospitals. It's unfair that CrowdStrike gets more than half the blame here. Not to mention, probably half these companies were actually forced to install this. Getting SOC2 is essentially letting a senior-branded KPMG or EY intern tell you how to enforce security policies based on their checklist. There is far more to blame in this incident than CrowdStrike themselves.
    - aiisjustanif 2y ago
      Compliance is one thing but a cybersecurity incident response team is very blind without an EDR not to mention you would have less protection from insider threat.
  - antonvs 2y ago
    That's true. But it's also true that companies depending on more or less random third parties to provide them with kernel modules for a monolithic kernel - and then giving them control over auto-updating those kernel modules - is beyond dumb.
  - HeatrayEnjoyer 2y ago
    Undoubtedly killing people too: 911 networks failed. Hospitals were crippled. Short of getting your hands on a nuclear football it's difficult to imagine an act with a higher effort:devastation ratio.
    - RCitronsBroker 2y ago
      You’re not wrong about the end result, but the breakdown of systems this complex goes deeper than placing the blame on some CrowdStrike employee. Whoever thought up the great idea to allow auto-update-able kernel modules for something as mission critical as emergency response or healthcare deserves just as much blame. I’ve worked in healthcare for my whole career, this is madness. Not that their process is without flaw, but can we remind ourselves of how stringently we assess medical devices? I cannot imagine it’s controversial to say that emergency response equipment is every bit as critical as a insulin pump. If they fail, someone dies.
      - hsbauauvhabzb 2y ago
        But at the same time, auditing every update to an assurance level beyond ‘it didn’t bsod in test’ is incredibly hard. I don’t disagree with anything you’ve said, but I’d be very interested in solving the problem of actually auditing constant updates from vendors.
        withinboredom 2y ago
        Even a rudimentary “delay autoupdate by two weeks” would have saved lives here. Let everyone else update first.
        exe34 2y ago
        staged releases. don't cripple all your systems in one go. hot backups that you only update after the main system isn't dead from an update.
        com 2y ago
        Automated CI/CD - many of us already do this hundreds of times a day. If you’re an emergency call centre, join a consortium of similar orgs and standardise tech and do it properly. Defer updates. Most things can wait 8-12 hours. Even more can wait 3 weeks (did this for all but security-critical npm package updates in one place). Demand legal changes to ensure fair liability for failure to undertake basic measures by service providers for paid software and services. Demand proper liability for C-suites not ensuring that actual risk management is in place instead of stupid box-ticking. Design better software. Seriously, the kinds of half-baked stuff that costs so much is incredible. It doesn’t take longer, and it doesn’t cost more to do things right, the only change is that management needs to be engaged with outcomes and have skin in the game. Execs should run the risk of going to jail for egregious failures.
- notepad0x90 2y ago
  Crowdstrike does not have a shortage of competitors, there are plenty in this field, at least a dozen. They just happen to be the best, primarily because of their intel, and you wouldn't believe this but because their sensor has always been very lightweight and easy to manage. The article is barely scratching the surface, supply chain security is a lot more scary than this.
  - mylastattempt 2y ago
    My last comment was solely made to point out that this article is a garbage article. As for competitors and the dangers of supply chain attacks, I'm with you.
  - khafra 2y ago
    > They just happen to be the best, primarily because of their intel, and you wouldn't believe this but because their sensor has always been very lightweight and easy to manage. I think an important part of their dominance is that they're the zero configuration EDR. Buy Carbon Black, and you'll need to know something about what's installed in your environment, what you do and don't want running. Buy SentinelOne, and you need to know that plus you have to learn a complex product. But with CS, you pretty much just install it, then get that sweet, sweet green checkmark on your security audit.
    - notepad0x90 2y ago
      You nailed it. It is a sweet green check mark but it also works well. The main reason CISO's get it so that if they ever do get compromised they can say they purchased the best security product out there. but there are still a certain portion of their customers that get targeted by APT's, think defense contractors, government orgs like the NRO, oil and gas, finance, power companies that have a large budget and that checkbox is the least of their worries.
- xqt40 2y ago
  You are right about that. The company seems to have bad QA process and that is not something that can be improved easily as it requires change in culture and management.
mikewarot 2y ago
I'm a grumpy old man on the internet.... let's just get that out of the way The root cause is NOT capitalism, nor is it users, Microsoft, or even CrowdStrike. You can't legislate, regulate, or "be more careful next time" your way out of this. Hell, blaming the users won't even work. Here are 3 stories: --- Imagine yourself as an inspector for the Army. The 17th Fortress has exploded this month, and nobody can figure out why. You've checked all the surviving off-site records, and are reasonably sure that the crates of dynamite that used to make up the foundations and structure of the cart were properly inspected, and even updated on a regular basis. You more closely inspect the records, looking for any possible soldier or supplier who might have caused this loss. It might possibly be communist infiltration, or one of those pacifists! You encounter an old civilian, who remembers a time when forts were built out of wood or bricks, and suggests that. But he's not a professional solder, what could he know. --- Imagine you're a fire inspector. You've been to your 4th case this month of complete electrical network outage. This time, the cause seems to be that Lisa Douglas at Green Acres had Eb Dawson climb the pole, and he plugged in one too many appliances to the electricial. If only there were a way to make sure that an overload anywhere couldn't take down the grid, and ruin so many people's days. You desperately want a day without house fires, and so many linemen being called out to test and repair circuits before connecting them back to the grid. It will take some time before the boilers and generators get back on line from their cold re-start. In the mean while, business in town has ground to a halt. The paperwork and processes to track and certify each appliance doesn't seem efficient. There's this grumpy old guy who talks about fuses and circuit breakers, but he's just a crank. --- The United States found itself embedded in yet another foreign entanglement in VietNam. There was a severe problem planning air strikes, because there were multiple sources required to plan them, and no single computer could be trusted with both of them. The strikes themselves were classified, but the locations of the enemy radar installations couldn't be trusted to the computers, because they were occasionally accessed by enemy sources. Thus the methods and means of locating the enemy radar equipment could become known, and thus rendered ineffective. A study was done[1], and the problems were solved. There were systems based on the results of these studies[5], and they worked well.[2] Unfortunately, people thought that it was un-necessary to incorporate these measures, and they defaulted to the broken ambient authority model we're stuck with today. Here's some more reading, if you're interested.[3] --- If you're bored... I've even got a conspiracy theory that explains how I think we actually got here, it it wasn't simply historical forces (which I think it was, 95% certainty).[4] If true, those forces would still be here today, actively suppressing any such stories. [1] https://csrc.nist.rip/publications/history/ande72.pdf https://csrc.nist.rip/publications/history/ande72.pdf [2] https://srl.cs.jhu.edu/pubs/SRL2003-02.pdf https://srl.cs.jhu.edu/pubs/SRL2003-02.pdf [3] https://github.com/dckc/awesome-ocap https://github.com/dckc/awesome-ocap [4] https://news.ycombinator.com/item?id=40107150 https://news.ycombinator.com/item?id=40107150 [5] https://web.archive.org/web/20120919111301/http://www.albany.edu/acc/courses/ia/classics/belllapadula1.pdf https://web.archive.org/web/20120919111301/http://www.albany...
jpgvm 2y ago
The correct lesson is to stop introducing more vulns into your systems by running "security" products. Crowdstrike was just an outage but could have just has easily been Solarwinds 2.0. Crowdstrike is probably less bad than the alternatives that I have run into that are largely developed by very low cost engineers cough TrendMicro cough but even so, they aren't NT kernel engineers nor do they have the NT kernel release process. Companies need to find ways to live without this crap or this will keep happening and it will be a lot worse one day. Self-compromising your own systems with RATs/MDMs/EDR/XDR/whatever other acronym soup needed to please the satanic CISSPs are just terrible ideas in general.
- ravenstine 2y ago
  The difference between a vulnerability and a bug is slim. This is why I don't immediately install every single software update; official code is more likely to cut me off from my livelihood than an actual exploit taking place.
- patrakov 2y ago
  I think that Russians are now in a unique position, having learned this lesson almost universally due to sanctions. Any foreign vendor of self-updating software can, in theory, be ordered to kill your system remotely or to deliberately introduce a backdoor. P.S. I do realize that the same applies to domestic products, especially to the ones mandatory to install, like Russian software that is required to be present on new smartphones sold in Russia. But my point is about people learning a lesson.
- M95D 2y ago
  You completely missed the point in the article. The correct lesson is to not use what everybody else uses. That way it won't fail the same way, at the same time.
- Semaphor 2y ago
  From many of the comments I read, the reason is often regulation.
  - 2y ago
iwontberude 2y ago
Having monopolies and oligopolies is like having a small gene pool.
akira2501 2y ago
"Was a Warning." No. It was just a failure. The warnings have been trumpeted for decades. It should have been no surprise that the giant company that was trusted to secure our single source of OS software against "supply chain attacks" ended up committing the largest "supply chain attack" yet seen on Earth. We are effectively still in the wild west. The gold rush has to end before we can truly civilize the place.
wruza 2y ago
It was a seizure, not warning. More stupid bandaids will be slapped in a hurry without considerations from people who understand how this tumor works.
jijji 2y ago
is it true that the owner of crowdstrike is an ex-employee of McAfee and the same company that got sold because they had massive downtime for basically the same reason
- usr1106 2y ago
  > CrowdStrike's CEO oversaw very similar McAfee meltdown https://news.ycombinator.com/item?id=41030352 https://news.ycombinator.com/item?id=41030352
theoa 2y ago
You are only as good as your weakest link: > A Microsoft spokesman said it cannot legally wall off its operating system in the same way Apple does because of an understanding it reached with the European Commission following a complaint. In 2009, Microsoft agreed it would give makers of security software the same level of access to Windows that Microsoft gets. https://www.wsj.com/tech/cybersecurity/microsoft-tech-outage-role-crowdstrike-50917b90?st=pkas1bzrhcoj0os&reflink=desktopwebshare_permalink https://www.wsj.com/tech/cybersecurity/microsoft-tech-outage...
- kevincox 2y ago
  Maybe Windows Defender should also not have the ability to crash the kernel. Instead Microsoft can provider proper hooks to pluggable drivers that can be used by both themselves and third party AV.
- usr1106 2y ago
  It's not Microsoft that should wall off the operating system. It's banks, airlines, health care providers that should not use Windows the way they currently do. No employee there needs the possibility to install any software themselves. Without the possibility to install software you don't need anti-virus software. These systems should just run immutable images, in A/B deployment, just in case the new image is broken. Of course that does solve the supply chain security. How do you make sure that the images contain know malware? But the problem does not not need to be addressed on millions of machines with millions of employees. It gets reduced to thousands.
  - wpm 2y ago
    Presumably, these employees probably need email, web access, and file sharing. There you go. Those are your threat vectors now.
  - xqt40 2y ago
    Your suggestion simply shows that you have no understanding of how things work. Terminals already does not allow users to install anything. As for rest of workstations and work laptops - users already don't install anything on them. The issue with Crowdstrike is not with users but with service that is maintaing these computers. A very frightening thing that all those companies are dependant on the f*ck off of their service provider and it costs them all their business. No viruses need installation - in fact it would be easiest thing if viruses were listed among installed programms. Are yiu representing your whole generation or you are only one such strange person? Also your suggestion os outdated by at least 60 years as it assumes that hardware that has no software update capabilities can't be hacked...
teeheelol 2y ago
The warning in this case is hire security people who actually have a clue and include vendor software in their risk assessment. Literally every time I see stuff like this go down, the security software had exactly zero engineering research put into it whereas everything else did. If people did this, CrowdStrike would either not exist or look completely different.
gquere 2y ago
The article advocates for even more market fragmentation? Even though that isn't the issue at all?
Falkon1313 2y ago
What baffles me is just how many IT personnel in so many organizations around the world apparently just blindly hit the "Deploy this zero-day update to all production systems without any testing" button instead of the "Test this update on our test systems first" button. Or maybe even just looking up the update online to see whether any problems had been reported before deploying it wholesale across their organizations. Are these the same IT people whose systems all went offline in the left-pad incident because they 'accidentally' set their production servers to be dependent on a third-party repository? I've worked at some low-budget places that didn't have much in the way of a vetting process, but even there auto-deploying unknown updates to third-party dependencies into production was always a capital N No.
- newdee 2y ago
  They didn’t. Most orgs run with at least an n-1 Sensor version, with test groups for latest. This was essentially a definitions update pushed by Crowdstrike to all customers, regardless of the deployed sensor version.
- TiredOfLife 2y ago
  This update bypassed all methods of controlling updates.
blooalien 2y ago
Sorry, but if decades of warnings from qualified security actual experts who were hired specifically for their expertise in such matters went ignored enough for long enough to reach this point, then this incident isn't gonna change much anything. It'll be news for a short while, then forgotten. No lessons will have been learned, and few if any changes will be made. More things like this will happen in the future. Guaranteed...
rho138 2y ago
[flagged]
mannyv 2y ago
If these machines were backend systems (which most of the ones that mattered were), you have to ask: why are they running malware detection when they should have minimal-to-no surface area for an attacker? That's the real question here.
zelon88 2y ago
The culture at MS$ is to servitize enough of their products and then force customers to use them. That way, the products won't be so exposed to users and Microsoft will be able to limit their own liability without actually improving the back end product. Servitization is a clever way to consolidate your perpetual licensed customers over to perpetual service contacts, while also further obfuscating and locking down the underlying operating environment. This is in the best interest of Microsoft bottom line, at the expense of all private business, government, or anyone who values consumer experience really. It reduces the number of drive-by security incidents, but when WW3 happens and 75% of our economy is hosted in a whopping 12 datacenters across 3 companies I'm sure we'll be screwed. I mean just depth charging Google fiber today would probably take down 25% of the world economy.

112 points by Kaibeezy 2y ago

gssa 2y ago

[dead]

Kaibeezy 2y ago

https://archive.md/pyW91 https://archive.md/pyW91

mylastattempt 2y ago

I found the article unbearable and just a convoluted way to say: this incident would have had a lot less impact if CrowdStrike had less customers or more competitors. A real page filler without any insight or solutions, just look at this paragraph, completely void of anything useful. > This time, the digital cataclysm was caused by well-intentioned people who made a mistake. That meant the fix came relatively quickly; CrowdStrike knew what had gone wrong. But we may not be so lucky next time. If a malicious actor had attacked CrowdStrike or a similarly essential bit of digital infrastructure, the disaster could have been much worse. Gee, the damage from an honest mistake (what does the author even base that on) is most likely easier to fix than the damage done by a malicious actor with bad intent. I feel so enlightened!

hsbauauvhabzb 2y ago
> mistake Funny way of spelling ‘complete failure in functional testing before going to prod causing billions of dollars worth of damage likely due to criminal level negligence’.
- tuatoru 2y ago
  Billions is probably an underestimate. When you factor on the time spent in meetings discussing endlessly what to do about this in corporations around the world, besides the cancelled flights and banking and hospital and 911 mayhem, a trillion isn't indefensible. Who needs North Korean hackers?
  - hsbauauvhabzb 2y ago
    If crowdstrikes internal competency failed to detect a consistent bsod, I doubt they’d notice North Korea dropping malware into their product solarwinds style.
- Galanwe 2y ago
  I mean, we've seen impacts of these unstaged updates in the deeper layers of banks, airports and hospitals. It's unfair that CrowdStrike gets more than half the blame here. Not to mention, probably half these companies were actually forced to install this. Getting SOC2 is essentially letting a senior-branded KPMG or EY intern tell you how to enforce security policies based on their checklist. There is far more to blame in this incident than CrowdStrike themselves.
  - aiisjustanif 2y ago
    Compliance is one thing but a cybersecurity incident response team is very blind without an EDR not to mention you would have less protection from insider threat.
- antonvs 2y ago
  That's true. But it's also true that companies depending on more or less random third parties to provide them with kernel modules for a monolithic kernel - and then giving them control over auto-updating those kernel modules - is beyond dumb.
- HeatrayEnjoyer 2y ago
  Undoubtedly killing people too: 911 networks failed. Hospitals were crippled. Short of getting your hands on a nuclear football it's difficult to imagine an act with a higher effort:devastation ratio.
  - RCitronsBroker 2y ago
    You’re not wrong about the end result, but the breakdown of systems this complex goes deeper than placing the blame on some CrowdStrike employee. Whoever thought up the great idea to allow auto-update-able kernel modules for something as mission critical as emergency response or healthcare deserves just as much blame. I’ve worked in healthcare for my whole career, this is madness. Not that their process is without flaw, but can we remind ourselves of how stringently we assess medical devices? I cannot imagine it’s controversial to say that emergency response equipment is every bit as critical as a insulin pump. If they fail, someone dies.
    - hsbauauvhabzb 2y ago
      But at the same time, auditing every update to an assurance level beyond ‘it didn’t bsod in test’ is incredibly hard. I don’t disagree with anything you’ve said, but I’d be very interested in solving the problem of actually auditing constant updates from vendors.
      - withinboredom 2y ago
        Even a rudimentary “delay autoupdate by two weeks” would have saved lives here. Let everyone else update first.
      - exe34 2y ago
        staged releases. don't cripple all your systems in one go. hot backups that you only update after the main system isn't dead from an update.
      - com 2y ago
        Automated CI/CD - many of us already do this hundreds of times a day. If you’re an emergency call centre, join a consortium of similar orgs and standardise tech and do it properly. Defer updates. Most things can wait 8-12 hours. Even more can wait 3 weeks (did this for all but security-critical npm package updates in one place). Demand legal changes to ensure fair liability for failure to undertake basic measures by service providers for paid software and services. Demand proper liability for C-suites not ensuring that actual risk management is in place instead of stupid box-ticking. Design better software. Seriously, the kinds of half-baked stuff that costs so much is incredible. It doesn’t take longer, and it doesn’t cost more to do things right, the only change is that management needs to be engaged with outcomes and have skin in the game. Execs should run the risk of going to jail for egregious failures.
notepad0x90 2y ago
Crowdstrike does not have a shortage of competitors, there are plenty in this field, at least a dozen. They just happen to be the best, primarily because of their intel, and you wouldn't believe this but because their sensor has always been very lightweight and easy to manage. The article is barely scratching the surface, supply chain security is a lot more scary than this.
- mylastattempt 2y ago
  My last comment was solely made to point out that this article is a garbage article. As for competitors and the dangers of supply chain attacks, I'm with you.
- khafra 2y ago
  > They just happen to be the best, primarily because of their intel, and you wouldn't believe this but because their sensor has always been very lightweight and easy to manage. I think an important part of their dominance is that they're the zero configuration EDR. Buy Carbon Black, and you'll need to know something about what's installed in your environment, what you do and don't want running. Buy SentinelOne, and you need to know that plus you have to learn a complex product. But with CS, you pretty much just install it, then get that sweet, sweet green checkmark on your security audit.
  - notepad0x90 2y ago
    You nailed it. It is a sweet green check mark but it also works well. The main reason CISO's get it so that if they ever do get compromised they can say they purchased the best security product out there. but there are still a certain portion of their customers that get targeted by APT's, think defense contractors, government orgs like the NRO, oil and gas, finance, power companies that have a large budget and that checkbox is the least of their worries.
xqt40 2y ago
You are right about that. The company seems to have bad QA process and that is not something that can be improved easily as it requires change in culture and management.

mikewarot 2y ago

I'm a grumpy old man on the internet.... let's just get that out of the way The root cause is NOT capitalism, nor is it users, Microsoft, or even CrowdStrike. You can't legislate, regulate, or "be more careful next time" your way out of this. Hell, blaming the users won't even work. Here are 3 stories: --- Imagine yourself as an inspector for the Army. The 17th Fortress has exploded this month, and nobody can figure out why. You've checked all the surviving off-site records, and are reasonably sure that the crates of dynamite that used to make up the foundations and structure of the cart were properly inspected, and even updated on a regular basis. You more closely inspect the records, looking for any possible soldier or supplier who might have caused this loss. It might possibly be communist infiltration, or one of those pacifists! You encounter an old civilian, who remembers a time when forts were built out of wood or bricks, and suggests that. But he's not a professional solder, what could he know. --- Imagine you're a fire inspector. You've been to your 4th case this month of complete electrical network outage. This time, the cause seems to be that Lisa Douglas at Green Acres had Eb Dawson climb the pole, and he plugged in one too many appliances to the electricial. If only there were a way to make sure that an overload anywhere couldn't take down the grid, and ruin so many people's days. You desperately want a day without house fires, and so many linemen being called out to test and repair circuits before connecting them back to the grid. It will take some time before the boilers and generators get back on line from their cold re-start. In the mean while, business in town has ground to a halt. The paperwork and processes to track and certify each appliance doesn't seem efficient. There's this grumpy old guy who talks about fuses and circuit breakers, but he's just a crank. --- The United States found itself embedded in yet another foreign entanglement in VietNam. There was a severe problem planning air strikes, because there were multiple sources required to plan them, and no single computer could be trusted with both of them. The strikes themselves were classified, but the locations of the enemy radar installations couldn't be trusted to the computers, because they were occasionally accessed by enemy sources. Thus the methods and means of locating the enemy radar equipment could become known, and thus rendered ineffective. A study was done[1], and the problems were solved. There were systems based on the results of these studies[5], and they worked well.[2] Unfortunately, people thought that it was un-necessary to incorporate these measures, and they defaulted to the broken ambient authority model we're stuck with today. Here's some more reading, if you're interested.[3] --- If you're bored... I've even got a conspiracy theory that explains how I think we actually got here, it it wasn't simply historical forces (which I think it was, 95% certainty).[4] If true, those forces would still be here today, actively suppressing any such stories. [1] https://csrc.nist.rip/publications/history/ande72.pdf https://csrc.nist.rip/publications/history/ande72.pdf [2] https://srl.cs.jhu.edu/pubs/SRL2003-02.pdf https://srl.cs.jhu.edu/pubs/SRL2003-02.pdf [3] https://github.com/dckc/awesome-ocap https://github.com/dckc/awesome-ocap [4] https://news.ycombinator.com/item?id=40107150 https://news.ycombinator.com/item?id=40107150 [5] https://web.archive.org/web/20120919111301/http://www.albany.edu/acc/courses/ia/classics/belllapadula1.pdf https://web.archive.org/web/20120919111301/http://www.albany...

jpgvm 2y ago

The correct lesson is to stop introducing more vulns into your systems by running "security" products. Crowdstrike was just an outage but could have just has easily been Solarwinds 2.0. Crowdstrike is probably less bad than the alternatives that I have run into that are largely developed by very low cost engineers cough TrendMicro cough but even so, they aren't NT kernel engineers nor do they have the NT kernel release process. Companies need to find ways to live without this crap or this will keep happening and it will be a lot worse one day. Self-compromising your own systems with RATs/MDMs/EDR/XDR/whatever other acronym soup needed to please the satanic CISSPs are just terrible ideas in general.

ravenstine 2y ago
The difference between a vulnerability and a bug is slim. This is why I don't immediately install every single software update; official code is more likely to cut me off from my livelihood than an actual exploit taking place.
patrakov 2y ago
I think that Russians are now in a unique position, having learned this lesson almost universally due to sanctions. Any foreign vendor of self-updating software can, in theory, be ordered to kill your system remotely or to deliberately introduce a backdoor. P.S. I do realize that the same applies to domestic products, especially to the ones mandatory to install, like Russian software that is required to be present on new smartphones sold in Russia. But my point is about people learning a lesson.
M95D 2y ago
You completely missed the point in the article. The correct lesson is to not use what everybody else uses. That way it won't fail the same way, at the same time.
Semaphor 2y ago
From many of the comments I read, the reason is often regulation.
- 2y ago

iwontberude 2y ago

Having monopolies and oligopolies is like having a small gene pool.

akira2501 2y ago

"Was a Warning." No. It was just a failure. The warnings have been trumpeted for decades. It should have been no surprise that the giant company that was trusted to secure our single source of OS software against "supply chain attacks" ended up committing the largest "supply chain attack" yet seen on Earth. We are effectively still in the wild west. The gold rush has to end before we can truly civilize the place.

wruza 2y ago

It was a seizure, not warning. More stupid bandaids will be slapped in a hurry without considerations from people who understand how this tumor works.

jijji 2y ago

is it true that the owner of crowdstrike is an ex-employee of McAfee and the same company that got sold because they had massive downtime for basically the same reason

usr1106 2y ago
> CrowdStrike's CEO oversaw very similar McAfee meltdown https://news.ycombinator.com/item?id=41030352 https://news.ycombinator.com/item?id=41030352

theoa 2y ago

You are only as good as your weakest link: > A Microsoft spokesman said it cannot legally wall off its operating system in the same way Apple does because of an understanding it reached with the European Commission following a complaint. In 2009, Microsoft agreed it would give makers of security software the same level of access to Windows that Microsoft gets. https://www.wsj.com/tech/cybersecurity/microsoft-tech-outage-role-crowdstrike-50917b90?st=pkas1bzrhcoj0os&reflink=desktopwebshare_permalink https://www.wsj.com/tech/cybersecurity/microsoft-tech-outage...

kevincox 2y ago
Maybe Windows Defender should also not have the ability to crash the kernel. Instead Microsoft can provider proper hooks to pluggable drivers that can be used by both themselves and third party AV.
usr1106 2y ago
It's not Microsoft that should wall off the operating system. It's banks, airlines, health care providers that should not use Windows the way they currently do. No employee there needs the possibility to install any software themselves. Without the possibility to install software you don't need anti-virus software. These systems should just run immutable images, in A/B deployment, just in case the new image is broken. Of course that does solve the supply chain security. How do you make sure that the images contain know malware? But the problem does not not need to be addressed on millions of machines with millions of employees. It gets reduced to thousands.
- wpm 2y ago
  Presumably, these employees probably need email, web access, and file sharing. There you go. Those are your threat vectors now.
- xqt40 2y ago
  Your suggestion simply shows that you have no understanding of how things work. Terminals already does not allow users to install anything. As for rest of workstations and work laptops - users already don't install anything on them. The issue with Crowdstrike is not with users but with service that is maintaing these computers. A very frightening thing that all those companies are dependant on the f*ck off of their service provider and it costs them all their business. No viruses need installation - in fact it would be easiest thing if viruses were listed among installed programms. Are yiu representing your whole generation or you are only one such strange person? Also your suggestion os outdated by at least 60 years as it assumes that hardware that has no software update capabilities can't be hacked...

teeheelol 2y ago

The warning in this case is hire security people who actually have a clue and include vendor software in their risk assessment. Literally every time I see stuff like this go down, the security software had exactly zero engineering research put into it whereas everything else did. If people did this, CrowdStrike would either not exist or look completely different.

gquere 2y ago

The article advocates for even more market fragmentation? Even though that isn't the issue at all?

Falkon1313 2y ago

What baffles me is just how many IT personnel in so many organizations around the world apparently just blindly hit the "Deploy this zero-day update to all production systems without any testing" button instead of the "Test this update on our test systems first" button. Or maybe even just looking up the update online to see whether any problems had been reported before deploying it wholesale across their organizations. Are these the same IT people whose systems all went offline in the left-pad incident because they 'accidentally' set their production servers to be dependent on a third-party repository? I've worked at some low-budget places that didn't have much in the way of a vetting process, but even there auto-deploying unknown updates to third-party dependencies into production was always a capital N No.

newdee 2y ago
They didn’t. Most orgs run with at least an n-1 Sensor version, with test groups for latest. This was essentially a definitions update pushed by Crowdstrike to all customers, regardless of the deployed sensor version.
TiredOfLife 2y ago
This update bypassed all methods of controlling updates.

blooalien 2y ago

Sorry, but if decades of warnings from qualified security actual experts who were hired specifically for their expertise in such matters went ignored enough for long enough to reach this point, then this incident isn't gonna change much anything. It'll be news for a short while, then forgotten. No lessons will have been learned, and few if any changes will be made. More things like this will happen in the future. Guaranteed...

rho138 2y ago

[flagged]

mannyv 2y ago

If these machines were backend systems (which most of the ones that mattered were), you have to ask: why are they running malware detection when they should have minimal-to-no surface area for an attacker? That's the real question here.

zelon88 2y ago

The culture at MS$ is to servitize enough of their products and then force customers to use them. That way, the products won't be so exposed to users and Microsoft will be able to limit their own liability without actually improving the back end product. Servitization is a clever way to consolidate your perpetual licensed customers over to perpetual service contacts, while also further obfuscating and locking down the underlying operating environment. This is in the best interest of Microsoft bottom line, at the expense of all private business, government, or anyone who values consumer experience really. It reduces the number of drive-by security incidents, but when WW3 happens and 75% of our economy is hosted in a whopping 12 datacenters across 3 companies I'm sure we'll be screwed. I mean just depth charging Google fiber today would probably take down 25% of the world economy.