5 ms·

How CrowdStrike Stopped Everything

28 points by pseudolus 2y ago

nitinreddy88 2y ago
Any new information available here? Sorry couldn't find anything new than what world knows so far.
- duxup 2y ago
  No new information, just a reporter collecting quotes and pasting them onto some basics.
  - deleted 2y ago
    [deleted]
portn0y 2y ago
[flagged]
- beernet 2y ago
  [flagged]
  - portn0y 2y ago
    [flagged]
- nine_zeros 2y ago
  > When do we get rid of the traditional finance mental models from history like we are religious mental models from history? Customers need to stop buying from companies that are publicly listed. Publicly listed companies are designed for the next quarter - cutting corner is the name of the game. Also see: Boeing, GE, IBM, and recently Meta products that are basically unusable.
  - portn0y 2y ago
    That’s far too reductive a solution. Too single variable. “Boeing” is a made up idea. Engineering an aircraft is not. Make HR a government agency, to remove redundancy and extract all gains from a rich minority who aren’t owed the allegiance of 100s of millions and billions across the globe. American Civic Life: The Story is not one where a handful did all the work, are worth all the prestige. Any one of them dies tonight we all have to move on. This system is so “War is peace.” And plenty of educated geniuses are in on it. If they can’t cope with actual science driven rational take on their value to humanity then more war less peace is what they’ll get. Humans are not philosophy driven they’re biology driven and too many are having their biology strained to the point of snapping
  - dhdiajaxbsjsj 2y ago
    > Customers need to stop buying from companies that are publicly listed This is not going to happen. It’s much easier for a smaller group to coordinate than a larger group. It’s the same reason democracy is so easily exploitable. “Just vote them out” and yet the US has been involved in wars for the entirety of many of our lives. Fixing our culture and installing elites that uphold it is the only answer. Not that those are easy things to do. Religion was how this was done in the past (and is done in other areas).
    - efitz 2y ago
      If you think about it, you can broadly classify (at least in US politics) right vs left as “which large groups you trust”. For example, republicans tend to trust corporations of any size and tend to distrust large government. The opposite is true of Democrats; they tend to want to centralize power in government and tend to distrust large corporations. The fallacy on both sides is that the problem isn’t the kind of organization, it’s the centralization of power. Personally I distrust any large group with any sort of power, be it economic or monopoly on use of force or whatever. The solution for government is “give the government less things to control” - easier said than done but the Constitution laid it out for us. I have been thinking about solutions for corporations, and I have come up with two ideas: 1. A progressive tax on corporations based on number of employees and/or contractors of any sort contributing to the corporation. 2. A progressive tax on corporations based on economic power, eg market capitalization and/or gross revenue and/or subscribers or interactive users. Or all of the above. The idea is to make corporations naturally less profitable as they grow, which would make centralization/monopolization financially unattractive for investors. In such a tax regime maybe we wouldn’t even need to take antitrust actions anymore as it would become self-regulating if we got the rates right. Just a thought.
- markus_zhang 2y ago
  I don't really think it's possible to get rid of the finance mental models. Maybe momentarily at the best -- and that's why excellent engineers leave when "the company becomes too big". Everyone living in the real world has to think about his/her own financials and companies are obliged to do that too.
  - chii 2y ago
    but there's the principal-agent problem (https://www.investopedia.com/terms/p/principal-agent-problem.asp https://www.investopedia.com/terms/p/principal-agent-problem...) laying in there somewhere. The people making financial decisions are incentivized to minimize the financial cost (or maximize the gains), but only for some time period that's shorter than the life-time of the company, which, as far as the owners/shareholders are concerned, should be infinite. So this has an inherent bias against long term decisions that would've been better, but makes short term look bad, in favour of better short term decisions. Then the agent just leaves for greener pasture after their "short term" is over.
    - markus_zhang 2y ago
      I don't think we disagree on this? Maybe I misread something...
jmclnx 2y ago
I know people many suffered with this and in some cases the suffering was real. For most people it was just an inconvenience. And I am a bit ashamed to say, the panic from users users who could not get to Office 365 or their Windows PC cannot boot still brings a bit of a smile to my face. I think there is a German word for this :) A relative worked from my house for those days because their internet was also broken. We worked for 2 different companies, me on a Linux Workstation, him on Windows 11 spending time with his help desk with me "translating". One lesson learned, the Help Desk people should be trained to avoid Tech Words when dealing with people who's workflow is just Email and Excel. Both the Help Desk and my relative had nothing but a high level of frustration dealing with each other.
- cedws 2y ago
  >I think there is a German word for this :) That would be ‘schadenfreude.’
- deleted 2y ago
  [deleted]
- jajko 2y ago
  If frustration or suffering of other people brings a good smile on your face, just because their employer has different IT policies compared to yours, that's not really cool or something to brag about, rather contrary.
cosmin800 2y ago
Not everything, there are companies that do not own, or use any single windows machine for this exact reason waiting to happen. Was this worse than msblaster or? I remember that, painful.
duxup 2y ago
No real insights here, just high level explanation and empty quotes from folks. > The root cause analysis (RCA) means that a CrowdStrike programmer(s) did not check their inputs before pushing an update to the CrowdStrike Falcon Windows Sensor in production. How is it this isn’t just automated and / or that update automatically run in a VM or something and when it crashed the rollout prevented? It’s not a new concept…
- moffkalast 2y ago
  These sort of things happen occasionally despite all the possible safeguards you can set up. It's just gonna be something else you didn't predict and frankly the wrong thing to focus on. The real problem this exposed is that somehow it's apparently the law for corporations to introduce a single point of failure into their entire IT system with absolutely zero fallback capacity or any workarounds at all. It's just entropy meeting stupidity. It's like it was mandated for planes to have a built in off switch that can be triggered remotely and people then blame the company for accidentally triggering it, crashing every single one and killing everyone, instead of asking why the fuck do all planes have an off switch?!
  - duxup 2y ago
    My understanding is this update crashed very consistently.
    - Jtsummers 2y ago
      Yep. Every Windows machine in our company that was online during the update period was brought down. The only ones to survive were offline for some reason (off, sleep mode, people traveling with them, on an air gapped network, etc.).
    - moffkalast 2y ago
      Yeah it probably could've been caught. Major disasters are always this way, someone does something they shouldn't have, two people that should've checked were on vacation and the third one was asleep, and a manager saying let's goooo because we're in a hurry and way behind! Come on, just skip the checklist this once, it'll be fine. It's always a perfect storm and it will always happen, we can just reduce the frequency with more and more safeguards. If something is so critical that people will die if it doesn't work, it needs backups and probably secondary backups, end of story. Hospitals that were brought down with this have no excuse.
mindslight 2y ago
The article didn't even touch on the social factor of insurers and other types of middle managers uniformly pushing everyone to install an unnecessary RCE-based piece of software to check off their finely crafted bullet points that demand centralized legibility at the expense of everything else. When Microsoft does something that knocks some large amount of systems out, it's at least understandable why such a monoculture exists. But this state of affairs was entirely self inflicted. And an article in CACM should really be addressing these factors, because everybody already knows Crowdstroke itself was supremely incompetent. The question is not how Crowdstroke can prevent this type of software bug from happening again, but rather how we as a society can prevent the creation of more centralizing companies like Crowdstroke, especially ones that leverage the regulatory apparatus to drive adoption of their top-down version of "security".

28 points by pseudolus 2y ago

nitinreddy88 2y ago

Any new information available here? Sorry couldn't find anything new than what world knows so far.

duxup 2y ago
No new information, just a reporter collecting quotes and pasting them onto some basics.
- deleted 2y ago
  [deleted]

portn0y 2y ago

[flagged]

beernet 2y ago
[flagged]
- portn0y 2y ago
  [flagged]
nine_zeros 2y ago
> When do we get rid of the traditional finance mental models from history like we are religious mental models from history? Customers need to stop buying from companies that are publicly listed. Publicly listed companies are designed for the next quarter - cutting corner is the name of the game. Also see: Boeing, GE, IBM, and recently Meta products that are basically unusable.
- portn0y 2y ago
  That’s far too reductive a solution. Too single variable. “Boeing” is a made up idea. Engineering an aircraft is not. Make HR a government agency, to remove redundancy and extract all gains from a rich minority who aren’t owed the allegiance of 100s of millions and billions across the globe. American Civic Life: The Story is not one where a handful did all the work, are worth all the prestige. Any one of them dies tonight we all have to move on. This system is so “War is peace.” And plenty of educated geniuses are in on it. If they can’t cope with actual science driven rational take on their value to humanity then more war less peace is what they’ll get. Humans are not philosophy driven they’re biology driven and too many are having their biology strained to the point of snapping
- dhdiajaxbsjsj 2y ago
  > Customers need to stop buying from companies that are publicly listed This is not going to happen. It’s much easier for a smaller group to coordinate than a larger group. It’s the same reason democracy is so easily exploitable. “Just vote them out” and yet the US has been involved in wars for the entirety of many of our lives. Fixing our culture and installing elites that uphold it is the only answer. Not that those are easy things to do. Religion was how this was done in the past (and is done in other areas).
  - efitz 2y ago
    If you think about it, you can broadly classify (at least in US politics) right vs left as “which large groups you trust”. For example, republicans tend to trust corporations of any size and tend to distrust large government. The opposite is true of Democrats; they tend to want to centralize power in government and tend to distrust large corporations. The fallacy on both sides is that the problem isn’t the kind of organization, it’s the centralization of power. Personally I distrust any large group with any sort of power, be it economic or monopoly on use of force or whatever. The solution for government is “give the government less things to control” - easier said than done but the Constitution laid it out for us. I have been thinking about solutions for corporations, and I have come up with two ideas: 1. A progressive tax on corporations based on number of employees and/or contractors of any sort contributing to the corporation. 2. A progressive tax on corporations based on economic power, eg market capitalization and/or gross revenue and/or subscribers or interactive users. Or all of the above. The idea is to make corporations naturally less profitable as they grow, which would make centralization/monopolization financially unattractive for investors. In such a tax regime maybe we wouldn’t even need to take antitrust actions anymore as it would become self-regulating if we got the rates right. Just a thought.
markus_zhang 2y ago
I don't really think it's possible to get rid of the finance mental models. Maybe momentarily at the best -- and that's why excellent engineers leave when "the company becomes too big". Everyone living in the real world has to think about his/her own financials and companies are obliged to do that too.
- chii 2y ago
  but there's the principal-agent problem (https://www.investopedia.com/terms/p/principal-agent-problem.asp https://www.investopedia.com/terms/p/principal-agent-problem...) laying in there somewhere. The people making financial decisions are incentivized to minimize the financial cost (or maximize the gains), but only for some time period that's shorter than the life-time of the company, which, as far as the owners/shareholders are concerned, should be infinite. So this has an inherent bias against long term decisions that would've been better, but makes short term look bad, in favour of better short term decisions. Then the agent just leaves for greener pasture after their "short term" is over.
  - markus_zhang 2y ago
    I don't think we disagree on this? Maybe I misread something...

jmclnx 2y ago

I know people many suffered with this and in some cases the suffering was real. For most people it was just an inconvenience. And I am a bit ashamed to say, the panic from users users who could not get to Office 365 or their Windows PC cannot boot still brings a bit of a smile to my face. I think there is a German word for this :) A relative worked from my house for those days because their internet was also broken. We worked for 2 different companies, me on a Linux Workstation, him on Windows 11 spending time with his help desk with me "translating". One lesson learned, the Help Desk people should be trained to avoid Tech Words when dealing with people who's workflow is just Email and Excel. Both the Help Desk and my relative had nothing but a high level of frustration dealing with each other.

cedws 2y ago
>I think there is a German word for this :) That would be ‘schadenfreude.’
deleted 2y ago
[deleted]
jajko 2y ago
If frustration or suffering of other people brings a good smile on your face, just because their employer has different IT policies compared to yours, that's not really cool or something to brag about, rather contrary.

cosmin800 2y ago

Not everything, there are companies that do not own, or use any single windows machine for this exact reason waiting to happen. Was this worse than msblaster or? I remember that, painful.

duxup 2y ago

No real insights here, just high level explanation and empty quotes from folks. > The root cause analysis (RCA) means that a CrowdStrike programmer(s) did not check their inputs before pushing an update to the CrowdStrike Falcon Windows Sensor in production. How is it this isn’t just automated and / or that update automatically run in a VM or something and when it crashed the rollout prevented? It’s not a new concept…

moffkalast 2y ago
These sort of things happen occasionally despite all the possible safeguards you can set up. It's just gonna be something else you didn't predict and frankly the wrong thing to focus on. The real problem this exposed is that somehow it's apparently the law for corporations to introduce a single point of failure into their entire IT system with absolutely zero fallback capacity or any workarounds at all. It's just entropy meeting stupidity. It's like it was mandated for planes to have a built in off switch that can be triggered remotely and people then blame the company for accidentally triggering it, crashing every single one and killing everyone, instead of asking why the fuck do all planes have an off switch?!
- duxup 2y ago
  My understanding is this update crashed very consistently.
  - Jtsummers 2y ago
    Yep. Every Windows machine in our company that was online during the update period was brought down. The only ones to survive were offline for some reason (off, sleep mode, people traveling with them, on an air gapped network, etc.).
  - moffkalast 2y ago
    Yeah it probably could've been caught. Major disasters are always this way, someone does something they shouldn't have, two people that should've checked were on vacation and the third one was asleep, and a manager saying let's goooo because we're in a hurry and way behind! Come on, just skip the checklist this once, it'll be fine. It's always a perfect storm and it will always happen, we can just reduce the frequency with more and more safeguards. If something is so critical that people will die if it doesn't work, it needs backups and probably secondary backups, end of story. Hospitals that were brought down with this have no excuse.

mindslight 2y ago

The article didn't even touch on the social factor of insurers and other types of middle managers uniformly pushing everyone to install an unnecessary RCE-based piece of software to check off their finely crafted bullet points that demand centralized legibility at the expense of everything else. When Microsoft does something that knocks some large amount of systems out, it's at least understandable why such a monoculture exists. But this state of affairs was entirely self inflicted. And an article in CACM should really be addressing these factors, because everybody already knows Crowdstroke itself was supremely incompetent. The question is not how Crowdstroke can prevent this type of software bug from happening again, but rather how we as a society can prevent the creation of more centralizing companies like Crowdstroke, especially ones that leverage the regulatory apparatus to drive adoption of their top-down version of "security".