3 ms·

Fuck up the world's computers, piss off all of its IT teams, and then send people $10 uber eats gift cards as if that'll get you anything, maybe a coke at best.

Fuck up the world's computers, piss off all of its IT teams, and then send people $10 uber eats gift cards as if that'll get you anything, maybe a coke at best.. but its further admitting fault. That's like, a tip. Like here, have a sandwich while you fix our fuckup. They don't care. They'll all get 6 figure bonuses too in management for 'weathering the storm' after the mishap and probably get more money because look what they can withstand, literally technology-murder and get away with it. It's almost movie-grade evil villainy tier stuff lol

trhway 2y ago
Because people on the receiving end are the same - they accepted and rolled out the update without even as much as “canarying” it. SolarWind was the same - the customers weren’t bothered even by mismatched integrity hashes. It is a tacit pact in our industry - we all screw up and cut the slack to each other. Who will cast the first stone?
- shrubble 2y ago
  Crowdstrike can force-push an update at any time of their choosing that the connected device will grab and load, is my understanding.
  - trhway 2y ago
    Don’t you see that you’re only enforcing my point?
    - shrubble 2y ago
      No, because "canary" in the context that you used it, has a specific meaning. If you believe they should have tested CrowdStrike more or been more skeptical of their claims before licensing, that's independent of the user/administrators doing canary-style testing.
- miguelazo 2y ago
  What’s the default? And what did their technical account manager recommend? My guess, no canary ring.
- yjftsjthsd-h 2y ago
  > they accepted and rolled out the update without even as much as “canarying” it. Well, no; AIUI part of the problem was precisely that this update was pushed in such a way that it skipped any canary system in place. There might be a separate conversation to question what percentage of their users were taking advantage of its staged rollout features, but it's rather immaterial when the incident in question bypassed them even if users had configured it sensibly.
  - sisve 2y ago
    But the customer installed CS software could do this. So they are partly to blame. I do not think you will find that tesla would allow a third party update to its car. Or a oil rig would allow third party updates to critical parts of its systems. So its understanding the context. I think a lot of places this is an risk that is ok. But maybe not everywhere. And I hope some companies with critical systems will learn from this
    - yjftsjthsd-h 2y ago
      > But the customer installed CS software could do this. So they are partly to blame. It depends on if/how it was communicated. If there's a big red box in the user manual that says, "this software might take updates that completely bypass any phased rollout you configure", then yes it was probably irresponsible to use it. If, however, the software lets you configure phased rollouts and fails to mention that they might just get ignored, then I don't see how the customer can be blamed at all. (And in both cases, if CS shipped such an update with exactly zero testing whatsoever, which strains credulity but is what I've read, then they still get most of the blame.)
  - deleted 2y ago
    [deleted]
bitexploder 2y ago
Honest question, what should they do? Uninstall the company and give up? Ignoring their actual response, what is a good response to this?
- gleenn 2y ago
  If my company causes billions in damages and endangers human lives, I can't imagine why my company isn't bankrupted and dissolved.
  - IAmGraydon 2y ago
    Ok, but the market makes that decision, not the company. Crowdstrike has no choice but to accept the sentence the market hands it. It’s just that the market appears to have sentenced it to…barely anything. Blame those still using CrowdStrike after this incident.
    - xenocratus 2y ago
      If your company causes damage at society-scale (hell, even if it does major damage to one person's life), the state should be ready to intervene and make the company pay the tab for the damage they caused? Like, that doesn't sound very controversial.
      - sb8244 2y ago
        Yea. Their contracts likely have clauses for all of that. I say likely, but we already know this is true because it's come out. The thing is, crowdstrike isn't the only incompetent party here. Many major companies (looking at Delta) probably made it worse for themselves with a very poor response after. So should crowdstrike pay beyond a reasonable measure because of Delta's poor response?
        eftychis 2y ago
        No contract clause can protect you from a gross negligence tort. (Or equivalent in one's respective civil law system.) This might be the easiest gross negligence tort case to show and litigate-- still hard but if everyone starts the lawsuits they can not pull the contract to protect them. They will try of course and they will fail in most but the obvious cases. What you can not sue them for is not forseeable damages -- e.g. I lost my dream job because the computer died during the interview. But ceasing operations of a company is generally fair game. And plaintiffs can argue that no reasonable person could forsee and mitigate against this disaster so the failure is not due to plaintiff's "fault" negligence.
mywacaday 2y ago
Crowdstrike should be held accountable but so should any reasonably sized enterprise that allows code to be pushed to their whole enterprise without testing. All the large enterprises I've worked for required Windows patches to be tested before being pushed to production why are crowdstrike updates treated differently?
- __turbobrew__ 2y ago
  It clearly states in the post mortem that rapid response updates were pushed globally without any say from the customer: https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf https://www.crowdstrike.com/wp-content/uploads/2024/08/Chann...
  - mywacaday 2y ago
    So in theory if a company like crowstrike were compromised by criminals/disgruntled employee/NSA the impact could not be avoided?
- HeatrayEnjoyer 2y ago
  Given the cybersecurity landscape it is not unusual for security software updates to be pushed globally without the option to test or even stagger them. When a potent new vulnerability becomes public knowledge (or semi-insider knowledge) and especially if there's already a PoC available, organizations only have minutes to a few hours before threat actors begin utilizing it. APTs and organized crime groups have 24/7 staff to weaponize and integrate new vulnerabilities into their workflow as rapidly as possible, or have other contracted other groups to provide this service.

trhway 2y ago

Because people on the receiving end are the same - they accepted and rolled out the update without even as much as “canarying” it. SolarWind was the same - the customers weren’t bothered even by mismatched integrity hashes. It is a tacit pact in our industry - we all screw up and cut the slack to each other. Who will cast the first stone?

shrubble 2y ago
Crowdstrike can force-push an update at any time of their choosing that the connected device will grab and load, is my understanding.
- trhway 2y ago
  Don’t you see that you’re only enforcing my point?
  - shrubble 2y ago
    No, because "canary" in the context that you used it, has a specific meaning. If you believe they should have tested CrowdStrike more or been more skeptical of their claims before licensing, that's independent of the user/administrators doing canary-style testing.
miguelazo 2y ago
What’s the default? And what did their technical account manager recommend? My guess, no canary ring.
yjftsjthsd-h 2y ago
> they accepted and rolled out the update without even as much as “canarying” it. Well, no; AIUI part of the problem was precisely that this update was pushed in such a way that it skipped any canary system in place. There might be a separate conversation to question what percentage of their users were taking advantage of its staged rollout features, but it's rather immaterial when the incident in question bypassed them even if users had configured it sensibly.
- sisve 2y ago
  But the customer installed CS software could do this. So they are partly to blame. I do not think you will find that tesla would allow a third party update to its car. Or a oil rig would allow third party updates to critical parts of its systems. So its understanding the context. I think a lot of places this is an risk that is ok. But maybe not everywhere. And I hope some companies with critical systems will learn from this
  - yjftsjthsd-h 2y ago
    > But the customer installed CS software could do this. So they are partly to blame. It depends on if/how it was communicated. If there's a big red box in the user manual that says, "this software might take updates that completely bypass any phased rollout you configure", then yes it was probably irresponsible to use it. If, however, the software lets you configure phased rollouts and fails to mention that they might just get ignored, then I don't see how the customer can be blamed at all. (And in both cases, if CS shipped such an update with exactly zero testing whatsoever, which strains credulity but is what I've read, then they still get most of the blame.)
- deleted 2y ago
  [deleted]

bitexploder 2y ago

Honest question, what should they do? Uninstall the company and give up? Ignoring their actual response, what is a good response to this?

gleenn 2y ago
If my company causes billions in damages and endangers human lives, I can't imagine why my company isn't bankrupted and dissolved.
- IAmGraydon 2y ago
  Ok, but the market makes that decision, not the company. Crowdstrike has no choice but to accept the sentence the market hands it. It’s just that the market appears to have sentenced it to…barely anything. Blame those still using CrowdStrike after this incident.
  - xenocratus 2y ago
    If your company causes damage at society-scale (hell, even if it does major damage to one person's life), the state should be ready to intervene and make the company pay the tab for the damage they caused? Like, that doesn't sound very controversial.
    - sb8244 2y ago
      Yea. Their contracts likely have clauses for all of that. I say likely, but we already know this is true because it's come out. The thing is, crowdstrike isn't the only incompetent party here. Many major companies (looking at Delta) probably made it worse for themselves with a very poor response after. So should crowdstrike pay beyond a reasonable measure because of Delta's poor response?
      - eftychis 2y ago
        No contract clause can protect you from a gross negligence tort. (Or equivalent in one's respective civil law system.) This might be the easiest gross negligence tort case to show and litigate-- still hard but if everyone starts the lawsuits they can not pull the contract to protect them. They will try of course and they will fail in most but the obvious cases. What you can not sue them for is not forseeable damages -- e.g. I lost my dream job because the computer died during the interview. But ceasing operations of a company is generally fair game. And plaintiffs can argue that no reasonable person could forsee and mitigate against this disaster so the failure is not due to plaintiff's "fault" negligence.

mywacaday 2y ago

Crowdstrike should be held accountable but so should any reasonably sized enterprise that allows code to be pushed to their whole enterprise without testing. All the large enterprises I've worked for required Windows patches to be tested before being pushed to production why are crowdstrike updates treated differently?

__turbobrew__ 2y ago
It clearly states in the post mortem that rapid response updates were pushed globally without any say from the customer: https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf https://www.crowdstrike.com/wp-content/uploads/2024/08/Chann...
- mywacaday 2y ago
  So in theory if a company like crowstrike were compromised by criminals/disgruntled employee/NSA the impact could not be avoided?
HeatrayEnjoyer 2y ago
Given the cybersecurity landscape it is not unusual for security software updates to be pushed globally without the option to test or even stagger them. When a potent new vulnerability becomes public knowledge (or semi-insider knowledge) and especially if there's already a PoC available, organizations only have minutes to a few hours before threat actors begin utilizing it. APTs and organized crime groups have 24/7 staff to weaponize and integrate new vulnerabilities into their workflow as rapidly as possible, or have other contracted other groups to provide this service.