6 ms·

The only reason this could be funny is because the software industry has found a way to excuse itself from any liability. There is no other industry where some

The only reason this could be funny is because the software industry has found a way to excuse itself from any liability. There is no other industry where someone could cause so much damage and laugh about it. Least of all because the liability itself would have led to its collapse. Can you imagine a company hired to reinforce a bridge to protect it from damage from a ship instead causes its collapse. How long is that company gonna last? Even if no one dies or is injured it will be run out of business. Only in Tech can such a company not only survive but laugh about it. And that’s even before we get to how amateurish the mistake these guys made was.

xyst 2y ago
100% agree. As mentioned in another post, the acceptance of this joke award is completely tone deaf. Hospitals, banks, airlines, governments, and of course various IT operations at companies that are forced to use this endpoint security crap and Windows were impacted. Many people suffered degraded quality of care at medical facilities. Surgeons losing access to critical imaging/labs during surgery. Probably many canceled and rescheduled surgeries as well. People’s flights were delayed or canceled. Imagine having to take a last minute flight to visit a loved one on their death bed only to get canceled because ClownStrike shit and incompetent IT departments/CTOs fucked them over. Many people/businesses unable to access critical banking services. Then the amount of lost productivity for office workers. Many hours lost for IT folks, often even working into the weekends. Time lost to dealing with ClownStrike bullshit when that time could have been spent with their families and friends. Fuck ClownStrike, George Kurtz, and this latest clown, Michael Sentonas
vasco 2y ago
The time to take yourself seriously is before the stuff happens. Being stuffy now doesn't add anything. Accepting the award means they have something to show every single new hire and everyone in that office will have a physical reminder to do better in the future. I get the other points about consequences, but I don't think accepting this award is in anyway problematic. It's one of those things that I expect only people that care about "appearances" would complain about.
- cnasc 2y ago
  I agree. They show up, cop to it, and collect a memento mori that will hopefully help motivate improvement in the future. They have a lot of work to do to repair their reputation, and I don’t think they’re foolish enough to think that this is anything more than a small step on a long path.
- bb88 2y ago
  Ever had someone make a mistake that cost you time or money and then tried to laugh it off as no big deal? That's what this feels like. The security industry needs to grow up. The best thing they should have done is fire the CEO, apologized profusely, and then use their army of sales people to help make things right on a one on one basis with their customers.
- bootlooped 2y ago
  A less charitable way to look at it is that they weren't taking things seriously before the incident, and they're still not taking things seriously now. What the most appropriate way to view it is, I don't know. I think I'd need to know way more than I do about Crowdstrike leadership.
  - evilduck 2y ago
    This wasn't their first serious blunder this year even, just the most damaging and visible. The nature of their mistakes seem to be exceedingly preventable too, with them failing at textbook SRE practices. Their CEO has now been at the helm of two different companies that have had similar problems under his leadership. The evidence keeps piling up and people want to keep making excuses for negligent behaviors. Why should we excuse facts for hypotheticals?
- chestertonsgate 2y ago
  I'm sorry, how has CrowdStrike at all demonstrated that they're going to do better?
Aardwolf 2y ago
On the other hand (maybe I'm just playing devil's advocate here): nobody died (I hope at least! It's possible if some hospital equipment, 911 calls, ... failed...) from this incident despite being such huge scale that almost everyone knows about it. It's almost as if humanity can be... fine... when all this computing equipment fails.
- evilduck 2y ago
  Then what value do they provide?
- joe_the_user 2y ago
  Many hospitals, including emergency rooms, were shutdown. Maybe no single death can be directly tied to the event but I'm pretty this effectively resulted in greater death. Maybe some of those inconvenience had to time stop and contemplate the world but there are parts of the world where computers stopping don't leave people just fine.
  - jcims 2y ago
    Agreed and I’m sure there are tons of anecdotesb just in this community. My daughter for example is a night shift labor and delivery nurse at a level three metro hospital. They were heavily impacted. All of the phones were down, all of their internal messaging was down, translation services are down, it was a rough couple of nights. I don’t have any direct experience with crowdstrike, but in my experience with security vendors in particular, they make it very difficult for customers to inject useful change management into the process. I’ve been in infosec for nearly thirty years and “my people” also need to shoulder some of the blame for catastrophizing delays in delivery of updates to preventative and detective controls. I’ve always known this but spending the last year ‘outside’ in central platform delivery and operations for a large financial has really brought that to light. Fortunately I know how to speak security so it helps us navigate but many aren’t so fortunate.
- allendoerfer 2y ago
  Stopping air travel has probably canceled a lot of unnecessary meetings and slowed down global warming. Maybe events like this provide value in that they indentify which systems are actually mission critical.
mrinfinitiesx 2y ago
Fuck up the world's computers, piss off all of its IT teams, and then send people $10 uber eats gift cards as if that'll get you anything, maybe a coke at best.. but its further admitting fault. That's like, a tip. Like here, have a sandwich while you fix our fuckup. They don't care. They'll all get 6 figure bonuses too in management for 'weathering the storm' after the mishap and probably get more money because look what they can withstand, literally technology-murder and get away with it. It's almost movie-grade evil villainy tier stuff lol
- trhway 2y ago
  Because people on the receiving end are the same - they accepted and rolled out the update without even as much as “canarying” it. SolarWind was the same - the customers weren’t bothered even by mismatched integrity hashes. It is a tacit pact in our industry - we all screw up and cut the slack to each other. Who will cast the first stone?
  - shrubble 2y ago
    Crowdstrike can force-push an update at any time of their choosing that the connected device will grab and load, is my understanding.
    - trhway 2y ago
      Don’t you see that you’re only enforcing my point?
      - shrubble 2y ago
        No, because "canary" in the context that you used it, has a specific meaning. If you believe they should have tested CrowdStrike more or been more skeptical of their claims before licensing, that's independent of the user/administrators doing canary-style testing.
  - miguelazo 2y ago
    What’s the default? And what did their technical account manager recommend? My guess, no canary ring.
delusional 2y ago
I don't think this is all that accurate. In the engineering space, Boeing has so far accepted responsibility for two fatal crashes and the fucking door falling out of an airplane and is still in business.
spacechild1 2y ago
To add insult to the injury, these people call themselves "software engineers".
- codetrotter 2y ago
  There is sparsely little real “engineering” that goes on in the field of “software engineering”, industry wide, in terms of ensuring that our software is reliable and secure. Performance and development velocity seem to take the driver seat always, in a whole lot of software. See also this talk by Bryan Cantrill, “Scale by the Bay 2018: Rust and Other Interesting Things” https://youtu.be/2wZ1pCpJUIM https://youtu.be/2wZ1pCpJUIM where he talks about software platform values, in the sense of what different programming languages and other things focus on. He also touches on the fact that while higher level layers in the stack might value security it kind of falls apart when the very microcode in our CPUs sacrifices security for performance. This was in a period of time where Spectre https://en.m.wikipedia.org/wiki/Spectre_(security_vulnerability) https://en.m.wikipedia.org/wiki/Spectre_(security_vulnerabil... and Meltdown https://en.m.wikipedia.org/wiki/Meltdown_(security_vulnerability) https://en.m.wikipedia.org/wiki/Meltdown_(security_vulnerabi... had reared their ugly heads.
  - blablabla123 2y ago
    > There is sparsely little real “engineering” that goes on in the field of “software engineering”, industry wide There surely is actual engineering but it's scattered unevenly across companies. It's funny that Crowdstrike did fuzz their code but didn't even check for correct arity. I think that the Cybersecurity industry isn't such a strong adopter of sophisticated engineering techniques as for instance in Web development where new testing techniques evolve every few years.
    - jrockway 2y ago
      I really don't think that's true. All software is undertested and it's likely that there isn't a significant differences between web apps and security apps. Having said that, writing ring 0 drivers an unsafe language sounds like an invitation to disaster. That's what went wrong with CrowdStrike. You don't need any testing to avoid crashing the OS when given a bad virus definition file. (Making the virus definition file do something useful... sure, you're gonna need tests for that.)
wayeq 2y ago
Statistically speaking it seems likely people really did die from this mistake, if only indirectly due to for example delays in medical care caused by the outage.
- gleenn 2y ago
  It would be hard to imagine it didn't: https://www.washingtonpost.com/business/2024/07/19/windows-outage-crowdstrike-cancellations-computer/ https://www.washingtonpost.com/business/2024/07/19/windows-o...
dakiol 2y ago
[flagged]
thayne 2y ago
So, you would prefer that they don't accept this "award", and in so doing admit that they messed up? And honestly, crowdstrike is more likely to go under than a company that failed to re-inforce a bridge. Their mistake caused measurable harm to many well funded companies that have the resources to sue crowdstrike in court. If crowdstrike survives, it will be because there isn't a lot of competition in their market, not because they can excuse themselves of liability.
- ospray 2y ago
  I would have liked to see Crowdstrikes legal councils face as they accepted this award. There is no way they ran it by them.
lispisok 2y ago
We've professionalized industries like engineering and medicine because incompetent practitioners are a threat to public health and safety. Software is now in everything and incompetent practitioners have been a threat to public health and safety for a long time now yet we do nothing about it.
- Onavo 2y ago
  Found the gatekeeper
  - wiseowise 2y ago
    Make sure to pick a surgeon from the street next time you’ll need an operation. Don’t want to gatekeep the profession, after all.
    - Onavo 2y ago
      Not necessarily from the street, but I do expect bio graduates to be trained in health care at scale, without being limited by residency programs. There's no point training 10 biology graduates only for 9 of them to work as waiters or on OnlyFans.
  - ThrowawayR2 2y ago
    Crowdstrike and countless other software failures before it is literally the proof that the gates need to be kept. The only question is whether we start doing our own gatekeeping or it eventually gets forced on us by heavy handed legislation like it was for doctors and civil engineers.
  - chestertonsgate 2y ago
    I'm sorry, when did the switch flip in this industry where we decided we didn't want to hire people with expertise and experience?
- scott_w 2y ago
  In construction, Grenfell happened and witnesses demanded immunity from prosecution to testify, because they knew they’d broken numerous laws in its construction and certification. Residents at similar buildings are the ones paying to make them safe for habitation, not the crooks that built them. Professionalisation is not a magic bullet.
Onavo 2y ago
Wait till you learn about finance, or oil and gas, or mining, or countless other legacy industries that quietly run America since the Rockefeller days. Many parts of Texas lost power for weeks due to Beryl and all the power companies got was a slap on the wrist (despite being explicitly warned about this scenario many times).
- scott_w 2y ago
  Hell look at Grenfell in London! 72 people died and the people that designed it wanted immunity from prosecution to testify at the inquiry! Similar buildings are charging the cost of upgrading the cladding to residents instead of doing the right thing and eating the cost themselves.
- robocat 2y ago
  Companies are too easy a target... What were the regulatory incentives? Was the electricity market designed to encourage resiliency? There was a systematic fault - it needed a collective solution not one that relies on individual companies doing the right thing???
- chestertonsgate 2y ago
  You would not want for a sympathetic ear if you also criticized these companies. The point is not that CrowdStrike is uniquely incompetent, not at all. Every critical organization needs to be held to a higher standard, not just incompetent security firms.
deleted 2y ago
[deleted]
lapphi 2y ago
I can think of at least one industry where the price of failure is almost always borne by the users and not the companies. Very closely integrated with tech as well.
crisdux 2y ago
[flagged]
booleandilemma 2y ago
It just illustrates how low the stakes are in tech really.
boxed 2y ago
This attitude is why we have a culture of fear and do-nothings.

xyst 2y ago

100% agree. As mentioned in another post, the acceptance of this joke award is completely tone deaf. Hospitals, banks, airlines, governments, and of course various IT operations at companies that are forced to use this endpoint security crap and Windows were impacted. Many people suffered degraded quality of care at medical facilities. Surgeons losing access to critical imaging/labs during surgery. Probably many canceled and rescheduled surgeries as well. People’s flights were delayed or canceled. Imagine having to take a last minute flight to visit a loved one on their death bed only to get canceled because ClownStrike shit and incompetent IT departments/CTOs fucked them over. Many people/businesses unable to access critical banking services. Then the amount of lost productivity for office workers. Many hours lost for IT folks, often even working into the weekends. Time lost to dealing with ClownStrike bullshit when that time could have been spent with their families and friends. Fuck ClownStrike, George Kurtz, and this latest clown, Michael Sentonas

vasco 2y ago

The time to take yourself seriously is before the stuff happens. Being stuffy now doesn't add anything. Accepting the award means they have something to show every single new hire and everyone in that office will have a physical reminder to do better in the future. I get the other points about consequences, but I don't think accepting this award is in anyway problematic. It's one of those things that I expect only people that care about "appearances" would complain about.

cnasc 2y ago
I agree. They show up, cop to it, and collect a memento mori that will hopefully help motivate improvement in the future. They have a lot of work to do to repair their reputation, and I don’t think they’re foolish enough to think that this is anything more than a small step on a long path.
bb88 2y ago
Ever had someone make a mistake that cost you time or money and then tried to laugh it off as no big deal? That's what this feels like. The security industry needs to grow up. The best thing they should have done is fire the CEO, apologized profusely, and then use their army of sales people to help make things right on a one on one basis with their customers.
bootlooped 2y ago
A less charitable way to look at it is that they weren't taking things seriously before the incident, and they're still not taking things seriously now. What the most appropriate way to view it is, I don't know. I think I'd need to know way more than I do about Crowdstrike leadership.
- evilduck 2y ago
  This wasn't their first serious blunder this year even, just the most damaging and visible. The nature of their mistakes seem to be exceedingly preventable too, with them failing at textbook SRE practices. Their CEO has now been at the helm of two different companies that have had similar problems under his leadership. The evidence keeps piling up and people want to keep making excuses for negligent behaviors. Why should we excuse facts for hypotheticals?
chestertonsgate 2y ago
I'm sorry, how has CrowdStrike at all demonstrated that they're going to do better?

Aardwolf 2y ago

On the other hand (maybe I'm just playing devil's advocate here): nobody died (I hope at least! It's possible if some hospital equipment, 911 calls, ... failed...) from this incident despite being such huge scale that almost everyone knows about it. It's almost as if humanity can be... fine... when all this computing equipment fails.

evilduck 2y ago
Then what value do they provide?
joe_the_user 2y ago
Many hospitals, including emergency rooms, were shutdown. Maybe no single death can be directly tied to the event but I'm pretty this effectively resulted in greater death. Maybe some of those inconvenience had to time stop and contemplate the world but there are parts of the world where computers stopping don't leave people just fine.
- jcims 2y ago
  Agreed and I’m sure there are tons of anecdotesb just in this community. My daughter for example is a night shift labor and delivery nurse at a level three metro hospital. They were heavily impacted. All of the phones were down, all of their internal messaging was down, translation services are down, it was a rough couple of nights. I don’t have any direct experience with crowdstrike, but in my experience with security vendors in particular, they make it very difficult for customers to inject useful change management into the process. I’ve been in infosec for nearly thirty years and “my people” also need to shoulder some of the blame for catastrophizing delays in delivery of updates to preventative and detective controls. I’ve always known this but spending the last year ‘outside’ in central platform delivery and operations for a large financial has really brought that to light. Fortunately I know how to speak security so it helps us navigate but many aren’t so fortunate.
allendoerfer 2y ago
Stopping air travel has probably canceled a lot of unnecessary meetings and slowed down global warming. Maybe events like this provide value in that they indentify which systems are actually mission critical.

mrinfinitiesx 2y ago

Fuck up the world's computers, piss off all of its IT teams, and then send people $10 uber eats gift cards as if that'll get you anything, maybe a coke at best.. but its further admitting fault. That's like, a tip. Like here, have a sandwich while you fix our fuckup. They don't care. They'll all get 6 figure bonuses too in management for 'weathering the storm' after the mishap and probably get more money because look what they can withstand, literally technology-murder and get away with it. It's almost movie-grade evil villainy tier stuff lol

trhway 2y ago
Because people on the receiving end are the same - they accepted and rolled out the update without even as much as “canarying” it. SolarWind was the same - the customers weren’t bothered even by mismatched integrity hashes. It is a tacit pact in our industry - we all screw up and cut the slack to each other. Who will cast the first stone?
- shrubble 2y ago
  Crowdstrike can force-push an update at any time of their choosing that the connected device will grab and load, is my understanding.
  - trhway 2y ago
    Don’t you see that you’re only enforcing my point?
    - shrubble 2y ago
      No, because "canary" in the context that you used it, has a specific meaning. If you believe they should have tested CrowdStrike more or been more skeptical of their claims before licensing, that's independent of the user/administrators doing canary-style testing.
- miguelazo 2y ago
  What’s the default? And what did their technical account manager recommend? My guess, no canary ring.

delusional 2y ago

I don't think this is all that accurate. In the engineering space, Boeing has so far accepted responsibility for two fatal crashes and the fucking door falling out of an airplane and is still in business.

spacechild1 2y ago

To add insult to the injury, these people call themselves "software engineers".

codetrotter 2y ago
There is sparsely little real “engineering” that goes on in the field of “software engineering”, industry wide, in terms of ensuring that our software is reliable and secure. Performance and development velocity seem to take the driver seat always, in a whole lot of software. See also this talk by Bryan Cantrill, “Scale by the Bay 2018: Rust and Other Interesting Things” https://youtu.be/2wZ1pCpJUIM https://youtu.be/2wZ1pCpJUIM where he talks about software platform values, in the sense of what different programming languages and other things focus on. He also touches on the fact that while higher level layers in the stack might value security it kind of falls apart when the very microcode in our CPUs sacrifices security for performance. This was in a period of time where Spectre https://en.m.wikipedia.org/wiki/Spectre_(security_vulnerability) https://en.m.wikipedia.org/wiki/Spectre_(security_vulnerabil... and Meltdown https://en.m.wikipedia.org/wiki/Meltdown_(security_vulnerability) https://en.m.wikipedia.org/wiki/Meltdown_(security_vulnerabi... had reared their ugly heads.
- blablabla123 2y ago
  > There is sparsely little real “engineering” that goes on in the field of “software engineering”, industry wide There surely is actual engineering but it's scattered unevenly across companies. It's funny that Crowdstrike did fuzz their code but didn't even check for correct arity. I think that the Cybersecurity industry isn't such a strong adopter of sophisticated engineering techniques as for instance in Web development where new testing techniques evolve every few years.
  - jrockway 2y ago
    I really don't think that's true. All software is undertested and it's likely that there isn't a significant differences between web apps and security apps. Having said that, writing ring 0 drivers an unsafe language sounds like an invitation to disaster. That's what went wrong with CrowdStrike. You don't need any testing to avoid crashing the OS when given a bad virus definition file. (Making the virus definition file do something useful... sure, you're gonna need tests for that.)

wayeq 2y ago

Statistically speaking it seems likely people really did die from this mistake, if only indirectly due to for example delays in medical care caused by the outage.

gleenn 2y ago
It would be hard to imagine it didn't: https://www.washingtonpost.com/business/2024/07/19/windows-outage-crowdstrike-cancellations-computer/ https://www.washingtonpost.com/business/2024/07/19/windows-o...

dakiol 2y ago

[flagged]

thayne 2y ago

So, you would prefer that they don't accept this "award", and in so doing admit that they messed up? And honestly, crowdstrike is more likely to go under than a company that failed to re-inforce a bridge. Their mistake caused measurable harm to many well funded companies that have the resources to sue crowdstrike in court. If crowdstrike survives, it will be because there isn't a lot of competition in their market, not because they can excuse themselves of liability.

ospray 2y ago
I would have liked to see Crowdstrikes legal councils face as they accepted this award. There is no way they ran it by them.

lispisok 2y ago

We've professionalized industries like engineering and medicine because incompetent practitioners are a threat to public health and safety. Software is now in everything and incompetent practitioners have been a threat to public health and safety for a long time now yet we do nothing about it.

Onavo 2y ago
Found the gatekeeper
- wiseowise 2y ago
  Make sure to pick a surgeon from the street next time you’ll need an operation. Don’t want to gatekeep the profession, after all.
  - Onavo 2y ago
    Not necessarily from the street, but I do expect bio graduates to be trained in health care at scale, without being limited by residency programs. There's no point training 10 biology graduates only for 9 of them to work as waiters or on OnlyFans.
- ThrowawayR2 2y ago
  Crowdstrike and countless other software failures before it is literally the proof that the gates need to be kept. The only question is whether we start doing our own gatekeeping or it eventually gets forced on us by heavy handed legislation like it was for doctors and civil engineers.
- chestertonsgate 2y ago
  I'm sorry, when did the switch flip in this industry where we decided we didn't want to hire people with expertise and experience?
scott_w 2y ago
In construction, Grenfell happened and witnesses demanded immunity from prosecution to testify, because they knew they’d broken numerous laws in its construction and certification. Residents at similar buildings are the ones paying to make them safe for habitation, not the crooks that built them. Professionalisation is not a magic bullet.

Onavo 2y ago

Wait till you learn about finance, or oil and gas, or mining, or countless other legacy industries that quietly run America since the Rockefeller days. Many parts of Texas lost power for weeks due to Beryl and all the power companies got was a slap on the wrist (despite being explicitly warned about this scenario many times).

scott_w 2y ago
Hell look at Grenfell in London! 72 people died and the people that designed it wanted immunity from prosecution to testify at the inquiry! Similar buildings are charging the cost of upgrading the cladding to residents instead of doing the right thing and eating the cost themselves.
robocat 2y ago
Companies are too easy a target... What were the regulatory incentives? Was the electricity market designed to encourage resiliency? There was a systematic fault - it needed a collective solution not one that relies on individual companies doing the right thing???
chestertonsgate 2y ago
You would not want for a sympathetic ear if you also criticized these companies. The point is not that CrowdStrike is uniquely incompetent, not at all. Every critical organization needs to be held to a higher standard, not just incompetent security firms.

deleted 2y ago

[deleted]

lapphi 2y ago

I can think of at least one industry where the price of failure is almost always borne by the users and not the companies. Very closely integrated with tech as well.

crisdux 2y ago

[flagged]

booleandilemma 2y ago

It just illustrates how low the stakes are in tech really.

boxed 2y ago

This attitude is why we have a culture of fear and do-nothings.