12 ms·
CrowdStrike accepting the PwnieAwards for "most epic fail" at defcon
- hartator 2y agoSo 911 not working in several cities and hospitals flows being slow down to an halt, and they have time to go to defcon for a joke?
- mminer237 2y agoAre there hospitals that haven't fixed their computers yet? Obviously CS messed up, but aside from paying for damages and making process changes, I'm not sure what else they can do at this time. Warning people to not repeat their mistakes doesn't seem like a bad use of time.
- citrin_ru 2y agoCS deserves the blame here but putting CS into a critical system like 911 IMHO is a huge mistake (but whoever done this likely knew they will dodge the blame so why they should care).
- dredmorbius 2y agoVia xcancel: <https://xcancel.com/singe/status/1822324795645575263 https://xcancel.com/singe/status/1822324795645575263>
- deleted 2y ago[deleted]
- Kenji 2y ago[dead]
- candiddevmike 2y agoWonder what kind of deliberation led to them accepting the award.
- cdchn 2y ago"No such thing as bad press" I assume.
- AnimalMuppet 2y agoThey screwed up. They know it, and everybody else knows it. Trying to pretend they didn't would just make them look even more lame. Or, viewed from the other side: Owning your failures makes you a grownup.
- H8crilA 2y agoShhh, the people here want blood. BTW, did you know that there's an endless stream of "satisfying" drama on YouTube? I heard that Mr. Beast is finally in some hot water!
- Biganon 2y agoThe outage caused actual human deaths. Yeah, most people here probably think the priority is criminal justice, which you might call "blood" in a dishonest attempt to make us appear cynical when they're the ones accepting funny nerdy awards after causing so much chaos. Maybe next time a doctor causes death because of their negligence, they should accept an "oopsie award"? It would be le funny lulz XD
- atonse 2y agoIn the cases where outages caused human deaths you’re 100% right that there should be worse consequences. But in many cases, hopefully it’s also a lesson to the places where people were harmed, to never let one piece of software get in the path of life or death without redundancies. (Not at all defending their screwup, I just don’t think EVERYONE deserves the same restitution, some more than others.)
- hakdbha 2y ago[dead]
- 2y ago
- chongli 2y agoI appreciate that we’re finding the humour in this catastrophe but what about the question of liability? I have seen a few stories on HN of the billions lost by this event but so far not much in the way of lawsuits. What is the situation? Are the licenses so ironclad that customers have no recourse? I could understand this in the case of consumers who might suffer minor inconvenience as their home PC is out of service for a few hours/days but it seems totally unacceptable for industries to accept this level of risk exposure. This is one of the big reasons civil engineering is considered such a serious discipline. If a bridge collapses, there’s not only financial liability but the potential for criminal liability as well. Civil engineering students have it drilled into their heads that if they behave unethically or otherwise take unacceptable risks as an engineer they face jail time for it. Is there any path for software engineers to reach this level of accountability and norms of good practice?
- gjsman-1000 2y ago> Is there any path for software engineers to reach this level of accountability and norms of good practice? Heck, no. Civil engineering doesn’t change. Gravity is a constant. Physics are constants. If Rome wrote an engineering manual, it would still be quite valid today. Imagine if we had standardized software engineering in 2003. Do you think the mandatory training about how to make safe ActiveX controls is going to save you? Do you think the mandatory training on how to safely embed a Java applet will protect your bank? Software is too diverse, too inconsistent, and too rapidly changing to have any chance at standardization. Maybe in several decades when WHATWG hasn’t passed a single new spec into the browser. (Edit: Also, it’s a fool’s errand, as there are literally hundreds of billions of lines of code running in production at this very moment. If you wrote an onerous engineering spec; there would not be enough programmers or lawyers on earth to rewrite and verify it all, even if given decades. This would result in Google, Apple, etc. basically getting grandfathered in while startups get the burden of following the rules - rules that China, India, and other countries happily won’t be enforcing.)
- figassis 2y agoPhysics will still be the same when your faulty software tells an airplane to dive.
- omoikane 2y agoPrevious Pwnie Award winners for comparison: https://en.wikipedia.org/wiki/Pwnie_Awards https://en.wikipedia.org/wiki/Pwnie_Awards
- rldjbpin 2y agomost of the previous "most epic fail" awards are naturally dominated by microsoft. blame shame can continue but if you believe if "production failures" such as this one is due to bad processes, then m$ definitely played a big role here.
- xyst 2y agoThis comes across as incredibly tone deaf. People suffered degraded medical care, billions lost in the airline industry, billions more lost in productivity, and ultimately its time that people cannot get back. Yet these clowns are accepting joke awards as if this is something to hang on your trophy wall. This is actually a c-level executive at ClownStrike, by the way. > Michael Sentonas serves as President and is responsible for CrowdStrike’s product and go-to-market functions, including its sales, marketing, product & engineering, threat intelligence, privacy & policy, corporate development, corporate strategy and CTO teams https://www.crowdstrike.com/about-crowdstrike/executive-team/michael-sentonas/ https://www.crowdstrike.com/about-crowdstrike/executive-team... The whole C-level executive suite at ClownStrike needs to go. This company needs a real CTO like Jeremy Rowley. Although I suspect a good person like him would never join the ranks of ClownStrike
- sigmoid10 2y agoWell, this is the second time their CEO caused a major outage by pushing a flawed update for a security product. This whole thing is probably a joke to him.
- deleted 2y ago[deleted]
- hn_throwaway_99 2y agoDid people actually watch the video??? I just don't understand how they think Michael Sentonas was making a joke of all this. If anything, he was acknowledging the horrible outcome of what happened. I don't think this absolves CrowdStrike of responsibility at all, but what would you like him to do, commit harikari on stage?
- xyst 2y agoClearly, step down. Including CEO (George Kurtz) The shit rolls down hill, starting from the c-suite. These clowns clearly cannot change the org and are blind to the issues. Keeping the same leadership means nothing will change. The fact that they even poke their head up for what is clearly a marketing/PR stunt without showing any substance shows how clueless they are. Guy has “20 years” of experience which clearly doesn’t amount to shit. Maybe 20 years of junior experience and falling upwards.
- textlapse 2y agoThis may end up in one of those court evidence videos or lawsuits - this isn’t a funny thing. This would have been a closed moment (just a bunch of security nerds discussing something) but instead this is now freely available for the wider general public who had major grievances to lampoon them.
- hn_throwaway_99 2y ago> This may end up in one of those court evidence videos or lawsuits - this isn’t a funny thing. I didn't take the CrowdStrike's executive as making light of the situation, at all. If anything, I thought his speech took it seriously, acknowledged it was a major, major fuckup, and basically said he was accepting the trophy as a mark of shame and as a cautionary tale for future CrowdStrike employees. I thought the exec accepting this was a true class act (to emphasize, saying that in no way should imply that I think it absolves CrowdStrike of responsibility, or liability, for what happened).
- textlapse 2y agoContext is everything. They had every chance to own up from the day of until now. A ‘lulz haha we goofed up’ in a nerdy security conference doesn’t seem like the right place or time.
- Brett_Riverboat 2y agoI work in IT and I happened to be the poor bastard on call when Clown Strike took out the majority of infrastructure. If it wasn't for my own personal refusal to use cloud based bullshit we would have probably been down for days instead of hours. The fact that people like my IT director saw nothing wrong with this and is taking 0 steps to negate such bullshit makes me quite worried that I will soon have to deal with some other catastrophic cloud based failure in the near future. I keep repeating ad-nauseam "only idiots rely on other peoples computers" And I stand behind that statement 100%
- xyst 2y agoCan only do so much when idiot CTOs take their advice from CTO summits, consultants with their own perverse incentives, and of course random conferences
- system2 2y agoI 100% agree. Additionally, I am amazed to see that people can pay outrageous fees for cloud services such as Azure VD. For the fraction of the yearly cloud budget, companies can create crazy stable, offline-capable infrastructure themselves.
- cqqxo4zV46cp 2y agoYou sound like a very aggressive person. I don’t think I’d like to work with you regardless of whether or not you were right. Maybe you’d get your point across better if you weren’t so aggressive about it.
- Brett_Riverboat 2y agok
- citrin_ru 2y agoWhat strange that there are mangers/executives who understand why it’s bad to have a SPOF (single point of failure) in the internal infrastructure but who are OK with having a product/service from an external vendor as an SPOF. Like if you have a contract and are paying money for it means that it is created and maintained by infallible superhuman (as opposite to internal engineers they don’t trust). Such misplaced trust puzzles me.
- aa_is_op 2y agoThey weren't even nominated, so this is such an epic failure they won it as a late entry. Also, that pony isn't typically super-glued to that structure, so this was also a special trophy! :)))))))))
- fredgrott 2y agoNo, the most epic fail is still held by Boeing...they launched an alpha capsule to ISS and still cannot get the astronauts back to Earth...
- swarnie 2y agoGlad you find it funny Chaps, forwarding this to my legal dept.
- addicted 2y agoThe only reason this could be funny is because the software industry has found a way to excuse itself from any liability. There is no other industry where someone could cause so much damage and laugh about it. Least of all because the liability itself would have led to its collapse. Can you imagine a company hired to reinforce a bridge to protect it from damage from a ship instead causes its collapse. How long is that company gonna last? Even if no one dies or is injured it will be run out of business. Only in Tech can such a company not only survive but laugh about it. And that’s even before we get to how amateurish the mistake these guys made was.
- xyst 2y ago100% agree. As mentioned in another post, the acceptance of this joke award is completely tone deaf. Hospitals, banks, airlines, governments, and of course various IT operations at companies that are forced to use this endpoint security crap and Windows were impacted. Many people suffered degraded quality of care at medical facilities. Surgeons losing access to critical imaging/labs during surgery. Probably many canceled and rescheduled surgeries as well. People’s flights were delayed or canceled. Imagine having to take a last minute flight to visit a loved one on their death bed only to get canceled because ClownStrike shit and incompetent IT departments/CTOs fucked them over. Many people/businesses unable to access critical banking services. Then the amount of lost productivity for office workers. Many hours lost for IT folks, often even working into the weekends. Time lost to dealing with ClownStrike bullshit when that time could have been spent with their families and friends. Fuck ClownStrike, George Kurtz, and this latest clown, Michael Sentonas
- vasco 2y agoThe time to take yourself seriously is before the stuff happens. Being stuffy now doesn't add anything. Accepting the award means they have something to show every single new hire and everyone in that office will have a physical reminder to do better in the future. I get the other points about consequences, but I don't think accepting this award is in anyway problematic. It's one of those things that I expect only people that care about "appearances" would complain about.
- 2y ago
- maxlin 2y agoThis shows great grit and ownership. Props
- thedougd 2y agoThey understand who’s buying their product. It’s not the information security teams who cleaned up this mess, but rather the operations and end user compute teams.
- mikeweiss 2y agoI'm wondering how on earth has the CEO and CTO managed to keep their jobs after this fiasco?
- skirge 2y agoSerious companies should have better QA. This includes Crowdstrike customers.
- badgersnake 2y agoIt took our IT department until Monday afternoon after to put out a message reaffirming their confidence in crowdstrike. Before any postmortem on their side or ours. I’m guessing they got offered a big discount to renew. My IT team have a lot more confidence in crowdstrike than I have in them.
- Aerbil313 2y agoNever in a hundred years I’d expect to see the CEO of Crowdstrike at Defcon. The two are the two extremes on a spectrum of corporate.
- bravetraveler 2y agoThis is how counter-culture is systematically uprooted
- ein0p 2y agoThat seems unwise and tone deaf - to laugh it up on stage after causing easily tens of billions of dollars in losses worldwide.
- hypeatei 2y agoThis issue goes beyond CrowdStrike and points to the general approach to security that is buying products off the shelf to satisfy regulators and insurers while not actually caring what it does or how it works. I'm not saying tech shouldn't be regulated, but our current model of "buy this thing to shed liability" doesn't work. The worst part is, the people who saw this coming (i.e. your IT department) probably can't do a damn thing about it because it's mandated at high levels in the company either for "cyber insurance" requirements or some other regulation. Madness.
- maccard 2y ago> The worst part is, the people who saw this coming (i.e. your IT department) probably can't do a damn thing about it because it's mandated at high levels in the company either for "cyber insurance" requirements or some other regulation. I've worked with many excellent IT people who feel this way, but the vast majority of my experience with IT departments has been that as long as the contract covers what it needs to, they don't actually care if it solves the problem or not. At a previous job, software similar to crowdstrike was installed on my workstation over a weekend, and I came back to 20% slower compile times (I was working on them at the time so I had dozens of measurements). I had ETL traces showing the problem was the software, but IT refused to acknowledge it because the vendor contract said there was no performance impact expected for our workload.
- hypeatei 2y agoThat is my experience, too. I attribute it to IT / sysadmin jobs having a lower bar to entry and becoming more of a "watered down" business unit that just follows orders without much say or care for anything.
- cqqxo4zV46cp 2y agoMost IT departments wouldn’t have seen this coming, and certainly would’ve been right to not base their entire security strategy around it. I’m not sure where this narrative is coming from. Falcon delivered and still delivers real, genuine security benefit to its customers. That does not mean that it eliminates all risk, and does not mean that it doesn’t introduce risk of its own. It’s literally a game of trafeoffs like all engineering problems. This shouldn’t be that foreign to anyone here. Suddenly HN is full of security experts that are fuelled with 20/20 hindsight and recency bias, explaining how companies could’ve dodged this bullet without considering what very real bullets were being dodged by using Falcon in the first place.
- solarkraft 2y agoIt would’ve been the best place to say sorry, but as far as I understand this would have legal consequences.
- mikewarot 2y agoComputer security issues cropped up during Viet Nam, and the US did the work, and found actually effective computer security models. We're in a society that has effectively memory holed them. Why is it necessary to have scanners running 24x7 on everything a computer is about to run? Why is it necessary to have Operating Systems that rely on ambient authority? Blaming Crowdstrike does nothing but distract from the fundamental design failure we ignore every day in our operating systems, Linux, MacOS, Windows, et al.
- deleted 2y ago[deleted]
- Rick76 2y agoI think they’re owning up to their mistakes instead of dodging the issue. I still feel that if they did the right testing, they shouldn’t be blamed for everything. It’s pretty standard for IT teams to avoid auto-updates and instead manually review them—especially in critical sectors like healthcare, aviation, and government. For instance, at my workplace, we’re not allowed to auto-update VsCode. They mentioned they ran tests which unfortunately returned false positives. While it’s true they could’ve been more thorough, the affected companies also dropped the ball by not doing their own checks
- wolrah 2y ago> I still feel that if they did the right testing, they shouldn’t be blamed for everything. This update crashed 100% of the Windows systems it got installed on, which means either their testing did not involve actually loading it on real world computers at all or that blue screening and boot looping did not cause the test to fail. It is objectively clear that they did not do the right testing. There is no excuse for this update having ever left the earliest stages of a proper test process. It's not like this is a case of an unexpected interaction with a configuration not found in the test lab. > It’s pretty standard for IT teams to avoid auto-updates and instead manually review them—especially in critical sectors like healthcare, aviation, and government. This component was not able to be controlled in this way. Systems that were configured to be delayed on other CrowdStrike updates still got this particular update immediately with no ability for IT departments to control them. > They mentioned they ran tests which unfortunately returned false positives. Again, whatever tests they actually ran clearly didn't involve actually loading the update in to the actual driver. Their explanation sounds like they may have validated the formatting of their update or something like that but then just sent it. > While it’s true they could’ve been more thorough, the affected companies also dropped the ball by not doing their own checks No they did not because they could not. They may have dropped the ball when installing Crowdstrike in the first place, but the whole reason this was such a widespread thing affecting so many high priority systems is that it wasn't able to be controlled in the ways IT departments would want.
- Rick76 2y ago
- Thompson88 2y ago[dead]
- Thompson88 2y agoWhile they're still blaming Microsoft as if it isn't possible to run the code they update outside of the kernel and only use the kernel mode code for observation and action, but not logic.