14 ms·
CrowdStrike global outage to cost US Fortune 500 companies $5.4B
- lnxg33k1 2y agoSo, here's 10 bucks, we good?
- throwaway240403 2y agoWorse than than, 10$ gift card
- lawn 2y agoI heard it had been withdrawn?
- metadat 2y agoCrowdStrike offers a $10 apology gift card to say sorry for outage - https://news.ycombinator.com/item?id=41058261 https://news.ycombinator.com/item?id=41058261 (129 comments) According to the discussion in the thread, you're correct. Also, it was a $10 giftcard for .. uber eats. Where you can't get anything for less than ten bucks.
- lawn 2y agoThat's truly some dark comedy.
- rqtwteye 2y agoThese are deeply insulting. My company sometimes sends out $10 cards for DoorDash. To actually get something I would have to add at least another $10 myself. I wonder if Uber Eats and DoorDash give these out for free to companies as promotion. I bet most people who use the cards spend another $20 or more.
- ck45 2y agoEven worse, the vouchers got canceled and can't be redeemed.
- dylan604 2y agoI guess it's easier to buy X number of gift cards and allow that company to deal with the individuals rather than paying each customer $10 individually. Also, does the IT manager get that giftcard? Do they share it with the rest of the team? Does the CTO get the card and shares it with the rest of the C-suite. What's the proper way of handling that other than reject with a harsh laugh in their face at the offer?
- bluSCALE4 2y agoOr a year membership of geek squad.
- xedrac 2y agoLooks good on a resume: - Wrote code responsible for $5.4 billion
- jajko 2y ago[flagged]
- deelowe 2y agoHow do these sort of estimates get made? Genuinely curious.
- floam 2y agoIn this case a guy asked an LLM. So dumb. https://news.ycombinator.com/item?id=41054756 https://news.ycombinator.com/item?id=41054756
- wizzwizz4 2y agoDo you have a source for this? I can't find estimates anywhere.
- eagerpace 2y agoSource?
- dougi3 2y agoDude is probably guessing
- bergie 2y ago"...and made an impact in thousands of lives" in resume-speak
- floam 2y agohttps://news.ycombinator.com/item?id=41054756 https://news.ycombinator.com/item?id=41054756 Please tell me you aren’t carelessly repeating a braindead figure you read because an LLM TOLD SOME GUY SO. Because you wouldn’t do that, right? Please.
- bdcravens 2y agoBankruptcy coming ....
- 7thaccount 2y agoI'm curious if that will actually happen as these companies need to use something to check their compliance box.
- femiagbabiaka 2y agoCorrect. To move away there have to be alternatives. Who here is building the alternative?
- SoftTalker 2y agoYeah I think it will be quickly forgotten. Most people already have where I work. It was just another IT fail in a long line of them.
- dylan604 2y agoThis was what I kept thinking about when it happened. Every job I've ever had has had days of some system or other being down for some amount of time. Hell, even WFH has put me offline when I've had extended power outage after a major storm. The only thing different about this was that it was all the companies at the same time because of one glitch. For all of the companies that did not use this system, it was just another day of the week.
- xyst 2y agoThere are competitors in this space. Palo Alto Networks comes to mind. Whether competitors have same issues as ClownStrike remains to be seen.
- AmericanChopper 2y agoThe two problems I see with this story are 1) These types of products cause incidents all the time, this was just a very high impact one that happened to affect everybody all at once. 2) Their product is very good compared to the competitors. All products in this space are black boxes, but CS is one of the least black-boxy, the alerts it produces are decent, the tooling is comes with is especially good (from an operator perspective), and the reporting it produces is exactly the sort of thing decision makers in big enterprise love to see. I doubt there’s going to be much churn from this, definitely not an existential amount. As much as I personally can’t stand the organisation, I think they absorbed most of the bad press on behalf of all the service providers they took offline.
- major505 2y ago[flagged]
- solardev 2y agoHow does a company this big not have automated tests for their config files, and not have gradual/staggered rollouts for their deployments? Is there some good reason for this approach (need to get config updates into the wild as quickly as possible to combat zero-days or zero-hours?) or was this just a massive oversight? Side rant... their postmortem took forever to get to the point, first explaining all their jargon and product names. Makes me really appreciate the Cloudflare ones.
- blackeyeblitzar 2y agoOften this is not because the team doesn’t know about these things but because they have low staffing or other priorities or deadlines. This event looks to me like company rot that can be laid at the CEO’s feet
- lupusreal 2y agoMy bet is they have some normal process for updates that has testing but that process is only enforced by policy, not code, and somebody simply decided it was a waste of their time.
- hakdbha 2y ago[dead]
- geodel 2y agoYeah, management should sternly tell All code / config must be tested before deployment to prod. Millions of companies have issued this order and after that they are running free of problem for decades. It is so straightforward and it always works.
- toomuchtodo 2y agoWithout regulation, best practices are simply opinions and suggestions. “Please do” is insufficient for critical infrastructure. See: financial infra regulatory apparatus. Incentives, outcomes, the usual.
- mberning 2y agoI think this is going to be a huge boon for dell. We had so many older computers that got completely hosed. Lots of Latitude 5400s died completely. All will need replacements.
- seattle_spring 2y agoHow did it hose them completely? I thought the problem is easily fixed by removing the offending config?
- Thaxll 2y agoNever heard of servers with bad hardware that never rebooted for years?
- leetbulb 2y agoMany people here have only ever used cloud servers.
- anthk 2y agoAnd yet here we are discussing laptops.
- School-Cotton 2y agoThe Latitude 5400 is a laptop.
- deleted 2y ago[deleted]
- SoftTalker 2y agoHow? The fix is to just remove the one bad "Channel File." Are there machines where that does not resolve the problem?
- nerdjon 2y agoHardware failure hosed or just needing a re-image hosed? Unless the hardware was already near failure I don’t see how this could cause hardware failure. The worst case scenario was the machine just constantly rebooting but after 3 (I think, somewhere around that number) it should have launched into WindowsRE.
- nottorp 2y agoIt's my understanding that CrowdStrike customers buy that thing to check a box in some security audit, not because it provides any other benefit. Let's blame bullshit compliance?
- wepple 2y agoWhat’s that understanding based on?
- nottorp 2y agoAll previous comments on HN about the incident... I've seen absolutely no one praising the thing as a security solution but a lot of people posting that it's bought to pass audits.
- jezzamon 2y agoActually there were a lot of pen testers that were speaking positively about CrowdStrike
- evanreichard 2y agoAs someone who used CrowdStike daily and worked as an MDR Analyst and Engineer at a top ranked MDR provider, CrowdStrike is a very capable piece of tech. While the driver for purchase is almost always to pass audits, it's still a good product.
- gruez 2y ago>not because it provides any other benefit. It probably does provide benefits against some clueless intern in accounting downloading a macro-enabled excel file that has a malware enabled.
- deleted 2y ago[deleted]
- LASR 2y agoI’m pretty certain CS has contracts that limit their liabilities in events like this. Probably a refund is all they’ll be on the hook for. Sadly, damage done like this is just chalked up to an accident, and swept under the rug.
- Terretta 2y ago> contracts that limit their liabilities… refund is all they’ll be on the hook for By cashing in this $10 Uber Eats coupon you agree to hold harmless... - https://news.ycombinator.com/item?id=41058261 https://news.ycombinator.com/item?id=41058261 - https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-apology-gift-card-to-say-sorry-for-outage/ https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-ap...
- mirashii 2y ago"A few people on twitter are saying this thing happened. We didn't actually talk to them, we didn't look at the emails and verify their authenticity ourselves, we just trusted some twitter screenshots and wrote a blogspam article stating it as truth. We put absolutely no critical thought into whether this was a likely thing, and we completely ignored the many government and media reports that are credibly sourced which state that there are known phishing scams and other threat actors trying to capitalize on this incident.” I highly doubt this is something that Crowdstrike actually did. Edit: Amazingly they did, the article has been updated with a statement. Amazingly stupid all around.
- vb-8448 2y agoIt depends on the actual root cause, gross negligence won't save them, regardless of what they put in the contracts. From my point of view, one of the greatest problem for them is that they bypassed customers deployment policies.
- sithadmin 2y ago>one of the greatest problem for them is that they bypassed customers deployment policies Caveat emptor. Falcon and other similar security products often push updates at-will, and they're fully transparent about this if you actually read the contract terms and understand the vendor's approach to operations. I have worked with many clients that elect not to use such tools in certain sensitive environments, specifically to mitigate the risk of being impacted by something like CrowdStrike's 7/19 event.
- logicchains 2y agoHopefully this will make BigCos think twice about forcing their employees to fill their computers with "security" malware that slows productivity to a crawl.
- disruptiveink 2y agoThe value proposition of Crowdstrike is exactly that: something that you can deploy to tick the regulatory checkbox of "we have endpoint protection from a reputable company everywhere" without consuming outrageous system resources. That's why they have so many enterprise customers. They're the only game in town that won't slow down your servers arbitrarily while still convincing an auditor that you do have an antivirus. Too bad they also crash your whole system every now and then.
- leandrod 2y agoFree just got cheaper. Yeah, I know using free software isn’t a panacea. Still it would be a step in the right direction, plus I could not refrain from the cheap shot at M$ Windows.
- xnyan 2y agoCloudstrike customers voluntarily agreed to allow Cloudstrike to push kernel drivers. What should Microsoft have done to prevent this?
- ck45 2y agoMove Windows Defender into user space and enforcing the same for all security software.
- IcyWindows 2y agoThis has nothing to do with how Defender works. Crowdstrike shipped a driver that they marked as a mandatory boot driver. The Windows OS could have had more recovery options otherwise.
- ck45 2y agoMoving Defender to user space is a requirement to lock down windows from a fair competition perspective. Microsoft is currently blaming the EU commission for not allowing them locking down Windows, compare https://www.telegraph.co.uk/business/2024/07/22/microsoft-blames-eu-rules-worlds-biggest-tech-outage/ https://www.telegraph.co.uk/business/2024/07/22/microsoft-bl...
- gruez 2y agoThis isn't your personal computer/homelab where you can get away with using common sense antivirus or even windows defender. Software like crowdstrike are often used in industries where they're mandated to install such software for compliance reasons (eg. PCI-DSS). Even if you were using linux you'd still need to install it, and crowdstrike previously had issues with their linux agent. It was just uncommon enough that it didn't hit the news.
- BillSaysThis 2y ago$5.4B seems way too low given the number of flights Delta cancelled and will be on the hook to refund. https://www.washingtonpost.com/transportation/2024/07/23/delta-flight-cancellations-dot-investigation-crowdstrike-outage/ https://www.washingtonpost.com/transportation/2024/07/23/del...
- gruez 2y agoDelta's revenue is only $58.05 billion in 20213. I'm not sure how a day or two of canceled flights is going to be anywhere near $5.4B.
- hysan 2y agoDelta only started to recover today, so it’s more like 4-5 days of canceled flights. Not saying it’s a huge difference but 1-2 greatly underplays how bad it was for people flying Delta vs any other airline.
- deleted 2y ago[deleted]
- Khaine 2y agoMost of that was due to knock on impacts. The Delta scheduling software couldn’t handle the number of changes that were being to staff schedules, and routes, and flights due to the impacts of the CrowdStrike outage.
- magic_man 2y agoWay more than delta flights were grounded. If you include all the hours lost by all those people and assign some monetary value it's probably more than that. Not only that, but all the hospitals. God forbid someone died because of it. Surprised crowd strike isn't bankrupt
- red-iron-pine 2y agothe hospitals alone are reasons to excoriate CS, airlines notwithstanding
- betaby 2y agoWell, 'security' check boxes have consequences.
- paxys 2y agoToday in "random number pulled out of someone's ass"
- mrinterweb 2y agoCan't wait for the class action lawsuit. The total impact is likely greater than $5.4B. A significant number of people must have died due to the impact this had on hospitals and emergency services.
- deleted 2y ago[deleted]
- btbuildem 2y agoRealistically speaking, the liability lies with every individual organization that installed the corporate spyware on their systems.
- chasd00 2y agohmm i was talking with my little sister about picking up some of their stock because these things always just blow over and the stock reverts back to where it basically was in time. I don't think many enterprises will switch because of the effort required and, instead, they'll just yell at the account reps for a while and then go back to paying the invoice. However, a big lawsuit is something i didn't think of.
- aeyes 2y agoEveryone signed their terms of use: https://www.crowdstrike.com/software-terms-of-use/ https://www.crowdstrike.com/software-terms-of-use/ Section 6.1: THERE IS NO WARRANTY THAT THE SOFTWARE OR ANY OTHER CROWDSTRIKE OFFERINGS WILL BE ERROR FREE, OR THAT THEY WILL OPERATE WITHOUT INTERRUPTION OR WILL FULFILL ANY OF SOFTWARE USER’S PARTICULAR PURPOSES OR NEEDS. THE SOFTWARE AND ALL OTHER CROWDSTRIKE OFFERINGS ARE NOT FAULT-TOLERANT AND ARE NOT DESIGNED OR INTENDED FOR USE IN ANY HAZARDOUS ENVIRONMENT REQUIRING FAIL-SAFE PERFORMANCE OR OPERATION. NEITHER THE SOFTWARE OR ANY OTHER CROWDSTRIKE OFFERINGS ARE FOR USE IN THE OPERATION OF AIRCRAFT NAVIGATION, NUCLEAR FACILITIES, COMMUNICATION SYSTEMS, WEAPONS SYSTEMS, DIRECT OR INDIRECT LIFE-SUPPORT SYSTEMS, AIR TRAFFIC CONTROL, OR ANY APPLICATION OR INSTALLATION WHERE FAILURE COULD RESULT IN DEATH, SEVERE PHYSICAL INJURY, OR PROPERTY DAMAGE. SOFTWARE USER AGREES THAT IT IS SOFTWARE USER’S RESPONSIBILITY TO ENSURE SAFE USE OF SOFTWARE AND ANY OTHER CROWDSTRIKE OFFERING IN SUCH APPLICATIONS AND INSTALLATIONS.
- ram_rar 2y agoCan someone with a background in contract negotiation, vendor onboarding, and business continuity risk management share their expertise? We'd love to hear about typical vendor contract provisions that protect customers in situations like this. If damages can be demonstrated, what are the chances of airlines successfully claiming compensation? Or, in practice, do such cases usually result in significant discounts during the next contract renewal rather than actual damages paid out?
- com 2y agoIf the liability is capped at the cost of the duration of the incident (70 minutes from Crowdstrike’s PR-messaging perspective) or one month’s service charge - both pretty normal in standard contracts, then it’s only outside of the contracts that some equity could be achieved. Not holding my breath though.
- red-iron-pine 2y ago> We'd love who is 'we'? Most contracts have indemnity clauses that protect from, or cap damages, due to vendor issues. You can get a court to overrule that if you can prove something like gross negligence, or that such provisions don't apply to something like safety-critical airline systems. CS could push back saying they just offer endpoint protection, and it's on your org for where you put it. Kind like Ikea saying "hey man we just make end tables" when someone decides to put Lack tables on every airplane, and they turn out to be super flammable.
- Thaxll 2y agoImagine Apple / Google pushing an update that bricked 2b+ mobile devices.
- EasyMark 2y agoIt's gonna happen someday lol, that's why I keep a backup device (just my previous gen phone) and don't update it except a few times a year.
- sandworm101 2y agoDoes anyone else feel a little sympathy for CrowdStrike? They pushed out something they should not have. OK. That is bad. But a couple days on and the bulk of the difficulties seem to be from how windows handled the situation: The BSODs, the boot loops, the inability to recover from a basic fault. I feel that if this did happen in a linux environment (it could) that it would be easier to isolate and boot systems into some sort of temporary mode. Linux would communicate and offer options. The windows-specific trend of just abandoning all hope, giving up and throwing the BSOD at the user ... CrowdStrike didn't create that.
- betaby 2y agoWhy there should be any sympathy for them? Their business is shitty boss-ware. I care about boss-ware makers as much as I care about tobacco companies.
- timtom123 2y agoNo? They did this on Linux too. https://www.theregister.com/2024/07/21/crowdstrike_linux_crashes_restoration_tools/ https://www.theregister.com/2024/07/21/crowdstrike_linux_cra...
- disruptiveink 2y agoWhat do you mean? They wrote the kernel driver. With great power comes great responsability. If you're writing a kernel driver that is deployed throughout a great portion of Fortune 500, with the money that that entails, then you should definitely be able to afford to pay people to write defensive code and have proper pipelines in place.
- sandworm101 2y agoAnd there is only the one kernel. No easy rollback option at boot, a previous version to at least get the system online.
- monksy 2y ago"We can't hold back releases from going into prod.. we have to deliver" "We don't have enough time to write tests" "Developers should be able to test their own code"
- ajma 2y agoBut they apologized with a $10 gift card. Soo. $5.4B - $10
- throw7 2y agoThe preliminary post incident review is here: https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio... It boils down to the "Content Validator" had a bug and gave a false positive. It's kind of crazy that the 'rapid response content' update was then free to go out direct to production machines with zero actual live testing. That's either due to c-suite excel cost-cutting/maximize profit or silicon valley yolo.
- slaw 2y agoSilicon valley working at midnight? It was a contractor from India.
- Havoc 2y agoDon't worry...they're handing out $10 vouchers to make up for it https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-apology-gift-card-to-say-sorry-for-outage/?guccounter=1 https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-ap...
- mike503 2y agoThat's a lot of $10 (non functional) Uber eats cards
- gkuhl21 2y agoIs Chaos Engineering an appropriate preventative measure for this sort of thing?
- alecco 2y agoI've never seen chaos testing for broken drivers or boot problems. It's usually taking things down ramdomly.
- red-iron-pine 2y ago"take everything down, randomly" but to the parent poster's point... maybe. sometimes you gotta throw a wrench in it and see what happens.
- hksgdk 2y ago[dead]