10 ms·
Who needs testing when apologizing to your customers is cheaper?
by nilsb 2y ago
Who needs testing when apologizing to your customers is cheaper?
- falcor84 2y agoI would assume that its enterprise customers have an uptime SLA as part of their contract, and that breaching it isn't very cheap for Crowdstrike.
- perbu 2y agoSoftware doesn't have uptime guarantees. They might have time-to-fix on critical issues, though. I assume this is gross negligence, which would leave them open to claims made through courts, though.
- jsiepkes 2y agoI highly doubt their SLA says something about compensating for damages. At most you won't have to pay for the time they were down. And even more ironically; A botched update doesn't mean they are down. It means you are down. So I don't even think their SLA applies to this.
- InsideOutSanta 2y agoYeah, they'll pay with "credits" for the downtime, if what is currently happening even technically qualifies as downtime.
- agrajag 2y agoReputational damage from this is going to be catastrophic. Even if that’s the limit of their liability it’s hard not to see customers leaving en masse.
- dandanua 2y agoThe company will perish, there is no doubt in that.
- icelancer 2y agoExtremely unlikely. This isn't the first blowup Crowdstrike has had; though it's the worst (IIRC), Crowdstrike is "too big to fail" with tons of enterprise customers who have insane switching costs, even after this nonsense. Unfortunately for all of us, Crowdstrike will be around for awhile.
- zik 2y agoBusinesses would be crazy to continue with Crowdstrike after this. It's going to cause billions in losses to a huge number of companies. If I was a risk assessment officer at a large company I'd be speed dialling every alternative right now.
- ajscanlan 2y agoit would be crazy not to at least investigate migration paths away from Crowdstrike, or better redundancies for yourself
- hello_moto 2y agoCybersecurity industry has regular and annual security testing/competitions done by various Organizations that simulates tons of attacks. Vendors are tested against these cases and graded with their effectiveness. I heard Crowdstrike is "best-in-market" for good reasons as others who have more deep knowledge of the industry have shared in this thread.
- zik 2y ago> I heard Crowdstrike is "best-in-market" A friend of mine who used to work for Crowdstrike tells me they're a hot mess internally and it's amazing they haven't had worse problems than this already.
- hello_moto 2y agoThat sounds like any other companies I have ever worked for: looks great from the outside but a hot mess on the inside. I have never worked for a company where everything is smooth sailing. What I noticed is that the smaller the company, the less hot mess they are but at the same time they're also struggling to pay the bill because they don't innovate fast.
- junto 2y agoIronically some /r/wallstreetbets poster put out an ill-informed “due diligence” post 11 hours ago concerning CrowdStrike being not worth $83 billion and placing puts on the stock. Everybody took the piss out of them for the post. Now they are quite likely to become very rich. https://www.reddit.com/r/wallstreetbets/s/jJ6xHewXXp https://www.reddit.com/r/wallstreetbets/s/jJ6xHewXXp
- BoringTimesGang 2y agoA convenient alibi?
- RateMyPE 2y agoThat user is the equivalent of using a screwdriver to look for gold and succeeding.
- persedes 2y agoWhat's even better is the reaction here: https://www.reddit.com/r/sysadmin/comments/1e6vx6n/comment/ldw0fgf/ https://www.reddit.com/r/sysadmin/comments/1e6vx6n/comment/l...
- deliveryboyman 2y agoNot sure what material in their post is ill-informed. Looks like what happened today is exactly what that poster warned of in one of their bullet points.
- rozap 2y agoYea, everyone is dunking on OP here. But they essentially said that crowdstrike's customers were all vulnerable to something like this. And we saw a similar thing play out only a few years ago with SolarWinds. It's not surprising that this happened. Ofc with making money the timing is the crucial part which is hard to predict.
- dclowd9901 2y agoAnd when it’s more costly for customers to walk back the mistake of adopting your service. Yeah, I get the impression a lot of SaaS companies operate on this model these days. We just signed with a relatively unknown CI platform, because they were available for support during our evaluation. I wonder how available they’ll be when we have a contract in place…
- helsinkiandrew 2y agoAs at 4am NY time CRWD has lost $10Bn (~13%) in marketcap. Of course they've tested, but just not enough for this issue (as is often the case). This is probably several seemingly non consequential issues coming together. I'm not sure why though, when the system is this important that even successfully tested updates aren't rolled out piecemeal though (or perhaps it has and we're only seeing the result of partial failures around the world)
- tehlike 2y agoTesting is never enough. In fact, it won't catch 99% of issues by the virtue of them often testing happy paths only, or that they test what humans can think of, and by no means they are exhaustive. A robust canarying mechanism is the only way you can limit the blast radius. Set up A/B testing infra at the binary level so you can ship updates selectively and compare their metrics. Been doing this for more than 10 years now, it's the ONLY way. Testing is not.
- wwtrv 2y agoDepends on what you mean by enough. It should be more than enough to catch issues like this one specifically. If they can't even manage that they'll fail at your approach as well.
- tehlike 2y agoCanary offers more bang for the buck, and is much easier to set up. So I kind of disagree.
- wwtrv 2y ago> Canary offers more bang for the buck I'm not sure that justifies potentially bricking the devices of hundreds(?) of your clients by shipping untested updates to them. Of course it depends... and would require deeper financial analysis.
- 2y ago
- kjkjadksj 2y agoExactly. They knocked half the world offline probably killed thousands in ERs and the stock is only down to about June lows.
- krspnda 2y agohah that tweet was one heck of an apology. "we deployed a fix to the issue, speak with your customer rep"
- hello_moto 2y agoUnfortunately cybersecurity still revolves around obscurity.