13 ms·

CrowdStrike ex-employees: 'Quality control was not part of our process'

565 points by everybodyknows 2y ago

Thompson88 2y ago
Confirmed: It was a management problem. I noticed in CrowdStrike's post mortem report that they didn't say a thing about management.
nailer 2y ago
It’s a UX designer. I don’t particularly like crowdstrike, but this person will know very little about their kernel Drivers.
ricardobayes 2y ago
I believe one of the biggest bad trends of the software industry as a whole is cutting down on QA/testing effort. A buggy product is almost always an unsuccessful one.
- breadwinner 2y ago
  Blame Facebook and Google for that. They became successful without QA engineers, so the rest of the industry decided to follow suit in an effort to stay modern.
bitcharmer 2y ago
Another company that got MBA-ified
sersi 2y ago
Crowdstrike was heavily pushed on us at a previous company both for compliance reason by some of our clients (BCG were the ones pushing us to use crowdstrike) and from our liability insurance company. It was really an uphill battle to convince everyone not to use Crowdstrike. Eventually I managed to but after many meetings where I had to spend a significant amount of time convincing different shareholders. I'm sure a lot of people just fold and go with them.
- mikeocool 2y ago
  Curious — did you go with a different EDR solution? Or were you able to convince people not to roll one out at all?
- wesselbindt 2y ago
  What made you unwilling to use CS at the time?
manvillej 2y ago
anyone feel like this and Boeing sound remarkably similar? Its almost like there is a lesson for executives here. hmmmm
- bitcharmer 2y ago
  The only lesson for these people is loss of bonuses. This will keep happening for as long as golden parachutes are a thing.
  - wesselbindt 2y ago
    How can we get rid of golden parachutes?
bmitc 2y ago
Has anyone actually worked at a place where quality control was treated as important? I wouldn't consider this exactly surprising.
- 6h6n56 2y ago
  Nope. Did everyone forget the tech motto "move fast and break things"? Where is the room for quality control in that philosophy? Corps won't even put resource into anti-fraud efforts if they believe the millions being stolen from their bottom line isn't worth the effort. I have seen this attitude working in FAANGS. None of this will change until tech workers stop being sadists and actually unionize.
- sudosysgen 2y ago
  Yes, at a trading company, where important central systems had a multiweek testing process (unless the change was marked as urgent, in which case it was faster) with a dedicated team and a full replica environment which would replay historical functions 1:1 (or in some cases live), and every change needed to have an automated rollback process. Unsurprising since it directly affects the bottom line.
  - bmitc 2y ago
    Very interesting. Thanks for sharing. > every change needed to have an automated rollback process How did you accomplish that?
    - sudosysgen 2y ago
      We had a state management and deployment system through which all changes were effected that would automatically rollback changes if the smoke test failed, or if one of the ops staff found an issue.
- m3047 2y ago
  Yes. It was a manufacturing facility and since the products were photosensitive the entire line operated in total darkness. It was two months before they turned the lights on and I could see what I was programming for. This was the first place I saw standups. [Edit: this was the 1990s] They were run by and for the "meat", the people running the line. "Level 2" only got to speak if we were blocked, or to briefly describe any new investigations we would be undertaking. Weirdly (maybe?) they didn't drug test. I thought of all the places I've worked, they would. But they didn't. They were firmly committed to the "no SPOFs" doctrine and had a "tap out" policy: if anyone felt you were distracted, they could "tap you out" for the day. It was no fault. I was there for six months and three or four times I was tapped out and (after the first time, because they asked what I did with my time off the first time) told to "go climb a rock". I tapped somebody out once, for what later gossip suggested was a family issue.
tonetheman 2y ago
[dead]
hitekker 2y ago
I was surprised by how dismissive these comments are. Former staff members, engineers included, are claiming that their former company's unsafe development culture contributed to a colossal world-wide outage & other previous outages. These employee's allegations ought to be seen as credible, or at least as informative. Instead, many seem to be attacking the UX designer commenting on 'Quality control was not part of our process'. My guess is that people are identifying with sentence said just before: "Speed [of shipping] is everything." Aka "Move fast and break things." The culture described by this article must mirror many of our lived experiences. The pure pleasure of shipping code, putting out fires, making an impact (positive or negative)... and then leaving it to the next engineers & managers to sort out, ignoring the mess until it explodes. Even when it does, no one gets blamed for the outage and soon everyone goes back to building features that get them promoted, regardless of quality. Through that ZIRP light, these process failures must look like a feature, not a bug. The emphasis on "quality" must also look like annoying roadblocks in the way of having fun on the customer's dime.
- wesselbindt 2y ago
  There's folks out there who enjoy putting out proverbial fires? I find rework like that quite frustrating
  - MichaelZuo 2y ago
    Well there are a handful of expert consultants who do, since they charge an eye watering price per hour for putting out fires.
  - hitekker 2y ago
    Absolutely. Some people are born firefighters. Nothing wrong with that. I once worked with a senior engineer who loved running incidents. He felt it was real engineering. He loved debugging thorny problems on a strict timeline, getting every engineer in a room and ordering them about, while also communicating widely to the company. Then, there's the rush of the all-clear and the kudos from stakeholders. Specific to his situation, I think he enjoyed the inflated ownership that the sudden urgency demanded. The system we owned was largely taken for granted by the org; a dead-end for a career. Calling incidents was a good way to get visibility at low-cost, i.e., no one would follow-up on our postmortem action items. It eventually became a problem, though, when the system we owned was essentially put into maintenance mode, aka zero development velocity. Then I estimate (balancing for other variables) the rate the senior engineer called an incident for not-incidents went up by 3x...
addled 2y ago
Yesterday morning I learned that someone I was acquainted with had just passed away and the funeral is scheduled for next week. They recently had a stroke at home just days after spending over a month in the hospital. Then I remembered that they were originally supposed to be getting an important surgery, but it was delayed because of the CrowdStrike outage. It took weeks for the stars to align again and the surgery to happen. It makes me wonder what the outcome would have been if they had gotten the surgery done that day, and not spent those extra weeks in the hospital with their condition and stressing about their future?
- oehpr 2y ago
  I appreciate your post here and I'm glad you shared, because it's an example of a distributed harm. One of millions to shake out of this incident, that doesn't have a dollar figure, so it doesn't really "count". To illustrate: If I were to do something horrible like kick a 3 year olds knee out and cripple them for life, I would be rightly labeled a monster. But If I were to say... advocate for education reform to push American Sign Language out of schools, so that deaf children grow up without a developmental language? We don't have words for that, and if we did, none of them would get near the cumulative scope and harm of that act. We simply do not address distributed harms correctly. And a big part of it is that we don't, we can't, see all the tangible harms it causes.
- namdnay 2y ago
  Not to defend Crowdstrike in any way, but it’s a bit unfair to only look at the downside. What if his hospital hadn’t bought an antivirus, and got hit by ransomware?
  - addled 2y ago
    Sure, and even if the surgery happened on time, they still might have had a stroke once they got home and had the same outcome. But as other posts on HN have discussed, anecdotes, especially your own, hit differently. It makes me thankful the software I work on isn't involved in life and death situations... But then again, it causes me to better consider the things my work could be responsible for (banking). Rushed work that causes a loan application to fail or transaction to be held unnecessarily shouldn't kill someone outright, but there can be real consequences that affect real people just like Rita.
SlightlyLeftPad 2y ago
Just another example of technical leadership being completely irresponsible and another example of tech companies prioritizing the wrong things. As a security company, this completely blows their credibility. i’m not convinced they learned anything from this and don’t expect this effect to change anything. This is a culture issue, not a technical one. One RCA isn’t going to change this. Reliability is a critical facet of security from a business continuity standpoint. Any business still using crowdstrike is out of their mind.
mrjin 2y ago
Wasn't that obvious? If any tests were performed at all, how anyone can manage to caused an outage at such scale?
MichaelRo 2y ago
This "security" thing is getting ridiculous. It's become the Gestapo of information technology, they can do anything they want when they want to your computer, cannot resist it and there's absolutely no transparency on what they do to you and why. I've recently changed jobs and the new employer, a large company, obviously has to have an IT compliance / security update policy because everyone else has it so if they stand out from the crowd and don't do it and somehow get hacked, it's 100x worse than constantly annoying employees and top of the line computers working like a 1970s terminal. It's rarely that a week passes without the obligatory update + restart. And at least once a month they update THE FUCKING BIOS! What the fuck can be so broken in those laptops that the BIOS is a constant security hazard?! And why would you buy software from someone who week after week after week tell you all you had so far was a hazardous piece of shit that cannot possibly function without constant pampering? Ahh and of course they botch it. Had to have the OS completely wiped out and reinstalled after the laptop started to behave more and more erratically, 100% caused by faulty updates on top of faulty patches trying to patch the faulty updates. Worked OK for a while afterwards then updates started piling up and so far I only lost use of the web camera (before it was Wifi then display adapter). There's literally no words how much I hate "the system" and the constant security update take it up the ass we're forced to put up with.
brownllc 2y ago
Our law firm, Brown, LLC, would like to speak with you to discuss your experience at Crowdstrike. Is there a good time and way to do this? You can call our office at (877) 561-0000 or view our site www.IFightForYourRights.com. Thank you and best of luck.
Sarkie 2y ago
It was shown in the RCA that their QA processes were shit
Alupis 2y ago
> “Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card. Look, I think just as poorly about CrowdStrike as anyone else out there... but you can find someone to say anything, especially when they have an axe to grind and a chance at some spotlight. Not to mention this guy was a designer and wouldn't be involved in QC anyway. > Of the 24 former employees who spoke to Semafor, 10 said they were laid off or fired and 14 said they left on their own. One was at the company as recently as this summer. Three former employees disagreed with the accounts of the others. Joey Victorino, who spent a year at the company before leaving in 2023, said CrowdStrike was “meticulous about everything it was doing.” So basically we have nothing.
- wpietri 2y ago
  > So basically we have nothing. No, what we have is a publication who is claiming that the people they talked to were credible and had points that were interesting and tended to match one another and/or other evidence. You can make the claim that Semafor is bad at their jobs, or even that they're malicious. But that's a hard claim to make given that in the paragraph you've quoted they are giving you the contrary evidence that they found. And this is a process many of us have done informally. When we talk to one ex-employee of a company, well maybe it was just that guy, or just where he was in the company. But when a bunch of people have the same complaint, it's worth taking it much more seriously.
- lr4444lr 2y ago
  In principle yes, I agree that former employees' sentiments have an obvious bias, but if they all trend in the same direction - people who worked in different times and functions and didn't know each other while on the job - that points to a likely underlying truth.
- _heimdall 2y ago
  I do agree with having to expect bias there, but who else do you really expect to speak out?Any current employee would very quickly become an ex-employee if they speak out with any specifics. I would expect any contractor that may have worked for CrowdStrike, or done something like a third-party audit, would be under an NDA covering their work. Who's left to speak out with any meaningful details?
monksy 2y ago
No shit.
nine_zeros 2y ago
Typical of tech companies these days. Quality is considered immaterial - or worse - put on low level managers and engineers who don't have the time to clearly examine quality and good roll out practices. C-Suite and investors don't seem to want to spend on quality. They should just price in that their stock investment could collapse any day.
- dgfitz 2y ago
  [flagged]
  - neverrroot 2y ago
    This ^^
0xbadcafebee 2y ago
Critical software infrastructure should be regulated the way critical physical infrastructure is. We don't trust the people who make buildings and bridges to "do the right thing" - we mandate it with regulations and inspections. (When your software not working strands millions of people around the globe, it's critical) And this was just a regular old "accident"; imagine the future, when a war has threat actors trying to knock things out.
- theideaofcoffee 2y ago
  "We can't regulate the industry because then the US loses to China" or "regulation will kill the US competitive advantage!" responses I've had to suggesting the same and I just can't. But I agree with you 100%. If it's safety critical, it should be under even more scrutiny than other things, it shouldn't be left to self-regulating QA-like processes in profit seeking companies and has to have a bit more scrutiny before the big button gets pressed. Edit: Disclaimer: The quotes aren't mine, just retorts I've received from others when I suggest the R-word.
  - janalsncm 2y ago
    > then the US loses to China Yeah it makes no sense. Was the US not losing to China when we own-goaled the biggest cybersecurity incident in history?
    - worik 2y ago
      > then the US loses to China Such a silly meme, too. Economics 101 China and USA would both benefit by halting the conflict and trading with each other
  - Zigurd 2y ago
    Not to mention humans going extinct because regulators are to blame for there being no city on Mars. Because that's definitely the reason there's no city on Mars.
- owl57 2y ago
pclmulqdq 2y ago
Everything that we know about CrowdStrike stinks of Knight Capital to me. A minor culture problem snowballed into complete dysfunction, eventually resulting in a company-ending bug.
- ForOldHack 2y ago
  Knight Capitol: "$10 million a minute. That’s about how much the trading problem that set off turmoil on the stock market on Wednesday morning is already costing the trading firm. The Knight Capital Group announced on Thursday that it lost $440 million when it sold all the stocks it accidentally bought Wednesday morning because a computer glitch. " Glitch. Oh... https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
  - 0cf8612b2e1e 2y ago
    I do not work in finance, but surely every trading company has had an algorithm go wild at some point. Just becomes a matter of how fast someone can pull the circuit breaker before the expensive failure becomes public.
    - bitcharmer 2y ago
      We have circuit breakers for that very purpose. Everyone on the street does. It's just that theirs seems to have failed for some reason.
      - pclmulqdq 2y ago
        Theirs didn't fail, and they did have one. The circuit breaker they had that would have worked was a big red button that killed all of their trading processes, which would have meant spending the rest of the day figuring out and unwinding their positions. Ihey were unwilling to push that button in the short time they had. If you read the reports to the SEC or the articles about it, you will note that. The follow-ups recommended that all firms adopt a big red button that is less catastrophic.
bb88 2y ago
Most interesting quote in the article: “It was hard to get people to do sufficient testing sometimes,” said Preston Sego, who worked at CrowdStrike from 2019 to 2023. His job was to review the tests completed by user experience developers that alerted engineers to bugs before proposed coding changes were released to customers. Sego said he was fired in February 2023 as an “insider threat” after he criticized the company’s return to-work policy on an internal Slack channel. Okay clearly that company has a culture issue. Imagine criticizing a policy and then getting labeled "insider threat".
- wesselbindt 2y ago
  > return to work I know you're just quoting the phrase, but what a gross and dishonest way of phrasing "return to office". Implies working remotely doesn't count as work. Smacks of PR. Yuck.
- nullvoxpopuli 2y ago
  I'd like to clarify: that my job was also to educate, modernize, and improve developer velocity through tooling and framework updates / changes (impacting every team in my department (UX / frontend engineering)). Reviewing tests is part of PR review. --- and before anyone asks, this is my statement on CrowdStrike calling everyone disgruntled: "I'm not disgruntled. But as a shareholder (and probably more primarily, someone who cares about coworkers), I am disappointed. For the most part, I'm still mourning the loss of working with the UX/Platform team."
  - bb88 2y ago
    I mourn the fact that your ex co-workers are still working for a shitty company.
    - nullvoxpopuli 2y ago
      The market for jobs isn't great, so i don't blame them. At the same time, i feel like big profit-chasing software companies are all like how CrowdStrike is. Many may be in the same type of company, but situations have not arisen that reveal how leadership really feels about employees.
seanw444 2y ago
And everybody gasped in surprise.
tamimio 2y ago
I think the whole world knew that already.
insane_dreamer 2y ago
> CrowdStrike disputed much of Semafor’s reporting I expect some ex-employees to be disgruntled and present things in a way that makes CroudStrike look bad. That happens with every company. BUT, CrowdStrike has ZERO credibility at this point. I don't believe a word they say.
- Zigurd 2y ago
  At some companies, like Boeing, the shorter list would be the gruntled employees.
  - insane_dreamer 2y ago
    > gruntled have never heard that word used is a non-negative way
    - dbattaglia 2y ago
      I’ve only heard it from Michael Scott: “Everyone here is extremely gruntled”.
    - beng-nl 2y ago
      Off-Topic, but do I have a story for you https://www.ling.upenn.edu/~beatrice/humor/how-i-met-my-wife.html https://www.ling.upenn.edu/~beatrice/humor/how-i-met-my-wife...
    - tsimionescu 2y ago
      Fun linguistics fact, but gruntled as the antonym of disgruntled is a back-formation. The word disgruntled is a bit strange, in that it uses "dis-" not as a reversal prefix (such as in dissatisfied or dissimilar), but as an intensifier. The original "gruntle" was related to grunt, grunting, it was similar to "grumble", denoting the sounds an annoyed crowd might make. But this old sense of gruntle, gruntling, gruntled has not been used since the 16th century. And in the past century, people have started back-forming a new "gruntle" by analyzing "dis-gruntled" as using the more common meaning of "dis-". A similar use of dis- as an intensifier apparently happened in "dismayed" (here from an Old French verb, esmaier , which meant to trouble, to disturb), and in "disturbed" (from Latin a word, turba, meaning turmoil). I haven't heard any one say they are "mayed" or "turbed", but people would probably see the same as "gruntled" if you used them.
chaps 2y ago
Worked on a team that deployed crowdstrike agents to organize and... Yeah. One of the biggest problems we had was that the daemon would log a massive amount of stuff... But had no config for it to stop or reduce it.
st3fan 2y ago
Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They probably have many many thousands of plain text secrets from their customers stored in their SIEM.
- batch12 2y ago
  Anyone with the right level of access to your Falcon instance can run commands on your endpoints (using RTR) and collect any data not already being collected.
- deleted 2y ago
  [deleted]
- x3n0ph3n3 2y ago
  Can you provide some more info on this? How do you know? Is this documented somewhere? I'm sure this is going to raise red-flags in my IT department.
  - st3fan 2y ago
    Ask them to search for the usual env var names like GITHUB_TOKEN or AWS_ACCESS_KEY_ID.
  - skewer99 2y ago
    AKIDs... ugh. They'll be there if you use AWS + Mac. Again, the plaintext is the problem. These environment variables get loaded from the command line, scripts, etc. - CrowdStrike and all of the best EDRs also collect and send home all of that, but probably in an encrypted stream?
    - zxexz 2y ago
      I usually remote dev on an instance in a VPC because of crap like this. If you like terrible ideas (I don't use this except for debugging IAM stuff, occasionally), you can use the IMDS like you were an AWS instance by giving a local loopback device the link-local ipv4 address 169.254.169.254/32 and binding traffic on the instance's 169.254.169.254/32 port 80 to your lo's port 80, and a local AWS SDK will use the IAM instance profile of the instance you're connected to. I'll repeat, this is not a good idea.
avree 2y ago
"“Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” Their 'expert' on engineering process is a senior UX designer? Somehow, I doubt they were very close to the kernel patch deployment process.
- acdha 2y ago
  They probably weren’t, but that still speaks to their general culture and is compatible with what we know about their kernel engineering culture (limited testing, no review, no use of common fail safe mechanisms).
  - esperent 2y ago
    > is compatible with what we know In other words, it confirms our biases and we're willing to accept it at face value despite there being only a single anecdotal piece of evidence.
    - acdha 2y ago
      It sounds like you might want to read their technical report. That’s neither anecdotal nor a single point, and it showed a pretty large gap in engineering leadership with numerous areas well behind the state of the art. That’s why I said it was compatible: both these former employees and their own report showed an emphasis on shipping rapidly but not the willingness to invest serious money in the safeguards needed to do so safely. If you want to construct another theory, feel free to do so.
  - hello_moto 2y ago
    A company can have different business units with different culture/mentality. I bet my ass anyone working in low-level code don't ship the way you do in Cloud.
    - acdha 2y ago
panic 2y ago
Why would it matter? The absolute worst case scenario happened and their stock is still up 50% YoY, beating the S&P 500.
- 0cf8612b2e1e 2y ago
  I thought you were joking. The stock market is incredible. Everyone must realize that crowdstrike has a captive audience with no alternatives that can meet corporate compliance.
  - intelVISA 2y ago
    Can't think of a bigger flex of how locked-in their market share is. On the plus side this should spur some disruptors into gear, assuming VCs are willing to pivot from wasting money funding LLM wrappers.
- hyperpape 2y ago
  It’s down 30% since the incident, and flat since 3 years ago. If it runs up a huge amount in the first half of the year and then the incident knocks off 30% of their market, that still means the incident was really bad.
  - hello_moto 2y ago
    Their stock has always been volatile but you can't ignore the fact that it hasn't been that bad after the incident.
goralph 2y ago
What are some alternatives to CrowdStrike?
- taspeotis 2y ago
  Personal: Nothing - Windows Defender is built into Windows. Business: Nothing - Windows Defender Advanced Threat Protection is built into the higher Microsoft 365 license tiers. It amazes me people chose to pay money to have all their PCs bluescreen.
  - neverrroot 2y ago
    This is a good example of very limited thinking.
  - Aeolun 2y ago
    mdatp is also a virus. So slow…
    - taspeotis 2y ago
      It can record some telemetry to help you understand why it's slow: https://learn.microsoft.com/en-us/defender-endpoint/troubleshoot-performance-issues https://learn.microsoft.com/en-us/defender-endpoint/troubles...
  - digitalsushi 2y ago
    if you had used 'some' before 'people' i could agree but some industries have to use a siem or they can be fined, so, i mean if there's a list of siems that are definitely not going to ever crash by messing around in the kernel lets get a list going
    - willy_k 2y ago
      Luckily the concern isn’t simply whether they could make a mistake and cause a crash by easing around in the kernel, it’s whether they’re likely to, and I’d argue that CrowdStrike is particularly likely to do so given their testing and rollout processes, and the culture that encompasses those failures
ramesh31 2y ago
If their (or your) shop is anything like mine, its' been a constant whittling of ancillary support roles (SDET, QA, SRE) and a shoving of all of the above into the sole responsibility of devs over the last few years. None of this is surprising at all.
nittanymount 2y ago
does it have competitors ?
xyst 2y ago
Switch off CrowdStrike junk. Those companies renewing contracts with them have idiots for leaders. Many competing platforms that can be a drop in placement for ClownStrike.
paulcole 2y ago
Well if they say that QA was part of the process then they’ll look like idiots because they sucked at the process. Don’t find this particularly interesting news.
hinkley 2y ago
I have only just begun to consider this question: when does risk taking become thrill seeking? At some point you go past questions of laziness or discipline and it becomes a neurosis. Like an addiction.
Cyclone_ 2y ago
Not justifying what they did with qc, but qc is missing from quite a few places in software development that I've been apart of. People might get the impression from the article that every software project is well tested, whereas in my experience most are rushed out.
- Borborygymus 2y ago
  Exactly. Much of the discourse around this topic has described ideal testing and deployment practise. Maybe it's different in Silicon Valley or investment banks, but for the sorts of companies I work for (telco mostly) things are very far from that ideal. My view of he industry is one of shocking technical ineptitude from all but a minority of very competent people who actually keep things running... Of management who prioritize short term cost reduction over quality at every opportunity, leading to appalling technical debt and demoralized, over-worked staff who rapidly stop giving a damn about quality, because speaking out about quality problems is penalized.
- padjo 2y ago
  I’ve worked for several multi billion dollar software companies. None of them had a dedicated QA function by design. Everything is about moving fast. That culture is ok if you’re making entertainment software or low criticality business software. It’s a very bad idea for critical software. Unfortunately the “move fast” attitude has metastasised to places where it has no place .
mattfrommars 2y ago
Side effect of the old adage, "move fast, fail fast"?
- deleted 2y ago
  [deleted]
Timber-6539 2y ago
Doesn't matter now. CRWD didn't go to zero. Meaning they get the chance to do this again.
noisy_boy 2y ago
Would be interesting to know from their employees if there have been any tangible changes in the blind pursuit of velocity, better QA etc in the aftermath of this fiasco.
known 2y ago
[dead]
jrm4 2y ago
Does anyone have a logical reason why this company should not be sued into oblivion?
- superposeur 2y ago
  Yes, because in point of fact this company is the best at what it does — preventing security breaches. The outage — disruptive as it was — was not a breach. This elemental fact is lost amidst all the knee jerk HN hate, but goes a long way toward explaining why the stock only took a modest hit.
  - hun3 2y ago
    That's a somewhat narrow definition of "security." The 3rd component of the CIA triad is often overlooked, yet the availability is what makes the protected asset—and, transitively, the protection itself—useful at the first place. The disruption is effectively a Denial of Service.
jokoon 2y ago
We need laws and regulations on software the same way we have for toys, cars, airplanes, boats, buildings. This silicon valley libertarian non sense needs to stop.
Thompson88 2y ago
Clearly there weren't any code review workflow processes in place, which is astonishing. That's why our primary focus is-transparency, accountability, and system integrity to bring a decentralized, transparent, and reliable platform for journalists, researchers, scientists, and content creators.

Confirmed: It was a management problem. I noticed in CrowdStrike's post mortem report that they didn't say a thing about management.

nailer 2y ago

It’s a UX designer. I don’t particularly like crowdstrike, but this person will know very little about their kernel Drivers.

ricardobayes 2y ago

I believe one of the biggest bad trends of the software industry as a whole is cutting down on QA/testing effort. A buggy product is almost always an unsuccessful one.

breadwinner 2y ago
Blame Facebook and Google for that. They became successful without QA engineers, so the rest of the industry decided to follow suit in an effort to stay modern.

bitcharmer 2y ago

Another company that got MBA-ified

sersi 2y ago

Crowdstrike was heavily pushed on us at a previous company both for compliance reason by some of our clients (BCG were the ones pushing us to use crowdstrike) and from our liability insurance company. It was really an uphill battle to convince everyone not to use Crowdstrike. Eventually I managed to but after many meetings where I had to spend a significant amount of time convincing different shareholders. I'm sure a lot of people just fold and go with them.

mikeocool 2y ago
Curious — did you go with a different EDR solution? Or were you able to convince people not to roll one out at all?
wesselbindt 2y ago
What made you unwilling to use CS at the time?

manvillej 2y ago

anyone feel like this and Boeing sound remarkably similar? Its almost like there is a lesson for executives here. hmmmm

bitcharmer 2y ago
The only lesson for these people is loss of bonuses. This will keep happening for as long as golden parachutes are a thing.
- wesselbindt 2y ago
  How can we get rid of golden parachutes?

bmitc 2y ago

Has anyone actually worked at a place where quality control was treated as important? I wouldn't consider this exactly surprising.

6h6n56 2y ago
Nope. Did everyone forget the tech motto "move fast and break things"? Where is the room for quality control in that philosophy? Corps won't even put resource into anti-fraud efforts if they believe the millions being stolen from their bottom line isn't worth the effort. I have seen this attitude working in FAANGS. None of this will change until tech workers stop being sadists and actually unionize.
sudosysgen 2y ago
Yes, at a trading company, where important central systems had a multiweek testing process (unless the change was marked as urgent, in which case it was faster) with a dedicated team and a full replica environment which would replay historical functions 1:1 (or in some cases live), and every change needed to have an automated rollback process. Unsurprising since it directly affects the bottom line.
- bmitc 2y ago
  Very interesting. Thanks for sharing. > every change needed to have an automated rollback process How did you accomplish that?
  - sudosysgen 2y ago
    We had a state management and deployment system through which all changes were effected that would automatically rollback changes if the smoke test failed, or if one of the ops staff found an issue.
m3047 2y ago
Yes. It was a manufacturing facility and since the products were photosensitive the entire line operated in total darkness. It was two months before they turned the lights on and I could see what I was programming for. This was the first place I saw standups. [Edit: this was the 1990s] They were run by and for the "meat", the people running the line. "Level 2" only got to speak if we were blocked, or to briefly describe any new investigations we would be undertaking. Weirdly (maybe?) they didn't drug test. I thought of all the places I've worked, they would. But they didn't. They were firmly committed to the "no SPOFs" doctrine and had a "tap out" policy: if anyone felt you were distracted, they could "tap you out" for the day. It was no fault. I was there for six months and three or four times I was tapped out and (after the first time, because they asked what I did with my time off the first time) told to "go climb a rock". I tapped somebody out once, for what later gossip suggested was a family issue.

tonetheman 2y ago

[dead]

hitekker 2y ago

I was surprised by how dismissive these comments are. Former staff members, engineers included, are claiming that their former company's unsafe development culture contributed to a colossal world-wide outage & other previous outages. These employee's allegations ought to be seen as credible, or at least as informative. Instead, many seem to be attacking the UX designer commenting on 'Quality control was not part of our process'. My guess is that people are identifying with sentence said just before: "Speed [of shipping] is everything." Aka "Move fast and break things." The culture described by this article must mirror many of our lived experiences. The pure pleasure of shipping code, putting out fires, making an impact (positive or negative)... and then leaving it to the next engineers & managers to sort out, ignoring the mess until it explodes. Even when it does, no one gets blamed for the outage and soon everyone goes back to building features that get them promoted, regardless of quality. Through that ZIRP light, these process failures must look like a feature, not a bug. The emphasis on "quality" must also look like annoying roadblocks in the way of having fun on the customer's dime.

wesselbindt 2y ago
There's folks out there who enjoy putting out proverbial fires? I find rework like that quite frustrating
- MichaelZuo 2y ago
  Well there are a handful of expert consultants who do, since they charge an eye watering price per hour for putting out fires.
- hitekker 2y ago
  Absolutely. Some people are born firefighters. Nothing wrong with that. I once worked with a senior engineer who loved running incidents. He felt it was real engineering. He loved debugging thorny problems on a strict timeline, getting every engineer in a room and ordering them about, while also communicating widely to the company. Then, there's the rush of the all-clear and the kudos from stakeholders. Specific to his situation, I think he enjoyed the inflated ownership that the sudden urgency demanded. The system we owned was largely taken for granted by the org; a dead-end for a career. Calling incidents was a good way to get visibility at low-cost, i.e., no one would follow-up on our postmortem action items. It eventually became a problem, though, when the system we owned was essentially put into maintenance mode, aka zero development velocity. Then I estimate (balancing for other variables) the rate the senior engineer called an incident for not-incidents went up by 3x...

addled 2y ago

Yesterday morning I learned that someone I was acquainted with had just passed away and the funeral is scheduled for next week. They recently had a stroke at home just days after spending over a month in the hospital. Then I remembered that they were originally supposed to be getting an important surgery, but it was delayed because of the CrowdStrike outage. It took weeks for the stars to align again and the surgery to happen. It makes me wonder what the outcome would have been if they had gotten the surgery done that day, and not spent those extra weeks in the hospital with their condition and stressing about their future?

oehpr 2y ago
I appreciate your post here and I'm glad you shared, because it's an example of a distributed harm. One of millions to shake out of this incident, that doesn't have a dollar figure, so it doesn't really "count". To illustrate: If I were to do something horrible like kick a 3 year olds knee out and cripple them for life, I would be rightly labeled a monster. But If I were to say... advocate for education reform to push American Sign Language out of schools, so that deaf children grow up without a developmental language? We don't have words for that, and if we did, none of them would get near the cumulative scope and harm of that act. We simply do not address distributed harms correctly. And a big part of it is that we don't, we can't, see all the tangible harms it causes.
namdnay 2y ago
Not to defend Crowdstrike in any way, but it’s a bit unfair to only look at the downside. What if his hospital hadn’t bought an antivirus, and got hit by ransomware?
- addled 2y ago
  Sure, and even if the surgery happened on time, they still might have had a stroke once they got home and had the same outcome. But as other posts on HN have discussed, anecdotes, especially your own, hit differently. It makes me thankful the software I work on isn't involved in life and death situations... But then again, it causes me to better consider the things my work could be responsible for (banking). Rushed work that causes a loan application to fail or transaction to be held unnecessarily shouldn't kill someone outright, but there can be real consequences that affect real people just like Rita.

SlightlyLeftPad 2y ago

Just another example of technical leadership being completely irresponsible and another example of tech companies prioritizing the wrong things. As a security company, this completely blows their credibility. i’m not convinced they learned anything from this and don’t expect this effect to change anything. This is a culture issue, not a technical one. One RCA isn’t going to change this. Reliability is a critical facet of security from a business continuity standpoint. Any business still using crowdstrike is out of their mind.

mrjin 2y ago

Wasn't that obvious? If any tests were performed at all, how anyone can manage to caused an outage at such scale?

MichaelRo 2y ago

This "security" thing is getting ridiculous. It's become the Gestapo of information technology, they can do anything they want when they want to your computer, cannot resist it and there's absolutely no transparency on what they do to you and why. I've recently changed jobs and the new employer, a large company, obviously has to have an IT compliance / security update policy because everyone else has it so if they stand out from the crowd and don't do it and somehow get hacked, it's 100x worse than constantly annoying employees and top of the line computers working like a 1970s terminal. It's rarely that a week passes without the obligatory update + restart. And at least once a month they update THE FUCKING BIOS! What the fuck can be so broken in those laptops that the BIOS is a constant security hazard?! And why would you buy software from someone who week after week after week tell you all you had so far was a hazardous piece of shit that cannot possibly function without constant pampering? Ahh and of course they botch it. Had to have the OS completely wiped out and reinstalled after the laptop started to behave more and more erratically, 100% caused by faulty updates on top of faulty patches trying to patch the faulty updates. Worked OK for a while afterwards then updates started piling up and so far I only lost use of the web camera (before it was Wifi then display adapter). There's literally no words how much I hate "the system" and the constant security update take it up the ass we're forced to put up with.

brownllc 2y ago

Our law firm, Brown, LLC, would like to speak with you to discuss your experience at Crowdstrike. Is there a good time and way to do this? You can call our office at (877) 561-0000 or view our site www.IFightForYourRights.com. Thank you and best of luck.

Sarkie 2y ago

It was shown in the RCA that their QA processes were shit

Alupis 2y ago

> “Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card. Look, I think just as poorly about CrowdStrike as anyone else out there... but you can find someone to say anything, especially when they have an axe to grind and a chance at some spotlight. Not to mention this guy was a designer and wouldn't be involved in QC anyway. > Of the 24 former employees who spoke to Semafor, 10 said they were laid off or fired and 14 said they left on their own. One was at the company as recently as this summer. Three former employees disagreed with the accounts of the others. Joey Victorino, who spent a year at the company before leaving in 2023, said CrowdStrike was “meticulous about everything it was doing.” So basically we have nothing.

wpietri 2y ago
> So basically we have nothing. No, what we have is a publication who is claiming that the people they talked to were credible and had points that were interesting and tended to match one another and/or other evidence. You can make the claim that Semafor is bad at their jobs, or even that they're malicious. But that's a hard claim to make given that in the paragraph you've quoted they are giving you the contrary evidence that they found. And this is a process many of us have done informally. When we talk to one ex-employee of a company, well maybe it was just that guy, or just where he was in the company. But when a bunch of people have the same complaint, it's worth taking it much more seriously.
lr4444lr 2y ago
In principle yes, I agree that former employees' sentiments have an obvious bias, but if they all trend in the same direction - people who worked in different times and functions and didn't know each other while on the job - that points to a likely underlying truth.
_heimdall 2y ago
I do agree with having to expect bias there, but who else do you really expect to speak out?Any current employee would very quickly become an ex-employee if they speak out with any specifics. I would expect any contractor that may have worked for CrowdStrike, or done something like a third-party audit, would be under an NDA covering their work. Who's left to speak out with any meaningful details?

monksy 2y ago

No shit.

nine_zeros 2y ago

Typical of tech companies these days. Quality is considered immaterial - or worse - put on low level managers and engineers who don't have the time to clearly examine quality and good roll out practices. C-Suite and investors don't seem to want to spend on quality. They should just price in that their stock investment could collapse any day.

dgfitz 2y ago
[flagged]
- neverrroot 2y ago
  This ^^

0xbadcafebee 2y ago

Critical software infrastructure should be regulated the way critical physical infrastructure is. We don't trust the people who make buildings and bridges to "do the right thing" - we mandate it with regulations and inspections. (When your software not working strands millions of people around the globe, it's critical) And this was just a regular old "accident"; imagine the future, when a war has threat actors trying to knock things out.

theideaofcoffee 2y ago
"We can't regulate the industry because then the US loses to China" or "regulation will kill the US competitive advantage!" responses I've had to suggesting the same and I just can't. But I agree with you 100%. If it's safety critical, it should be under even more scrutiny than other things, it shouldn't be left to self-regulating QA-like processes in profit seeking companies and has to have a bit more scrutiny before the big button gets pressed. Edit: Disclaimer: The quotes aren't mine, just retorts I've received from others when I suggest the R-word.
- janalsncm 2y ago
  > then the US loses to China Yeah it makes no sense. Was the US not losing to China when we own-goaled the biggest cybersecurity incident in history?
  - worik 2y ago
    > then the US loses to China Such a silly meme, too. Economics 101 China and USA would both benefit by halting the conflict and trading with each other
- Zigurd 2y ago
  Not to mention humans going extinct because regulators are to blame for there being no city on Mars. Because that's definitely the reason there's no city on Mars.
owl57 2y ago

pclmulqdq 2y ago

Everything that we know about CrowdStrike stinks of Knight Capital to me. A minor culture problem snowballed into complete dysfunction, eventually resulting in a company-ending bug.

ForOldHack 2y ago
Knight Capitol: "$10 million a minute. That’s about how much the trading problem that set off turmoil on the stock market on Wednesday morning is already costing the trading firm. The Knight Capital Group announced on Thursday that it lost $440 million when it sold all the stocks it accidentally bought Wednesday morning because a computer glitch. " Glitch. Oh... https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
- 0cf8612b2e1e 2y ago
  I do not work in finance, but surely every trading company has had an algorithm go wild at some point. Just becomes a matter of how fast someone can pull the circuit breaker before the expensive failure becomes public.
  - bitcharmer 2y ago
    We have circuit breakers for that very purpose. Everyone on the street does. It's just that theirs seems to have failed for some reason.
    - pclmulqdq 2y ago
      Theirs didn't fail, and they did have one. The circuit breaker they had that would have worked was a big red button that killed all of their trading processes, which would have meant spending the rest of the day figuring out and unwinding their positions. Ihey were unwilling to push that button in the short time they had. If you read the reports to the SEC or the articles about it, you will note that. The follow-ups recommended that all firms adopt a big red button that is less catastrophic.

bb88 2y ago

Most interesting quote in the article: “It was hard to get people to do sufficient testing sometimes,” said Preston Sego, who worked at CrowdStrike from 2019 to 2023. His job was to review the tests completed by user experience developers that alerted engineers to bugs before proposed coding changes were released to customers. Sego said he was fired in February 2023 as an “insider threat” after he criticized the company’s return to-work policy on an internal Slack channel. Okay clearly that company has a culture issue. Imagine criticizing a policy and then getting labeled "insider threat".

wesselbindt 2y ago
> return to work I know you're just quoting the phrase, but what a gross and dishonest way of phrasing "return to office". Implies working remotely doesn't count as work. Smacks of PR. Yuck.
nullvoxpopuli 2y ago
I'd like to clarify: that my job was also to educate, modernize, and improve developer velocity through tooling and framework updates / changes (impacting every team in my department (UX / frontend engineering)). Reviewing tests is part of PR review. --- and before anyone asks, this is my statement on CrowdStrike calling everyone disgruntled: "I'm not disgruntled. But as a shareholder (and probably more primarily, someone who cares about coworkers), I am disappointed. For the most part, I'm still mourning the loss of working with the UX/Platform team."
- bb88 2y ago
  I mourn the fact that your ex co-workers are still working for a shitty company.
  - nullvoxpopuli 2y ago
    The market for jobs isn't great, so i don't blame them. At the same time, i feel like big profit-chasing software companies are all like how CrowdStrike is. Many may be in the same type of company, but situations have not arisen that reveal how leadership really feels about employees.

seanw444 2y ago

And everybody gasped in surprise.

tamimio 2y ago

I think the whole world knew that already.

insane_dreamer 2y ago

> CrowdStrike disputed much of Semafor’s reporting I expect some ex-employees to be disgruntled and present things in a way that makes CroudStrike look bad. That happens with every company. BUT, CrowdStrike has ZERO credibility at this point. I don't believe a word they say.

Zigurd 2y ago
At some companies, like Boeing, the shorter list would be the gruntled employees.
- insane_dreamer 2y ago
  > gruntled have never heard that word used is a non-negative way
  - dbattaglia 2y ago
    I’ve only heard it from Michael Scott: “Everyone here is extremely gruntled”.
  - beng-nl 2y ago
    Off-Topic, but do I have a story for you https://www.ling.upenn.edu/~beatrice/humor/how-i-met-my-wife.html https://www.ling.upenn.edu/~beatrice/humor/how-i-met-my-wife...
  - tsimionescu 2y ago
    Fun linguistics fact, but gruntled as the antonym of disgruntled is a back-formation. The word disgruntled is a bit strange, in that it uses "dis-" not as a reversal prefix (such as in dissatisfied or dissimilar), but as an intensifier. The original "gruntle" was related to grunt, grunting, it was similar to "grumble", denoting the sounds an annoyed crowd might make. But this old sense of gruntle, gruntling, gruntled has not been used since the 16th century. And in the past century, people have started back-forming a new "gruntle" by analyzing "dis-gruntled" as using the more common meaning of "dis-". A similar use of dis- as an intensifier apparently happened in "dismayed" (here from an Old French verb, esmaier , which meant to trouble, to disturb), and in "disturbed" (from Latin a word, turba, meaning turmoil). I haven't heard any one say they are "mayed" or "turbed", but people would probably see the same as "gruntled" if you used them.

chaps 2y ago

Worked on a team that deployed crowdstrike agents to organize and... Yeah. One of the biggest problems we had was that the daemon would log a massive amount of stuff... But had no config for it to stop or reduce it.

st3fan 2y ago

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They probably have many many thousands of plain text secrets from their customers stored in their SIEM.

batch12 2y ago
Anyone with the right level of access to your Falcon instance can run commands on your endpoints (using RTR) and collect any data not already being collected.
deleted 2y ago
[deleted]
x3n0ph3n3 2y ago
Can you provide some more info on this? How do you know? Is this documented somewhere? I'm sure this is going to raise red-flags in my IT department.
- st3fan 2y ago
  Ask them to search for the usual env var names like GITHUB_TOKEN or AWS_ACCESS_KEY_ID.
- skewer99 2y ago
  AKIDs... ugh. They'll be there if you use AWS + Mac. Again, the plaintext is the problem. These environment variables get loaded from the command line, scripts, etc. - CrowdStrike and all of the best EDRs also collect and send home all of that, but probably in an encrypted stream?
  - zxexz 2y ago
    I usually remote dev on an instance in a VPC because of crap like this. If you like terrible ideas (I don't use this except for debugging IAM stuff, occasionally), you can use the IMDS like you were an AWS instance by giving a local loopback device the link-local ipv4 address 169.254.169.254/32 and binding traffic on the instance's 169.254.169.254/32 port 80 to your lo's port 80, and a local AWS SDK will use the IAM instance profile of the instance you're connected to. I'll repeat, this is not a good idea.

avree 2y ago

"“Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” Their 'expert' on engineering process is a senior UX designer? Somehow, I doubt they were very close to the kernel patch deployment process.

acdha 2y ago
They probably weren’t, but that still speaks to their general culture and is compatible with what we know about their kernel engineering culture (limited testing, no review, no use of common fail safe mechanisms).
- esperent 2y ago
  > is compatible with what we know In other words, it confirms our biases and we're willing to accept it at face value despite there being only a single anecdotal piece of evidence.
  - acdha 2y ago
    It sounds like you might want to read their technical report. That’s neither anecdotal nor a single point, and it showed a pretty large gap in engineering leadership with numerous areas well behind the state of the art. That’s why I said it was compatible: both these former employees and their own report showed an emphasis on shipping rapidly but not the willingness to invest serious money in the safeguards needed to do so safely. If you want to construct another theory, feel free to do so.
- hello_moto 2y ago
  A company can have different business units with different culture/mentality. I bet my ass anyone working in low-level code don't ship the way you do in Cloud.
  - acdha 2y ago

panic 2y ago

Why would it matter? The absolute worst case scenario happened and their stock is still up 50% YoY, beating the S&P 500.

0cf8612b2e1e 2y ago
I thought you were joking. The stock market is incredible. Everyone must realize that crowdstrike has a captive audience with no alternatives that can meet corporate compliance.
- intelVISA 2y ago
  Can't think of a bigger flex of how locked-in their market share is. On the plus side this should spur some disruptors into gear, assuming VCs are willing to pivot from wasting money funding LLM wrappers.
hyperpape 2y ago
It’s down 30% since the incident, and flat since 3 years ago. If it runs up a huge amount in the first half of the year and then the incident knocks off 30% of their market, that still means the incident was really bad.
- hello_moto 2y ago
  Their stock has always been volatile but you can't ignore the fact that it hasn't been that bad after the incident.

goralph 2y ago

What are some alternatives to CrowdStrike?

taspeotis 2y ago
Personal: Nothing - Windows Defender is built into Windows. Business: Nothing - Windows Defender Advanced Threat Protection is built into the higher Microsoft 365 license tiers. It amazes me people chose to pay money to have all their PCs bluescreen.
- neverrroot 2y ago
  This is a good example of very limited thinking.
- Aeolun 2y ago
  mdatp is also a virus. So slow…
  - taspeotis 2y ago
    It can record some telemetry to help you understand why it's slow: https://learn.microsoft.com/en-us/defender-endpoint/troubleshoot-performance-issues https://learn.microsoft.com/en-us/defender-endpoint/troubles...
- digitalsushi 2y ago
  if you had used 'some' before 'people' i could agree but some industries have to use a siem or they can be fined, so, i mean if there's a list of siems that are definitely not going to ever crash by messing around in the kernel lets get a list going
  - willy_k 2y ago
    Luckily the concern isn’t simply whether they could make a mistake and cause a crash by easing around in the kernel, it’s whether they’re likely to, and I’d argue that CrowdStrike is particularly likely to do so given their testing and rollout processes, and the culture that encompasses those failures

ramesh31 2y ago

If their (or your) shop is anything like mine, its' been a constant whittling of ancillary support roles (SDET, QA, SRE) and a shoving of all of the above into the sole responsibility of devs over the last few years. None of this is surprising at all.

nittanymount 2y ago

does it have competitors ?

xyst 2y ago

Switch off CrowdStrike junk. Those companies renewing contracts with them have idiots for leaders. Many competing platforms that can be a drop in placement for ClownStrike.

paulcole 2y ago

Well if they say that QA was part of the process then they’ll look like idiots because they sucked at the process. Don’t find this particularly interesting news.

hinkley 2y ago

I have only just begun to consider this question: when does risk taking become thrill seeking? At some point you go past questions of laziness or discipline and it becomes a neurosis. Like an addiction.

Cyclone_ 2y ago

Not justifying what they did with qc, but qc is missing from quite a few places in software development that I've been apart of. People might get the impression from the article that every software project is well tested, whereas in my experience most are rushed out.

Borborygymus 2y ago
Exactly. Much of the discourse around this topic has described ideal testing and deployment practise. Maybe it's different in Silicon Valley or investment banks, but for the sorts of companies I work for (telco mostly) things are very far from that ideal. My view of he industry is one of shocking technical ineptitude from all but a minority of very competent people who actually keep things running... Of management who prioritize short term cost reduction over quality at every opportunity, leading to appalling technical debt and demoralized, over-worked staff who rapidly stop giving a damn about quality, because speaking out about quality problems is penalized.
padjo 2y ago
I’ve worked for several multi billion dollar software companies. None of them had a dedicated QA function by design. Everything is about moving fast. That culture is ok if you’re making entertainment software or low criticality business software. It’s a very bad idea for critical software. Unfortunately the “move fast” attitude has metastasised to places where it has no place .

mattfrommars 2y ago

Side effect of the old adage, "move fast, fail fast"?

deleted 2y ago
[deleted]

Timber-6539 2y ago

Doesn't matter now. CRWD didn't go to zero. Meaning they get the chance to do this again.

noisy_boy 2y ago

Would be interesting to know from their employees if there have been any tangible changes in the blind pursuit of velocity, better QA etc in the aftermath of this fiasco.

known 2y ago

[dead]

jrm4 2y ago

Does anyone have a logical reason why this company should not be sued into oblivion?

superposeur 2y ago
Yes, because in point of fact this company is the best at what it does — preventing security breaches. The outage — disruptive as it was — was not a breach. This elemental fact is lost amidst all the knee jerk HN hate, but goes a long way toward explaining why the stock only took a modest hit.
- hun3 2y ago
  That's a somewhat narrow definition of "security." The 3rd component of the CIA triad is often overlooked, yet the availability is what makes the protected asset—and, transitively, the protection itself—useful at the first place. The disruption is effectively a Denial of Service.

jokoon 2y ago

We need laws and regulations on software the same way we have for toys, cars, airplanes, boats, buildings. This silicon valley libertarian non sense needs to stop.

Thompson88 2y ago

Clearly there weren't any code review workflow processes in place, which is astonishing. That's why our primary focus is-transparency, accountability, and system integrity to bring a decentralized, transparent, and reliable platform for journalists, researchers, scientists, and content creators.