11 ms·

> “Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two y

by Alupis 2y ago

> “Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card. Look, I think just as poorly about CrowdStrike as anyone else out there... but you can find someone to say anything, especially when they have an axe to grind and a chance at some spotlight. Not to mention this guy was a designer and wouldn't be involved in QC anyway. > Of the 24 former employees who spoke to Semafor, 10 said they were laid off or fired and 14 said they left on their own. One was at the company as recently as this summer. Three former employees disagreed with the accounts of the others. Joey Victorino, who spent a year at the company before leaving in 2023, said CrowdStrike was “meticulous about everything it was doing.” So basically we have nothing.

wpietri 2y ago
> So basically we have nothing. No, what we have is a publication who is claiming that the people they talked to were credible and had points that were interesting and tended to match one another and/or other evidence. You can make the claim that Semafor is bad at their jobs, or even that they're malicious. But that's a hard claim to make given that in the paragraph you've quoted they are giving you the contrary evidence that they found. And this is a process many of us have done informally. When we talk to one ex-employee of a company, well maybe it was just that guy, or just where he was in the company. But when a bunch of people have the same complaint, it's worth taking it much more seriously.
lr4444lr 2y ago
In principle yes, I agree that former employees' sentiments have an obvious bias, but if they all trend in the same direction - people who worked in different times and functions and didn't know each other while on the job - that points to a likely underlying truth.
_heimdall 2y ago
I do agree with having to expect bias there, but who else do you really expect to speak out?Any current employee would very quickly become an ex-employee if they speak out with any specifics. I would expect any contractor that may have worked for CrowdStrike, or done something like a third-party audit, would be under an NDA covering their work. Who's left to speak out with any meaningful details?
_fat_santa 2y ago
> Quality control was not really part of our process or our conversation. Is anyone really surprised or learned any new information? For us that have worked for tech companies, this is one of those repeating complaints that you hear across orgs that indicates a less than stellar engineering culture. I've worked with numerous F500 orgs and I would say 3/5 orgs that I worked in, their code was so bad that it made me wonder how they haven't had a major incident yet.
skenderbeu 2y ago
Disgruntled are the Crowdstrike customers that had to deal with the outage. These employees have a lot of reputation to lose for coming forward. Crowstrike is a disgrace of a company and many others like it are doing the same behaviors but they just haven't gotten caught yet. Software development has become a disgrace when the bottom line of squeezing margins to please investors took over.
- darby_nine 2y ago
  > These employees have a lot of reputation to lose for coming forward. Employees don't typically have much reputation to ruin. I am perfectly content putting this on their shoulders.
iudqnolq 2y ago
There are some very specific accusations backed up by non-denials from crowdstrike. Ex-employees said bugs caused the log monitor to drop entries. Crowdstrike responded the project was never designed to alert in real time. But Crowdstrike's website currently advertises it as working in real time. Ex-employees said people trained to monitor laptops were assigned to monitor AWS accounts with no extra training. Crowdstrike replied that "there were no experienced ‘cloud threat hunters’ to be had" in 2022 and that optional training was available to the employees.
nyc_data_geek1 2y ago
>>So basically we have nothing. Except the biggest IT outage ever. And a postmortem showing their validation checks were insufficient. And a rollout process that did not stage at all, just rawdogged straight to global prod. And no lab where the new code was actually installed and run prior to global rawdogging. I'd say there's smoke, and numerous accounts of fire, which this can be taken in the context of.
- nickcocker 2y ago
  [dead]
- mewpmewp2 2y ago
  There definitely was a huge outage, but based on the given information we still can't know for sure how much they invested in testing and quality control. There's always a chance of failure even for the most meticulous companies. Now I'm not defending or excusing the company, but a singular event like this can happen to anyone and nothing is 100%. If thorough investigation revealed poor quality control investment compared to what would be appropriate for a company like this, then we can say for sure.
  - hulitu 2y ago
    [flagged]
    - dang 2y ago
      Could you please stop posting unsubstantive comments and/or flamebait? Posts like this one and https://news.ycombinator.com/item?id=41542151 https://news.ycombinator.com/item?id=41542151 are definitely not what we're trying for on HN. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
  - daedrdev 2y ago
    Two things are clear though Nobody ran this update The update was pushed globally to all computers With that alone we know they have failed the simplest of quality control methods for a piece of software as widespread as theirs. This is even excluding that there should have been some kind of error handling to allow the computer to boot if they did push bad code.
sonofhans 2y ago
If design isn’t involved in QC you’re not doing QC very well. If design isn’t plugged into development process enough to understand QC then you’re not doing design very well.
- tw04 2y ago
  Why would a UX designer be involved in any way, shape, or form in kernel level code patches? They would literally never ship an update if they had that many hands in the pot for something completely unrelated. Should they also have their sales reps and marketing folks pre-brief before they make any code changes?
  - zipy124 2y ago
    I would agree if it was a UI designer, but a good UX designer designs for the users, which in this case including the system admins who will be updating kernel level code patches. Ensuring they have a good experience e.g no crashes, is their job. A recommendation would likely be for example small roll-outs to minimise the number of people having a bad user experience on a roll-out that goes wrong.
  - sonofhans 2y ago
    A UX designer might have told them it was a bad idea to deploy the patch widely without testing a smaller cohort, for instance. That’s an obvious measure that they skipped this time.
    - newshackr 2y ago
      But that doesn't have anything to do with what UX designers typically do
      - diatone 2y ago
        Not true; UX designers typically are responsible for advocating for a robust, intuitive experience for users. The fact that kernel updates don’t have a user interface doesn’t make them exempt from asking the simple question: how will this affect users? And the subsequent question: is there a chance that deploying this eviscerates the user experience? Granted, a company that isn’t focused on the user experience as much as it is on other things might not prioritise this as much in the first place.
darby_nine 2y ago
I feel like crowdstrike is perfectly capable of mounting its own defense
JumpCrisscross 2y ago
> This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card To you and me, maybe. To the insurers and airlines paying out over the problem, maybe not.
bdcravens 2y ago
I'm going with principle of least astonishment, where productivity is more highly valued in most companies than quality control.
insane_dreamer 2y ago
> So basically we have nothing. Except the fact that CrowdStrike fucked up the one thing they weren't supposed to fuck up. So yeah, at this point I'm taking the ex-employees' word, because it confirms the results that we already know -- there is no way that update could have gone out had there been proper "safety first" protocols in place and CrowdStrike was "meticulous".
theideaofcoffee 2y ago
I just don't think a company like Crowdstrike has a leg to stand on when leveling the "disgruntled" label in the face of their, let's face it, astoundingly epic fuck up. It's the disgruntled employees that I think would have the most clear picture of what was going on, regardless of them being in QA/QC or not because they, at that point, don't really care any more and will be more forthright with their thoughts. I'd certainly trust their info more than a company yes-man which is probably where some of that opposing messaging came from.
- paulcole 2y ago
  Why would you trust a company no-man any more than a company yes-man? They both have agendas and biases. Is it just that you personally prefer one set of biases (anti-company) more than the other (pro-company)?
  - insane_dreamer 2y ago
    Well, in this case, we know one side (pro-company) fucked up big time. The other side (anti-company) may or may not have fucked up. That makes it easier to trust one side over another.
    - paulcole 2y ago
      You’ve kind of set yourself up in a no-lose situation here. If the employees fucked up then you’ll say the company still fucked up because it wasn’t managing the employees well. And then in that situation you’ll still believe the lying employees who say its the company’s fault while leaving out their culpability.
  - theideaofcoffee 2y ago
    Yes, I am very much biased toward being anti-company and I make no apologies for that. I've been in the corporate world long enough to know first-hand the sins that PR and corporate management commits on the company's behalf and the harm it does. I find information coming from the individual more reliable than having it filtered through corpo PR, legal, ass-covering nonsense, the latter group often wanting to preserve the status quo than getting out actual info.
tooltower 2y ago
This is like online reviews. If you selectively take positive or negative reviews and somehow censor the rest, the reviews are worthless. Yet, if you report on all the ones you find, it's still useful. Yes, I'm more likely to leave reviews if I'm unsatisfied. Yes, people are more likely to leave CS if they were unhappy. Biased data, but still useful data.
denkmoon 2y ago
Well they certainly don't care about the speed of the endpoints their malware runs on. Shit has ruined my macos laptop's performance.
- nullvoxpopuli 2y ago
  All EDR software does (at least on macos) Source: me, a developer who also codes in free time and notices how bad fs perf is especially. I've had the CrowdStrike sensor, and my current company is using cyberhaven. So.. while 2 data points don't technically make a pattern, it does begin to raise suspicion.
Aeolun 2y ago
Honestly, this article describes nearly all companies (from the perspective of the engineers) so I’m not sure I find it hard to believe this one is the same.
zik 2y ago
Here's some anecdotal evidence - a friend worked at CrowdStrike and was horrified at how incredibly disorganised the whole place was. They said it was completely unsurprising to them that the outage occurred. More surprising to them was that it hadn't happened more often given what a clusterfrock the place was.

by Alupis 2y ago

wpietri 2y ago

> So basically we have nothing. No, what we have is a publication who is claiming that the people they talked to were credible and had points that were interesting and tended to match one another and/or other evidence. You can make the claim that Semafor is bad at their jobs, or even that they're malicious. But that's a hard claim to make given that in the paragraph you've quoted they are giving you the contrary evidence that they found. And this is a process many of us have done informally. When we talk to one ex-employee of a company, well maybe it was just that guy, or just where he was in the company. But when a bunch of people have the same complaint, it's worth taking it much more seriously.

lr4444lr 2y ago

In principle yes, I agree that former employees' sentiments have an obvious bias, but if they all trend in the same direction - people who worked in different times and functions and didn't know each other while on the job - that points to a likely underlying truth.

_heimdall 2y ago

I do agree with having to expect bias there, but who else do you really expect to speak out?Any current employee would very quickly become an ex-employee if they speak out with any specifics. I would expect any contractor that may have worked for CrowdStrike, or done something like a third-party audit, would be under an NDA covering their work. Who's left to speak out with any meaningful details?

_fat_santa 2y ago

> Quality control was not really part of our process or our conversation. Is anyone really surprised or learned any new information? For us that have worked for tech companies, this is one of those repeating complaints that you hear across orgs that indicates a less than stellar engineering culture. I've worked with numerous F500 orgs and I would say 3/5 orgs that I worked in, their code was so bad that it made me wonder how they haven't had a major incident yet.

skenderbeu 2y ago

Disgruntled are the Crowdstrike customers that had to deal with the outage. These employees have a lot of reputation to lose for coming forward. Crowstrike is a disgrace of a company and many others like it are doing the same behaviors but they just haven't gotten caught yet. Software development has become a disgrace when the bottom line of squeezing margins to please investors took over.

darby_nine 2y ago
> These employees have a lot of reputation to lose for coming forward. Employees don't typically have much reputation to ruin. I am perfectly content putting this on their shoulders.

iudqnolq 2y ago

There are some very specific accusations backed up by non-denials from crowdstrike. Ex-employees said bugs caused the log monitor to drop entries. Crowdstrike responded the project was never designed to alert in real time. But Crowdstrike's website currently advertises it as working in real time. Ex-employees said people trained to monitor laptops were assigned to monitor AWS accounts with no extra training. Crowdstrike replied that "there were no experienced ‘cloud threat hunters’ to be had" in 2022 and that optional training was available to the employees.

nyc_data_geek1 2y ago

>>So basically we have nothing. Except the biggest IT outage ever. And a postmortem showing their validation checks were insufficient. And a rollout process that did not stage at all, just rawdogged straight to global prod. And no lab where the new code was actually installed and run prior to global rawdogging. I'd say there's smoke, and numerous accounts of fire, which this can be taken in the context of.

nickcocker 2y ago
[dead]
mewpmewp2 2y ago
There definitely was a huge outage, but based on the given information we still can't know for sure how much they invested in testing and quality control. There's always a chance of failure even for the most meticulous companies. Now I'm not defending or excusing the company, but a singular event like this can happen to anyone and nothing is 100%. If thorough investigation revealed poor quality control investment compared to what would be appropriate for a company like this, then we can say for sure.
- hulitu 2y ago
  [flagged]
  - dang 2y ago
    Could you please stop posting unsubstantive comments and/or flamebait? Posts like this one and https://news.ycombinator.com/item?id=41542151 https://news.ycombinator.com/item?id=41542151 are definitely not what we're trying for on HN. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
- daedrdev 2y ago
  Two things are clear though Nobody ran this update The update was pushed globally to all computers With that alone we know they have failed the simplest of quality control methods for a piece of software as widespread as theirs. This is even excluding that there should have been some kind of error handling to allow the computer to boot if they did push bad code.

sonofhans 2y ago

If design isn’t involved in QC you’re not doing QC very well. If design isn’t plugged into development process enough to understand QC then you’re not doing design very well.

tw04 2y ago
Why would a UX designer be involved in any way, shape, or form in kernel level code patches? They would literally never ship an update if they had that many hands in the pot for something completely unrelated. Should they also have their sales reps and marketing folks pre-brief before they make any code changes?
- zipy124 2y ago
  I would agree if it was a UI designer, but a good UX designer designs for the users, which in this case including the system admins who will be updating kernel level code patches. Ensuring they have a good experience e.g no crashes, is their job. A recommendation would likely be for example small roll-outs to minimise the number of people having a bad user experience on a roll-out that goes wrong.
- sonofhans 2y ago
  A UX designer might have told them it was a bad idea to deploy the patch widely without testing a smaller cohort, for instance. That’s an obvious measure that they skipped this time.
  - newshackr 2y ago
    But that doesn't have anything to do with what UX designers typically do
    - diatone 2y ago
      Not true; UX designers typically are responsible for advocating for a robust, intuitive experience for users. The fact that kernel updates don’t have a user interface doesn’t make them exempt from asking the simple question: how will this affect users? And the subsequent question: is there a chance that deploying this eviscerates the user experience? Granted, a company that isn’t focused on the user experience as much as it is on other things might not prioritise this as much in the first place.

darby_nine 2y ago

I feel like crowdstrike is perfectly capable of mounting its own defense

JumpCrisscross 2y ago

> This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card To you and me, maybe. To the insurers and airlines paying out over the problem, maybe not.

bdcravens 2y ago

I'm going with principle of least astonishment, where productivity is more highly valued in most companies than quality control.

insane_dreamer 2y ago

> So basically we have nothing. Except the fact that CrowdStrike fucked up the one thing they weren't supposed to fuck up. So yeah, at this point I'm taking the ex-employees' word, because it confirms the results that we already know -- there is no way that update could have gone out had there been proper "safety first" protocols in place and CrowdStrike was "meticulous".

theideaofcoffee 2y ago

I just don't think a company like Crowdstrike has a leg to stand on when leveling the "disgruntled" label in the face of their, let's face it, astoundingly epic fuck up. It's the disgruntled employees that I think would have the most clear picture of what was going on, regardless of them being in QA/QC or not because they, at that point, don't really care any more and will be more forthright with their thoughts. I'd certainly trust their info more than a company yes-man which is probably where some of that opposing messaging came from.

paulcole 2y ago
Why would you trust a company no-man any more than a company yes-man? They both have agendas and biases. Is it just that you personally prefer one set of biases (anti-company) more than the other (pro-company)?
- insane_dreamer 2y ago
  Well, in this case, we know one side (pro-company) fucked up big time. The other side (anti-company) may or may not have fucked up. That makes it easier to trust one side over another.
  - paulcole 2y ago
    You’ve kind of set yourself up in a no-lose situation here. If the employees fucked up then you’ll say the company still fucked up because it wasn’t managing the employees well. And then in that situation you’ll still believe the lying employees who say its the company’s fault while leaving out their culpability.
- theideaofcoffee 2y ago
  Yes, I am very much biased toward being anti-company and I make no apologies for that. I've been in the corporate world long enough to know first-hand the sins that PR and corporate management commits on the company's behalf and the harm it does. I find information coming from the individual more reliable than having it filtered through corpo PR, legal, ass-covering nonsense, the latter group often wanting to preserve the status quo than getting out actual info.

tooltower 2y ago

This is like online reviews. If you selectively take positive or negative reviews and somehow censor the rest, the reviews are worthless. Yet, if you report on all the ones you find, it's still useful. Yes, I'm more likely to leave reviews if I'm unsatisfied. Yes, people are more likely to leave CS if they were unhappy. Biased data, but still useful data.

denkmoon 2y ago

Well they certainly don't care about the speed of the endpoints their malware runs on. Shit has ruined my macos laptop's performance.

nullvoxpopuli 2y ago
All EDR software does (at least on macos) Source: me, a developer who also codes in free time and notices how bad fs perf is especially. I've had the CrowdStrike sensor, and my current company is using cyberhaven. So.. while 2 data points don't technically make a pattern, it does begin to raise suspicion.

Aeolun 2y ago

Honestly, this article describes nearly all companies (from the perspective of the engineers) so I’m not sure I find it hard to believe this one is the same.

zik 2y ago

Here's some anecdotal evidence - a friend worked at CrowdStrike and was horrified at how incredibly disorganised the whole place was. They said it was completely unsurprising to them that the outage occurred. More surprising to them was that it hadn't happened more often given what a clusterfrock the place was.