4 ms·

That Salesforce outage: Global DNS downfall started by one engineer’s quick fix

19 points by turtlegrids 5y ago

vlovich123 5y ago
> And the engineer who sidestepped Salesforce's carefully crafted policies and took down the platform? "We have taken action with that particular employee," said Dieken. And that’s a clear signal I never want to work under this exec ever.
- spoonjim 5y ago
  Why? A mistake is a mistake but if you are bypassing policies to cowboy code then you deserve to be fired.
  - carlmr 5y ago
    The question to me is not what was written in policy, rather what was normal in the engineering culture at that company. They say they have a process for this, but how many engineers were following this process? How many fixes were deployed as EBFs before? How common was it to use EBFs with existing tried-and-true scripts? How clear is it in this manual process what you're doing? How much pressure was there to do an EBF? Depending on the surrounding culture around this, which I can't know from outside, he could be anything from a scapegoat to a cowboy coder. If you fire scapegoats you don't fix the issue.
  - ckdarby 5y ago
    That is not an acceptable stance at an organisation of this size. Policies don't just exist on paper they're enforced through the process. No employee should just be able to "cowboy" a solution like this. How does someone bypass the policy if the only way to get your code out is through pipeline? That pipeline required other individuals in the company to approve. Even the most basic regulation that Salesforce would have to deal with talks about speration of roles & responsibilities. Great, a dev can approve their code to go out but ops is responsible for deploying that out to production for this very reason.
- _the_inflator 5y ago
  It is pretty neutral. They learned a lot from this outage and it did not read „We fired the person on the spot“.
  - ckdarby 5y ago
    Except that whole section where they talk about why they're not sure the individual did it this way. If the employee wasn't not fired they'd write the reasoning the employee gave or their thought process in better detail. Hard to write any of that when you can't ask the employee because they've been fired.
ckdarby 5y ago
> "We're not blaming one employee," said Chief Availability Officer Darryn Dieken > And the engineer who sidestepped Salesforce's carefully crafted policies and took down the platform? "We have taken action with that particular employee," said Dieken As @vlovich123 has mentioned as well, this is a clear signal to avoid this company because of this exec. The action that should have happened was at Dieken's level. There doesn't seem to be any level of acceptance from Dieken. The entire press release focuses on an individual and not the collective organisation. Look at how much better this reads from a culture standpoint, general PR, leadership, etc: We failed our customers today. Today we found that our processes within our company don't provide the necessary automated checks to make sure our emergency rolls out are done in the same predictable fashion no matter the change. I as Chief Availability Office take this burden on starting with myself and working with my teams have crafted a short term fix and longer term plans to prevent an issue like this from ever happening. We're doing...blah blah

19 points by turtlegrids 5y ago

> And the engineer who sidestepped Salesforce's carefully crafted policies and took down the platform? "We have taken action with that particular employee," said Dieken. And that’s a clear signal I never want to work under this exec ever.

spoonjim 5y ago
Why? A mistake is a mistake but if you are bypassing policies to cowboy code then you deserve to be fired.
- carlmr 5y ago
  The question to me is not what was written in policy, rather what was normal in the engineering culture at that company. They say they have a process for this, but how many engineers were following this process? How many fixes were deployed as EBFs before? How common was it to use EBFs with existing tried-and-true scripts? How clear is it in this manual process what you're doing? How much pressure was there to do an EBF? Depending on the surrounding culture around this, which I can't know from outside, he could be anything from a scapegoat to a cowboy coder. If you fire scapegoats you don't fix the issue.
- ckdarby 5y ago
  That is not an acceptable stance at an organisation of this size. Policies don't just exist on paper they're enforced through the process. No employee should just be able to "cowboy" a solution like this. How does someone bypass the policy if the only way to get your code out is through pipeline? That pipeline required other individuals in the company to approve. Even the most basic regulation that Salesforce would have to deal with talks about speration of roles & responsibilities. Great, a dev can approve their code to go out but ops is responsible for deploying that out to production for this very reason.
_the_inflator 5y ago
It is pretty neutral. They learned a lot from this outage and it did not read „We fired the person on the spot“.
- ckdarby 5y ago
  Except that whole section where they talk about why they're not sure the individual did it this way. If the employee wasn't not fired they'd write the reasoning the employee gave or their thought process in better detail. Hard to write any of that when you can't ask the employee because they've been fired.

ckdarby 5y ago

> "We're not blaming one employee," said Chief Availability Officer Darryn Dieken > And the engineer who sidestepped Salesforce's carefully crafted policies and took down the platform? "We have taken action with that particular employee," said Dieken As @vlovich123 has mentioned as well, this is a clear signal to avoid this company because of this exec. The action that should have happened was at Dieken's level. There doesn't seem to be any level of acceptance from Dieken. The entire press release focuses on an individual and not the collective organisation. Look at how much better this reads from a culture standpoint, general PR, leadership, etc: We failed our customers today. Today we found that our processes within our company don't provide the necessary automated checks to make sure our emergency rolls out are done in the same predictable fashion no matter the change. I as Chief Availability Office take this burden on starting with myself and working with my teams have crafted a short term fix and longer term plans to prevent an issue like this from ever happening. We're doing...blah blah