4 ms·

> At least production got fixed and money kept flowing. You were denied the tools to get your job done. You've put yourself at risk by applying an unapproved w

> At least production got fixed and money kept flowing. You were denied the tools to get your job done. You've put yourself at risk by applying an unapproved workaround. Never ever do this (unless you hold substantial shares). Let the company's bottom line take the fall. If that's the only thing they care about, that's your only way to make the problem visible.

mystifyingpoi 2y ago
Unfortunately the real world isn't black and white. Yes, according to the company policies, I should watch the world burn and do nothing, while looking at the company bleeding money due to customers SLA being broken. Of course, after submitting a ticket to get nmap approved, which takes days. Extra points if I'm on oncall, then racking that sweet incident money is great. But the underlying SRE culture here is that, if you know what you are doing and have a functioning brain of a responsible person, you'd be forgiven a jump over the fence, if it means putting out a fire on the other side of it. We aren't kids.
- bshacklett 2y ago
  There’s a middle ground. Get the appropriate stakeholders involved in the decision, including security. Let security be the ones to keep the system down, if it cones to that. Or, let the business operations folks make the decision to go over security’s head. Either way, this is not something an engineer tasked with fixing an outage should be making the decision on.
  - ibash 2y ago
    > this is not something an engineer tasked with fixing an outage should be making the decision on I don’t get this at all. I’d much prefer a team of highly empowered and highly responsible engineers than impotent engineers who need hand holding in case they make a mistake.
    - bshacklett 2y ago
      Well, good thing that wasn’t what I suggested. Engineers _should_ have leeway in how they resolve issues. As I read, though, you have a company policy which explicitly disallows the action you needed to take to fix the problem (if I misread, my apologies). Getting the stakeholders involved is the responsible thing to do when policies need to be broken. Ideally, the way this kind of situation gets handled should be documented as part of a break-glass policy, so there’s no ambiguity. If that’s not the case, though, the business should get to decide, alongside the policy maker (e.g.: security), whether that policy should be broken as part of an emergency fix, and how to remediate the policy drift after the crisis. If you’re all tight enough that you’re allowed to make these kinds of decisions in the heat of the moment, that’s great, but it should be agreed upon, and documented, beforehand.
- rollcat 2y ago
  Well I found out the hard way that company culture or values can mean nothing if you don't CYA. Granted, the shop was small enough that our team was in charge of both the security policies and ops, but still, on one unfortunate occasion I've stepped outside my area of responsibility to "do what's right" and got punished. The next time I've been in a similar situation - well, I've walked away from the fire and grabbed the popcorn. By the way, I'm still burnt out. This work is stressful. Don't let it take away what's already scarce for you.

mystifyingpoi 2y ago

Unfortunately the real world isn't black and white. Yes, according to the company policies, I should watch the world burn and do nothing, while looking at the company bleeding money due to customers SLA being broken. Of course, after submitting a ticket to get nmap approved, which takes days. Extra points if I'm on oncall, then racking that sweet incident money is great. But the underlying SRE culture here is that, if you know what you are doing and have a functioning brain of a responsible person, you'd be forgiven a jump over the fence, if it means putting out a fire on the other side of it. We aren't kids.

bshacklett 2y ago
There’s a middle ground. Get the appropriate stakeholders involved in the decision, including security. Let security be the ones to keep the system down, if it cones to that. Or, let the business operations folks make the decision to go over security’s head. Either way, this is not something an engineer tasked with fixing an outage should be making the decision on.
- ibash 2y ago
  > this is not something an engineer tasked with fixing an outage should be making the decision on I don’t get this at all. I’d much prefer a team of highly empowered and highly responsible engineers than impotent engineers who need hand holding in case they make a mistake.
  - bshacklett 2y ago
    Well, good thing that wasn’t what I suggested. Engineers _should_ have leeway in how they resolve issues. As I read, though, you have a company policy which explicitly disallows the action you needed to take to fix the problem (if I misread, my apologies). Getting the stakeholders involved is the responsible thing to do when policies need to be broken. Ideally, the way this kind of situation gets handled should be documented as part of a break-glass policy, so there’s no ambiguity. If that’s not the case, though, the business should get to decide, alongside the policy maker (e.g.: security), whether that policy should be broken as part of an emergency fix, and how to remediate the policy drift after the crisis. If you’re all tight enough that you’re allowed to make these kinds of decisions in the heat of the moment, that’s great, but it should be agreed upon, and documented, beforehand.
rollcat 2y ago
Well I found out the hard way that company culture or values can mean nothing if you don't CYA. Granted, the shop was small enough that our team was in charge of both the security policies and ops, but still, on one unfortunate occasion I've stepped outside my area of responsibility to "do what's right" and got punished. The next time I've been in a similar situation - well, I've walked away from the fire and grabbed the popcorn. By the way, I'm still burnt out. This work is stressful. Don't let it take away what's already scarce for you.