4 ms·

Sorry, sometimes breaking prod is your fault. As a senior engineer there is no excuse for forgetting a where clause on your SQL delete. And doubly no excuse for

Sorry, sometimes breaking prod is your fault. As a senior engineer there is no excuse for forgetting a where clause on your SQL delete. And doubly no excuse for not having somebody else look at it.

theli0nheart 5y ago
Humans forget things and make mistakes, no matter what your job title is. Good systems and organizations take this into account and are fault tolerant.
- mattm 5y ago
  That's true but a senior engineer is expected to know when they are doing something potentially risky and should know when to ask someone to double check their work.
  - fsociety 5y ago
    That’s all and good if your senior engineer is a program. But they are a human, and humans are prone to making silly mistakes. Instead systems should be designed such that you can’t easily take them down due to operator errors.
dilyevsky 5y ago
This is a bad take. Everyone makes mistakes. I once saw principal engineer take down 15 thousand machine cluster offline bc he fat-fingered a command. The “blameless” part of pm culture is designed to address process and architecture rather than individual failures. Ofc if someone is intentionally subverting the process that’s a different story…
- darkerside 5y ago
  Are you saying that wasn't his fault? I agree it's a false dichotomy between the process failing and people sharing blame.
  - joshuamorton 5y ago
    Yeah, if the difference between the command to do something safe and the command to do something dangerous is 1 character or whatnot, you can't blame someone when they screw up and fat finger the dangerous one. Most of the tools I write require explicit --nodryrun if they're doing something dangerous or irreversible, and beyond that the really scary ones require some kind of acknowledgement that "THIS WILL TURN 1073 MACHINES OFF, ARE YOU CERTAIN, TYPE 1073 TO CONFIRM YOU WANT TO DISABLE 1073 MACHINES."
    - darkerside 5y ago
      I would 1000% expect a principal engineer to know the difference between a safe and unsafe process and refuse to engage in an unsafe one without many redundant safety checks. It's the principal engineer's job to make the unsafe process safe, is it not?
      - joshuamorton 5y ago
        Ah yes, just magically know that the command in the playbook isn't safe and refuse to use it during an incident. I want to meet your principal engineers who can fully understand a command's complete set of side effects solely from a bash snippet telling you how to invoke it! Like yes, it is the principal's (well really everyone's, but sure insofar as someone is ultimately responsible, we can make the principal engineer responsible) responsibility to ensure processes are safe. But you can't do that by magically "refusing to engage in an unsafe process". Because it's difficult to know a priori if a process is inherently unsafe. Usually you figure this out when something goes wrong, and then you take that learning from a postmortem/incident report and try to apply it to other similar tools. But no, I wouldn't blame anyone, not even a principal engineer whose job it is to ensure safe systems for accidentally fat fingering a dangerous command that they didn't know about. They perhaps bear some responsibility for not having previously fixed it, but they bear approximately none for misusing a tool that makes itself easy to misuse.
hacker_newz 5y ago
It's a process problem. If an engineer shouldn't be able to run a SQL delete query without a where clause, then there should be processes, procedures, or tooling in place to prevent that.

Sorry, sometimes breaking prod is your fault. As a senior engineer there is no excuse for forgetting a where clause on your SQL delete. And doubly no excuse for not having somebody else look at it.

theli0nheart 5y ago

Humans forget things and make mistakes, no matter what your job title is. Good systems and organizations take this into account and are fault tolerant.

mattm 5y ago
That's true but a senior engineer is expected to know when they are doing something potentially risky and should know when to ask someone to double check their work.
- fsociety 5y ago
  That’s all and good if your senior engineer is a program. But they are a human, and humans are prone to making silly mistakes. Instead systems should be designed such that you can’t easily take them down due to operator errors.

dilyevsky 5y ago

This is a bad take. Everyone makes mistakes. I once saw principal engineer take down 15 thousand machine cluster offline bc he fat-fingered a command. The “blameless” part of pm culture is designed to address process and architecture rather than individual failures. Ofc if someone is intentionally subverting the process that’s a different story…

darkerside 5y ago
Are you saying that wasn't his fault? I agree it's a false dichotomy between the process failing and people sharing blame.
- joshuamorton 5y ago
  Yeah, if the difference between the command to do something safe and the command to do something dangerous is 1 character or whatnot, you can't blame someone when they screw up and fat finger the dangerous one. Most of the tools I write require explicit --nodryrun if they're doing something dangerous or irreversible, and beyond that the really scary ones require some kind of acknowledgement that "THIS WILL TURN 1073 MACHINES OFF, ARE YOU CERTAIN, TYPE 1073 TO CONFIRM YOU WANT TO DISABLE 1073 MACHINES."
  - darkerside 5y ago
    I would 1000% expect a principal engineer to know the difference between a safe and unsafe process and refuse to engage in an unsafe one without many redundant safety checks. It's the principal engineer's job to make the unsafe process safe, is it not?
    - joshuamorton 5y ago
      Ah yes, just magically know that the command in the playbook isn't safe and refuse to use it during an incident. I want to meet your principal engineers who can fully understand a command's complete set of side effects solely from a bash snippet telling you how to invoke it! Like yes, it is the principal's (well really everyone's, but sure insofar as someone is ultimately responsible, we can make the principal engineer responsible) responsibility to ensure processes are safe. But you can't do that by magically "refusing to engage in an unsafe process". Because it's difficult to know a priori if a process is inherently unsafe. Usually you figure this out when something goes wrong, and then you take that learning from a postmortem/incident report and try to apply it to other similar tools. But no, I wouldn't blame anyone, not even a principal engineer whose job it is to ensure safe systems for accidentally fat fingering a dangerous command that they didn't know about. They perhaps bear some responsibility for not having previously fixed it, but they bear approximately none for misusing a tool that makes itself easy to misuse.

hacker_newz 5y ago

It's a process problem. If an engineer shouldn't be able to run a SQL delete query without a where clause, then there should be processes, procedures, or tooling in place to prevent that.