5 ms·

> Their job description is not "follow industry best practices" or "look for signs of a break in using their tools". Their job is to keep their company and cust

> Their job description is not "follow industry best practices" or "look for signs of a break in using their tools". Their job is to keep their company and customer's data secure. At this job they failed. No. Their job is to use the resources they’ve been allocated to manage risk within the organization to the risk level senior management has agreed to accept. Maybe that shouldn’t be their job, but it is. There isn’t a security team on the planet that gets to dictate security to the rest of the organization or unilaterally make decisions that influence the running of the business to achieve the level of risk mitigation that they themselves would prefer to attain. That is putting aside the fact that defending against a determined nation-state adversary is a nigh-on impossible task that would require countermeasures like ‘don’t connect to the Internet at all’, ‘don’t hire anyone you haven’t personally known since childhood’, ‘hand-deliver your product to your customers’, and other equally impractical mitigations. Nobody outside of Solarwinds knows if their security team succeeded or failed in the mission they were given.

josephg 6y ago
I hear what you're saying and I appreciate that perspective. I agree, and there's also something slippery in that argument taken to its extreme that we need to be careful of. Some people here on HN made the same argument when Equifax leaked personal information about millions of americans. They said it was ultimately management's fault and not the engineers' fault for not allocating enough resources to security. And the same argument was used by the engineers who made the Therac-25 radiology machine. In that case, software bugs resulted in a handful of deaths due to lethal radiation. Upper management can't be responsible for everything that happens in their business. Engineering isn't their job or their expertise. Thats why they hire software engineers and security engineers - to be the local experts. We need to bear responsibility for the decisions we make in our field. And engineers have a duty not just to the companies we work for, but also to society at large. If we leave our personal judgement at the door in the morning, we fail in our duty to society. To go back to the bridge metaphor, if a bridge falls down, its not good enough for the civil engineers involved to blame management for not giving them enough time / budget / whatever. They also bear some responsibility for the disaster. This has been enshrined in case law too, at the Nuremberg trials. "I was just doing my job" wasn't considered a good enough excuse for the guards in WW2 concentration camps. These are big examples, but I think the principle is fractally true. And the inverse also holds. Praise and blame go together. The biomedical engineers in the labs also deserve praise for the covid19 vaccines they've invented, even if upper management told them to do it. We aren't management's slaves.
- marcus0x62 6y ago
  The difference, when it comes to civil engineers and building bridges, is we as a society have recognized their expertise and made it illegal to build a bridge except as designed by a civil engineer. A more apt analogy, in my opinion, to the day to day realities of managing production applications and infrastructure is the regulation surrounding the maintenance of certified aircraft. There are minimum competency standards that are enforced by law, it is unlawful in almost all circumstances for a non-certified person to perform any maintenance or repair on a certified aircraft, and, crucially, an aircraft cannot return to service unless a certified mechanic signs off on the repair. Not the CEO of the company that owns the airplane, not some middle manager, only the expert (mechanic and, sometimes, inspector) can sign off on returning the plane to service. Without that kind of legal cover, management can and will steamroll over anybody who is impeding their initiative of the day.
  - josephg 6y ago
    Sure; but politicians don’t know anything about technology. They usually don’t even decide what’s right and wrong. They take what culture has decided is right and wrong and codify it in law. The law is a trailing, not a leading indicator of ethical practice. Do you think planes were falling out of the sky left and right before those air safety laws came into effect? No. The engineers at some companies pushed for sane, safe practices first. Later they were adopted by the industry and later still they were enshrined in law. Before those laws were passed, airlines still had a duty of care to their passengers, ethically and (I think) legally. Likewise it’s up to us to decide what sane, secure software engineering looks like. Not politicians. Not management. It has to be us. Nobody else is qualified to make those choices. At some point those ideas might be codified in law; but we need to figure out what that looks like first. (And to be clear what you’re arguing for - imagine the reverse. Imagine if inventing security best practices was outsourced to politicians!) The idea that management should feel free to steamroll over their own employees’ judgement for the sake of the initiative of the day is toxic. And that’s exactly the sort of work culture which creates global security issues like this one. Of course a balance has to be reached, but you don’t do anyone any favours by being management (and the law’s) highly paid keyboard.

josephg 6y ago

I hear what you're saying and I appreciate that perspective. I agree, and there's also something slippery in that argument taken to its extreme that we need to be careful of. Some people here on HN made the same argument when Equifax leaked personal information about millions of americans. They said it was ultimately management's fault and not the engineers' fault for not allocating enough resources to security. And the same argument was used by the engineers who made the Therac-25 radiology machine. In that case, software bugs resulted in a handful of deaths due to lethal radiation. Upper management can't be responsible for everything that happens in their business. Engineering isn't their job or their expertise. Thats why they hire software engineers and security engineers - to be the local experts. We need to bear responsibility for the decisions we make in our field. And engineers have a duty not just to the companies we work for, but also to society at large. If we leave our personal judgement at the door in the morning, we fail in our duty to society. To go back to the bridge metaphor, if a bridge falls down, its not good enough for the civil engineers involved to blame management for not giving them enough time / budget / whatever. They also bear some responsibility for the disaster. This has been enshrined in case law too, at the Nuremberg trials. "I was just doing my job" wasn't considered a good enough excuse for the guards in WW2 concentration camps. These are big examples, but I think the principle is fractally true. And the inverse also holds. Praise and blame go together. The biomedical engineers in the labs also deserve praise for the covid19 vaccines they've invented, even if upper management told them to do it. We aren't management's slaves.

marcus0x62 6y ago
The difference, when it comes to civil engineers and building bridges, is we as a society have recognized their expertise and made it illegal to build a bridge except as designed by a civil engineer. A more apt analogy, in my opinion, to the day to day realities of managing production applications and infrastructure is the regulation surrounding the maintenance of certified aircraft. There are minimum competency standards that are enforced by law, it is unlawful in almost all circumstances for a non-certified person to perform any maintenance or repair on a certified aircraft, and, crucially, an aircraft cannot return to service unless a certified mechanic signs off on the repair. Not the CEO of the company that owns the airplane, not some middle manager, only the expert (mechanic and, sometimes, inspector) can sign off on returning the plane to service. Without that kind of legal cover, management can and will steamroll over anybody who is impeding their initiative of the day.
- josephg 6y ago
  Sure; but politicians don’t know anything about technology. They usually don’t even decide what’s right and wrong. They take what culture has decided is right and wrong and codify it in law. The law is a trailing, not a leading indicator of ethical practice. Do you think planes were falling out of the sky left and right before those air safety laws came into effect? No. The engineers at some companies pushed for sane, safe practices first. Later they were adopted by the industry and later still they were enshrined in law. Before those laws were passed, airlines still had a duty of care to their passengers, ethically and (I think) legally. Likewise it’s up to us to decide what sane, secure software engineering looks like. Not politicians. Not management. It has to be us. Nobody else is qualified to make those choices. At some point those ideas might be codified in law; but we need to figure out what that looks like first. (And to be clear what you’re arguing for - imagine the reverse. Imagine if inventing security best practices was outsourced to politicians!) The idea that management should feel free to steamroll over their own employees’ judgement for the sake of the initiative of the day is toxic. And that’s exactly the sort of work culture which creates global security issues like this one. Of course a balance has to be reached, but you don’t do anyone any favours by being management (and the law’s) highly paid keyboard.