4 ms·

As one of those "admin/ops" people with experience of fixing terribly set up systems, a common issue is that users just can't tell the difference between system

As one of those "admin/ops" people with experience of fixing terribly set up systems, a common issue is that users just can't tell the difference between systems that work insecurely and systems that work securely, but they will immediately notice if a system does not work because the security policy is too strict. You get the best security when everyone is involved in the security design from the "ground up", but quite often there's not enough communication between the people developing some application, whose work may be valued by the number of features they ship and "velocity", and the operations side of things whose work is to provide the infrastructure for running the software, and to keep it running. At worst you just get some 3rd-party consultant to set up a thing somehow and then afterwards have to reverse-engineer it to figure out what the hell they did and how to prevent it from going up in flames immediately.

mlthoughts2018 6y ago
I always find the lack of communication goes the other direction. Security and compliance teams just enforce an IAM policy without talking to application or product teams. It gets rolled out and lots of things break, even things that have legitimate needs to work the way they do and have considered security best practices heavily already, and then security and compliance just throws their hands up and says too bad, refactor it from first principles regardless of the level of effort, staffing requirements, competing priorities, etc.
- Spivak 6y ago
  I have never seen a compliance team make the choice to just break production unless they (and by proxy you) are in hot water with auditors.
  - mlthoughts2018 6y ago
    That’s weird, because I see it constantly, even for minor systems where the relationship to a compliance requirement is minor / optional. I’ve actually never seen it happen when there is a real security or auditor issue at stake - I’ve only (repeatedly) seen compliance & security teams demand enforcement of a policy that breaks production in circumstances where the whole thing could have been easily prevented if they had gone to product teams and had a conversation first, but they didn’t. The most recent one I lived through a few months ago was when compliance just all of a sudden decided to wholsesale enforce a bunch or org-wide settings changes to every GitHub repo in the company, and it caused several outages and a huge amount of unplanned triage work as the settings were very sensitive for a bunch of continuous integration systems and jobs. This was at a Fortune 500 company with a big, well-staffed compliance team. They had to roll back their changes and delay the new settings by several months because only through breaking production did they realize their proposes settings workflow was not feasible given in-house system requirements. And of course, no apologies at all. This is pretty run of the mill. I’ve seen the same thing from compliance and security teams in a few other large, “household name” tech companies, and also in a few mid-range startups. Compliance teams number one MO is to blame product teams for not partnering with them, but it’s the compliance teams who refuse to do the partnering.
    - tatersolid 6y ago
      > I’ve only (repeatedly) seen compliance & security teams demand enforcement of a policy that breaks production in circumstances where the whole thing could have been easily prevented if they had gone to product teams and had a conversation first, but they didn’t. This is exactly backwards. Product devs need to reach out to security early in the design phase. There’s no way for a separate security org to understand the app or use case after the fact. If you want to do $newthing your product dev management needs to involve security, finance, compliance, legal, etc. That’s their job. Developers don’t get to ignore all the normal business constraints the real world offers. Building within constraints is what engineering is all about.

mlthoughts2018 6y ago

I always find the lack of communication goes the other direction. Security and compliance teams just enforce an IAM policy without talking to application or product teams. It gets rolled out and lots of things break, even things that have legitimate needs to work the way they do and have considered security best practices heavily already, and then security and compliance just throws their hands up and says too bad, refactor it from first principles regardless of the level of effort, staffing requirements, competing priorities, etc.

Spivak 6y ago
I have never seen a compliance team make the choice to just break production unless they (and by proxy you) are in hot water with auditors.
- mlthoughts2018 6y ago
  That’s weird, because I see it constantly, even for minor systems where the relationship to a compliance requirement is minor / optional. I’ve actually never seen it happen when there is a real security or auditor issue at stake - I’ve only (repeatedly) seen compliance & security teams demand enforcement of a policy that breaks production in circumstances where the whole thing could have been easily prevented if they had gone to product teams and had a conversation first, but they didn’t. The most recent one I lived through a few months ago was when compliance just all of a sudden decided to wholsesale enforce a bunch or org-wide settings changes to every GitHub repo in the company, and it caused several outages and a huge amount of unplanned triage work as the settings were very sensitive for a bunch of continuous integration systems and jobs. This was at a Fortune 500 company with a big, well-staffed compliance team. They had to roll back their changes and delay the new settings by several months because only through breaking production did they realize their proposes settings workflow was not feasible given in-house system requirements. And of course, no apologies at all. This is pretty run of the mill. I’ve seen the same thing from compliance and security teams in a few other large, “household name” tech companies, and also in a few mid-range startups. Compliance teams number one MO is to blame product teams for not partnering with them, but it’s the compliance teams who refuse to do the partnering.
  - tatersolid 6y ago
    > I’ve only (repeatedly) seen compliance & security teams demand enforcement of a policy that breaks production in circumstances where the whole thing could have been easily prevented if they had gone to product teams and had a conversation first, but they didn’t. This is exactly backwards. Product devs need to reach out to security early in the design phase. There’s no way for a separate security org to understand the app or use case after the fact. If you want to do $newthing your product dev management needs to involve security, finance, compliance, legal, etc. That’s their job. Developers don’t get to ignore all the normal business constraints the real world offers. Building within constraints is what engineering is all about.