4 ms·

In our company we went the other road. We have the developers write the policies (since I mean, they know what their app needs) and test it in dev environments.

In our company we went the other road. We have the developers write the policies (since I mean, they know what their app needs) and test it in dev environments. After that, ops guys step in during code review to check for too broad allow in the policies. So far it seems to work in acceptable manner.

brendoelfrendo 6y ago
This sounds like a good process, but it depends a lot on the relationship between dev and ops... I've seen too many dev shops push against changes requested by ops or security because their main pressure is to ship features fast. And then it turns into a management fight and whoever has the more influential management gets the final say while the other side is forced to grumble.
- sushshshsh 6y ago
  Currently experiencing this at my current employer. I have a suspicion that slowing the dev and deployment process is in everyone's perceived best interest, until of course the day where we are out competed.
  - partyboat1586 6y ago
    And rightly or wrongly you will be outcompeted by places with more risk appetite, until they have a security breach before they are big enough to swallow the cost.
- scarface74 6y ago
  Well, developers can fight all they want to. I was the dev lead at a company that had to be HIPAA compliant. My neck was on the line if we were found to be out of compliance along with the security director and the operations people. Now as a consultant working with many large customers, “shipping fast” is nowhere near as important as security.
  - partyboat1586 6y ago
    I'm in the same position and we haven't built our infrastructure yet. Did you use a managed service? Or did you build it slowly and carefully on AWS? I'm currently stuck between the two options. Managed service seems the easiest way to be HIPAA compliant but I'd rather we managed our own infrastructure on AWS since it gives us more flexibility for stuff like blue green deploys and it would be cheaper.
    - scarface74 6y ago
      Back then, I was building a green field project on prem. It was more about limiting access and auditing. In the middle of the implementation a mandate came from on high to “move to the cloud”. I didn’t know anything about AWS back then, they hired an MSP who was just a bunch of old school netops people who knew how to click around on the console and gave us a bunch of VMs. Long story short, I studied for one AWS certification so I could talk the talk. I learned both all of the things that I could have taken advantage of and saw how much they were making and that changed my whole m.o. and decided to get some experience with AWS and go into consulting. Next company I went to, the founders outsourced everything technically to an outsourcing company - software and infrastructure and they treated AWS as an overpriced colo. Everything was in one account and everyone had access to it. At first, they were just aggregating publicly available information about doctors for hospitals so it wasn’t a big deal. They brought a new CTO in and started bringing development in house. I led the charge to first separate out the environment to different accounts, establish a sane CI/CD process and then lock down who had access to prod. Of course they had secret access keys in config files everywhere. We had to audit the code to make sure that no code was using keys. Locally, every SDK can automatically retrieve the keys from your global config file (that’s nowhere near your git repo) and on AWS it gets permissions based on the attached roles. Then of course we had to lock down roles. But we couldn’t have the granular permissions we needed because even though they had lots of microservices (we sold access to our APIs to businesses). They were all on two “pet” EC2 instances. Next step was to move the .Net Core APIs to Docker/Fargate and further restrict the attached roles to those. Finally, we had to audit all of our AWS dependencies and add encryption where necessary and then sign a BAA with AWS and bring in auditors. By the time I left a month ago, we could pass the needed certifications and expand our offerings. It took a lot of upskilling, hiring an internal ops person (I’m a developer who knows AWS) instead of depending heavily on the MSP. I left for greener pastures - I’m a consultant with AWS.
- Aeolun 6y ago
  We have machines to tell people their policy (in dev) is insecure and will be deleted in 30 minutes. Not really easy to argue with the machine.
  - ozim 6y ago
    Nice that just like having bots in discord. When administrators hide behind bots with decisions so users cannot retaliate. It also is nice combined with shadow bans, user thinks his posts are going through but no one replies. Unfortunately for development it is better to get tight integration of dev and ops so you could solve it by discussion and cooperation. Not sure if you can build such teams that often but that would be great.
thayne 6y ago
In my company, developers write policies, but they have to be approved by someone with ops expertise (usually me). And the policies are often either too broad, or not sufficiently tested, and missing permissions for things it needed. Sometimes the same policy has both problems. I don't blame the developers. You can hardly expect every developer to become an expert on AWS's IAM system. Especially given how inconsistent it can be.
- partyboat1586 6y ago
  >or not sufficiently tested How do you test a policy?
  - thayne 6y ago
    You set up a policy in a test environment and run the code there. Of course, generally, the policy can't be identical across environments, so you can run into errors.

brendoelfrendo 6y ago

This sounds like a good process, but it depends a lot on the relationship between dev and ops... I've seen too many dev shops push against changes requested by ops or security because their main pressure is to ship features fast. And then it turns into a management fight and whoever has the more influential management gets the final say while the other side is forced to grumble.

sushshshsh 6y ago
Currently experiencing this at my current employer. I have a suspicion that slowing the dev and deployment process is in everyone's perceived best interest, until of course the day where we are out competed.
- partyboat1586 6y ago
  And rightly or wrongly you will be outcompeted by places with more risk appetite, until they have a security breach before they are big enough to swallow the cost.
scarface74 6y ago
Well, developers can fight all they want to. I was the dev lead at a company that had to be HIPAA compliant. My neck was on the line if we were found to be out of compliance along with the security director and the operations people. Now as a consultant working with many large customers, “shipping fast” is nowhere near as important as security.
- partyboat1586 6y ago
  I'm in the same position and we haven't built our infrastructure yet. Did you use a managed service? Or did you build it slowly and carefully on AWS? I'm currently stuck between the two options. Managed service seems the easiest way to be HIPAA compliant but I'd rather we managed our own infrastructure on AWS since it gives us more flexibility for stuff like blue green deploys and it would be cheaper.
  - scarface74 6y ago
    Back then, I was building a green field project on prem. It was more about limiting access and auditing. In the middle of the implementation a mandate came from on high to “move to the cloud”. I didn’t know anything about AWS back then, they hired an MSP who was just a bunch of old school netops people who knew how to click around on the console and gave us a bunch of VMs. Long story short, I studied for one AWS certification so I could talk the talk. I learned both all of the things that I could have taken advantage of and saw how much they were making and that changed my whole m.o. and decided to get some experience with AWS and go into consulting. Next company I went to, the founders outsourced everything technically to an outsourcing company - software and infrastructure and they treated AWS as an overpriced colo. Everything was in one account and everyone had access to it. At first, they were just aggregating publicly available information about doctors for hospitals so it wasn’t a big deal. They brought a new CTO in and started bringing development in house. I led the charge to first separate out the environment to different accounts, establish a sane CI/CD process and then lock down who had access to prod. Of course they had secret access keys in config files everywhere. We had to audit the code to make sure that no code was using keys. Locally, every SDK can automatically retrieve the keys from your global config file (that’s nowhere near your git repo) and on AWS it gets permissions based on the attached roles. Then of course we had to lock down roles. But we couldn’t have the granular permissions we needed because even though they had lots of microservices (we sold access to our APIs to businesses). They were all on two “pet” EC2 instances. Next step was to move the .Net Core APIs to Docker/Fargate and further restrict the attached roles to those. Finally, we had to audit all of our AWS dependencies and add encryption where necessary and then sign a BAA with AWS and bring in auditors. By the time I left a month ago, we could pass the needed certifications and expand our offerings. It took a lot of upskilling, hiring an internal ops person (I’m a developer who knows AWS) instead of depending heavily on the MSP. I left for greener pastures - I’m a consultant with AWS.
Aeolun 6y ago
We have machines to tell people their policy (in dev) is insecure and will be deleted in 30 minutes. Not really easy to argue with the machine.
- ozim 6y ago
  Nice that just like having bots in discord. When administrators hide behind bots with decisions so users cannot retaliate. It also is nice combined with shadow bans, user thinks his posts are going through but no one replies. Unfortunately for development it is better to get tight integration of dev and ops so you could solve it by discussion and cooperation. Not sure if you can build such teams that often but that would be great.

thayne 6y ago

In my company, developers write policies, but they have to be approved by someone with ops expertise (usually me). And the policies are often either too broad, or not sufficiently tested, and missing permissions for things it needed. Sometimes the same policy has both problems. I don't blame the developers. You can hardly expect every developer to become an expert on AWS's IAM system. Especially given how inconsistent it can be.

partyboat1586 6y ago
>or not sufficiently tested How do you test a policy?
- thayne 6y ago
  You set up a policy in a test environment and run the code there. Of course, generally, the policy can't be identical across environments, so you can run into errors.