5 ms·

Those problems seem to come from the separation of the IAM admin from the developer. I'm coding a server now. My IAM roles are defined in a template, and I just

Those problems seem to come from the separation of the IAM admin from the developer. I'm coding a server now. My IAM roles are defined in a template, and I just add new permissions to the template as I need them. My code has the bare minimum permissions that it needs, and it doesn't seem at all onerous for the benefit it provides. So I think the problem is less "IAM is hard", and more "coordination is hard". The one big exception I've run into is that to launch a CloudFormation template, the role practically needs admin access. I'm considering offloading the launch to a minimal Lambda function with the requisite (very broad) permissions. Does anyone have a better approach?

lovehashbrowns 6y ago
I agree that "coordination is hard" is the root issue here. For things I develop, it's easier for me to specifically say what permissions the IAM roles need, and I can get them down to least privilege. Then sometimes I need to onboard an application written in a language I don't know, and it's a beast of an application. I ask the developer what permissions it needs, and they say "I don't know" but give me a full admin role they know works. I'd love to go through the whole process of determining what permissions it needs, except that I have deadlines, and they have deadlines, and our deadlines are visible to management, and I have additional projects that I need to finish, and my team is already too small. It's like the universe is just telling me to give the application full admin.
- coredog64 6y ago
  Time will tell if this works out for me, but lately I’ve been explaining the risk to the developers’ manager, that I wouldn’t do this, but if they assume the risk that I’ll do what they ask. I figure eventually we’ll get compromised and/or some misbehaving app will take down production and then people will pay attention.
dasil003 6y ago
This works if all developers understand IAM and don’t just throw a wildcard in the first time they don’t understand something.
- bobbiechen 6y ago
  Agreed, there is a large learning curve and it took me a long time to wrap my head around it. It requires a lot of knowledge and discipline, which sooner or later will create holes. For example, if you need to pass a role to a service, you'll need PassRole; if you grant it on " * ", then oops, you might have just created the opportunity for privilege escalation [1]. There's also probably issues specific to your company: allowing access to read resource foo in general is not an issue, except your specific company stores sensitive data there. If every developer is expected to be a security expert, the security risks increase, and the productivity overhead may even be worse than having a dedicated security/IAM team that gatekeeps permissions. [1] https://rhinosecuritylabs.com/aws/aws-privilege-escalation-methods-mitigation/ https://rhinosecuritylabs.com/aws/aws-privilege-escalation-m...
- dopylitty 6y ago
  I agree with this but I think AWS could have designed the interface for IAM policies better. There are a lot of actions where the resource has to be “”. There are also many situations where the Principal is because you’re using Conditions to restrict the access (eg by the org id). The resources are also “typed” despite the UI being json. This leads to confusion when a policy doesn’t work because the string in the resource is of the wrong type (eg S3 bucket vs S3 object). IAM happily lets you create the policy and there might be a small warning in the console that some of your policies somewhere have invalid resources for their actions but if you’re using CloudFormation you’ll never see those warnings. It begs for an automated linter that understands the type system and can fail your merge request or highlight the code in your IDE if the policy is invalid. AFAIK CFN-lint doesn’t do this but it certainly should.
jimbobimbo 6y ago
My team approaches this problem by using separate identities with different access policies for doing different things. The identity can be an alternative user account (disconnected from the primary corp domain) or a service principal that does only limited set of things. For example, an alternative user account with restricted time window for access is used to even get to performing infrastructure management tasks. Then each cluster has unique service principal attached to it for pulling containers and retrieving infra-related secrets, but cannot access application-related secrets. Applications that run on clusters use completely different managed identity which doesn't have access to infra-related secrets, but has access to application-related secrets. On top of that, wherever possible, we restrict access to secrets only to GET operations, so you need to know the name of the secret beforehand in order to access it. The latter is not always possible, but if it is possible, we use it. We use a bunch of scripts that go on and create all necessary identities and set up security policies, which helps a lot with ensuring the process is fully repeatable and risks of user mistakes are mitigated.
- jimbobimbo 6y ago
  And to add to this - it wasn't the case from the very beginning. It took us shipping a set of services with ISO and SOC 2 requirements to arrive to security model we're employing now. It also helps a lot having corp-wide robust security and compliance teams that drive security mindset across the company. They create a lot of pain for dev teams, but this ultimately results in much better security stance across the board.
Trisell 6y ago
At my company I am working to get all of our IAM policies ironed out in AWS-CDK. Through this any developer can pull down the git repo containing the IAM roles. They can makes changes and submit MRs but the only approvers of those MRs are part of our Access Management team. This all of the IAM roles can be crafted by the devs but must be explained and understood by the AM team through a code review process. After that automation takes over and then the policy can be referenced by other CDK/CF templates.
freeone3000 6y ago
We've managed to mitigate this by doing cloudformation launches manually with the developer's own credentials, and have each resource take on the IAM role it needs. (Since you need sts:AssumeRole for CFN creation anyway, it's not really useful to limit the cloudformation role, it's going to be essentially admin in any event.)

lovehashbrowns 6y ago

I agree that "coordination is hard" is the root issue here. For things I develop, it's easier for me to specifically say what permissions the IAM roles need, and I can get them down to least privilege. Then sometimes I need to onboard an application written in a language I don't know, and it's a beast of an application. I ask the developer what permissions it needs, and they say "I don't know" but give me a full admin role they know works. I'd love to go through the whole process of determining what permissions it needs, except that I have deadlines, and they have deadlines, and our deadlines are visible to management, and I have additional projects that I need to finish, and my team is already too small. It's like the universe is just telling me to give the application full admin.

coredog64 6y ago
Time will tell if this works out for me, but lately I’ve been explaining the risk to the developers’ manager, that I wouldn’t do this, but if they assume the risk that I’ll do what they ask. I figure eventually we’ll get compromised and/or some misbehaving app will take down production and then people will pay attention.

dasil003 6y ago

This works if all developers understand IAM and don’t just throw a wildcard in the first time they don’t understand something.

bobbiechen 6y ago
Agreed, there is a large learning curve and it took me a long time to wrap my head around it. It requires a lot of knowledge and discipline, which sooner or later will create holes. For example, if you need to pass a role to a service, you'll need PassRole; if you grant it on " * ", then oops, you might have just created the opportunity for privilege escalation [1]. There's also probably issues specific to your company: allowing access to read resource foo in general is not an issue, except your specific company stores sensitive data there. If every developer is expected to be a security expert, the security risks increase, and the productivity overhead may even be worse than having a dedicated security/IAM team that gatekeeps permissions. [1] https://rhinosecuritylabs.com/aws/aws-privilege-escalation-methods-mitigation/ https://rhinosecuritylabs.com/aws/aws-privilege-escalation-m...
dopylitty 6y ago
I agree with this but I think AWS could have designed the interface for IAM policies better. There are a lot of actions where the resource has to be “”. There are also many situations where the Principal is because you’re using Conditions to restrict the access (eg by the org id). The resources are also “typed” despite the UI being json. This leads to confusion when a policy doesn’t work because the string in the resource is of the wrong type (eg S3 bucket vs S3 object). IAM happily lets you create the policy and there might be a small warning in the console that some of your policies somewhere have invalid resources for their actions but if you’re using CloudFormation you’ll never see those warnings. It begs for an automated linter that understands the type system and can fail your merge request or highlight the code in your IDE if the policy is invalid. AFAIK CFN-lint doesn’t do this but it certainly should.

jimbobimbo 6y ago

My team approaches this problem by using separate identities with different access policies for doing different things. The identity can be an alternative user account (disconnected from the primary corp domain) or a service principal that does only limited set of things. For example, an alternative user account with restricted time window for access is used to even get to performing infrastructure management tasks. Then each cluster has unique service principal attached to it for pulling containers and retrieving infra-related secrets, but cannot access application-related secrets. Applications that run on clusters use completely different managed identity which doesn't have access to infra-related secrets, but has access to application-related secrets. On top of that, wherever possible, we restrict access to secrets only to GET operations, so you need to know the name of the secret beforehand in order to access it. The latter is not always possible, but if it is possible, we use it. We use a bunch of scripts that go on and create all necessary identities and set up security policies, which helps a lot with ensuring the process is fully repeatable and risks of user mistakes are mitigated.

jimbobimbo 6y ago
And to add to this - it wasn't the case from the very beginning. It took us shipping a set of services with ISO and SOC 2 requirements to arrive to security model we're employing now. It also helps a lot having corp-wide robust security and compliance teams that drive security mindset across the company. They create a lot of pain for dev teams, but this ultimately results in much better security stance across the board.

Trisell 6y ago

At my company I am working to get all of our IAM policies ironed out in AWS-CDK. Through this any developer can pull down the git repo containing the IAM roles. They can makes changes and submit MRs but the only approvers of those MRs are part of our Access Management team. This all of the IAM roles can be crafted by the devs but must be explained and understood by the AM team through a code review process. After that automation takes over and then the policy can be referenced by other CDK/CF templates.

freeone3000 6y ago

We've managed to mitigate this by doing cloudformation launches manually with the developer's own credentials, and have each resource take on the IAM role it needs. (Since you need sts:AssumeRole for CFN creation anyway, it's not really useful to limit the cloudformation role, it's going to be essentially admin in any event.)