6 ms·
Secure Access to 100 AWS Accounts
- appdrag 9y agoHey segment, are you aware that you can have several env (dev, stag, prod, ...) on the same AWS account? :p You can secure each environement with different credentials (IAM) so no need to create several AWS accounts!
- Thaxll 9y agoIt's a bad idea to use multiple env in the same AWS account.
- rpedela 9y agoCan you expand on that?
- throwbacktictac 9y agoTheir reason for multiple accounts seems to be due to API rate limiting imposed by aws? At least thats my understanding.
- zimbatm 9y agoRoute53 API rate limits are per account and very low. It sucks not being able to fix the production A record because somebody did a lot of changes on the staging zone.
- erik_seaberg 9y agoI was also alarmed to see that dynamodb:DescribeStream is limited to 10 QPS per account, making low-latency stream consumers into noisy neighbors of each other. Then there's all the "open a support ticket" per-account limits on table provisioning and running EC2 instances…
- jeffnappi 9y agoIt is recommended to use multiple accounts for isolation. AWS has recently launched Organizations to make creation and management of multiple accounts simple. See https://aws.amazon.com/organizations https://aws.amazon.com/organizations Tools like aws-vault and now aws-okta make securing credentials a piece of cake.
- rwitoff 9y agoIn practice, cramming all this into the same account doesn't work. Segment is following best practice here. For example, IAM doesn't provide the granularity in resources and conditions that you'd want to effectively isolate the blast radius of developer keys. ec2:TerminateInstances didn't (doesn't?) support VPC level conditions, so being able to terminate one instance meant you could terminate all instances. Similarly, you might want your engineering team to iam:PutUserPolicy in development, but have a much more restricted group in production which isn't possible with IAM today. I've taken this pretty far in the past to attempt segmenting within one account, but always run into limits: https://github.com/witoff/self-service-iam https://github.com/witoff/self-service-iam
- dastbe 9y agoThe other bit would be blast radius. What if someone does get access to your single account? How confident are you that your policies were airtight? By using many accounts, you create clear isolation boundaries that require opt-in sharing.
- user5994461 9y ago>>> By using many accounts, you create clear isolation boundaries that require opt-in sharing. In theory yes. In practice, you will achieve the opposite of that. Developers and ops will have to juggle between 10 keys and accounts to get anything. The keys will end up saved and written all over the systems. It will be impossible to have audit between all the accounts and access.
- ejcx 9y agoOp here. I don't think you read the blog post! Our entire engineering org has a grand total of 0 AWS keys! Per-account isolation is great for security and especially reliability, if you run in to constant ratelimit issues like we do.
- deleted 9y ago[deleted]
- 9y ago
- paulddraper 9y agoThis just in: directories are pointless because you can put all files in the same one! --- There are advantages in multiple accounts, for organization and security. Random example: adding tags to ec2 instances is an account-wide permission, for all ec2 instances. Multiple various things by 100 and accounts make sense.
- ejcx 9y agoOP here. You _can_ do this but IAM Policy wise it is a mess and not very maintainable. You end up needing to constantly grant and revoke access to individual resources if you went this route. Instead, it's nice to give engineers access to everything that's in their account, and not worry about IAM policies.
- erik_seaberg 9y agoReally the problem is that IAM is not a capability system. If you can write a policy at all, there are no limits on what you can put in it, so we can only let trusted admins touch policies. An engineer should be able to grant some of their own privileges to the things they run.
- mooreds 9y agoLots of good information and considerations here: https://aws.amazon.com/answers/account-management/aws-multi-account-security-strategy/ https://aws.amazon.com/answers/account-management/aws-multi-... As with all of AWS (and software, and life) there is no one size fits all answer.
- CSDude 9y agoBut you have to prefix all of your resources then, i.e if you are using SQS, DynamoDB, Lambda's and it becomes a huge mess.
- ejcx 9y agoThanks for posting this here. I'm the author of this blog post. Feel free to ask me any questions you might have.
- scrollaway 9y agoTangential: Why Okta over some other provider? GSuite can be used as an SAML provider, can't it?
- ejcx 9y agoI honestly don't know of the feature parity between the two, but we are definitely Okta power-users having added close to 500 apps in the last year. I don't know if we could drive all of the same processes we have built (especially on the HR side) using GSuite.
- marcc 9y agoGSuite can be an IdP for SAML but it’s sometimes limited. AFAIK you can’t use it to connect to AWS using SAML because AWS makes some assumptions about attributes in the identity. I may be wrong here, but I tried to do this a few months ago (with Duo and Yubikeys added in) and was not able to get it working. I ended up using Okta and it was simple.
- debaserab2 9y agoYou can do it. It feels a little hacky - you have to execute a few actions via gsuites API instead of via a UI. https://aws.amazon.com/blogs/security/how-to-set-up-federated-single-sign-on-to-aws-using-google-apps/ https://aws.amazon.com/blogs/security/how-to-set-up-federate...
- wgjordan 9y agoAbout a year ago, I migrated my team's AWS access from manually-provisioned AWS keys to GSuite authentication, using a combination of SAML and OpenID Connect (OIDC): - Detailed instructions for Google-federated login to the AWS Management Console through SAML are available [1], and worked more or less as described. - CLI/API access is a bit trickier and less thoroughly documented, but is possible using the 'Web Identity Federation' feature [2]. Basically, you generate/refresh temporary AWS credentials by passing a Google OIDC token to the AssumeRoleWithWebIdentity API. The tricky part is keeping both your Google OIDC and AWS STS tokens conveniently refreshed. I wrote some open-source glue code for this that hooks into the AWS Ruby SDK and CLI [3]. It's still a little rough around the edges and not yet extracted into a standalone project, but it's been working well enough for my team over the last year. [1] https://aws.amazon.com/blogs/security/how-to-set-up-federated-single-sign-on-to-aws-using-google-apps/ https://aws.amazon.com/blogs/security/how-to-set-up-federate... [2] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_oidc.html https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_pr... [3] https://github.com/code-dot-org/code-dot-org/blob/staging/lib/cdo/aws/google_credentials.rb https://github.com/code-dot-org/code-dot-org/blob/staging/li...
- bsaraogi 9y agoHow would this compare to an LDAP, an LDAP with org wide access control can pretty much enable all of this(keeping in mind GDPR & other security concerns) from just 1 AWS account ?
- ah- 9y agoHow are you dealing with software that expect to get aws tokens? Especially with sts AssumeRoles only lasting up to one hour? For example, if you want to run the Athena JDBC driver locally or read something from S3 in a longish (>1h) running Python script.
- stiveridibla 9y agoI can't answer for Segment, but our AWS account strategy is very similar, so we've found that there is still a need for a centralized KMS solution (system-to-system auth). Our teams are working on a project to wrap more tooling around Vault and make it a centralized KMS. T-Mobile has done some work on this... https://opensource.t-mobile.com/blog/posts/introducing-tvault/ https://opensource.t-mobile.com/blog/posts/introducing-tvaul...
- idunno246 9y agoIf you use aws-vault --server or something like it that mocks a metadata server, all the sdks handle rotation properly. Just need to fork to connect it to whatever generates tokens
- zimbatm 9y agoIt's good to see an article talking about this. A lot of organizations could benefit from using more than one account to enforce security (IAM is hard!) and separation of concerns. It doesn't really explain it but to do this, the root account has to be enrolled for AWS Organization[1]. This is what is being used to handle all the accounts and consolidate the billing. It also allows to create rules span all the accounts. Recently terraform gained support for the Organization API[2] so it's possible to control the account list in a declarative manner. The biggest issue is that now that there are a lot of accounts, the developers need a way to switch between them. Using the IAM assume-role mechanism is a good way to avoid needing a lot of AWS keys per developer. I don't know if I agree with using Okta as it adds another party that now has access to AWS. I don't see the difference between having a AWS secret or and Okta secret in the keychain security-wise. Okta might provide audit logging facilities but so does AWS. In either case you will need to generate a `~/.aws/config` per developer. There is also a Chrome plugin[3] that can read this file format and populate the AWS Console switch role. I don't know if the extension publisher is reputable yet as it gives a lot of access to the extension. [1]: https://aws.amazon.com/organizations/ https://aws.amazon.com/organizations/ [2]: https://github.com/terraform-providers/terraform-provider-aws/pull/903 https://github.com/terraform-providers/terraform-provider-aw... [3]: https://chrome.google.com/webstore/detail/aws-extend-switch-roles/jpmkfafbacpgapdghgdpembnojdlgkdl https://chrome.google.com/webstore/detail/aws-extend-switch-...
- toomuchtodo 9y agoIt should be noted that Orgs should be setup when new AWS accounts are created, as it can be tricky to perform once your have systems in production. I would even go so far as to say if you want to commit to Orgs, create a new AWS account hierarchy and move your applications and infrastructure into it. EDIT: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scp.html https://docs.aws.amazon.com/organizations/latest/userguide/o... "Warning: We strongly recommend that you do not attach SCPs to the root of your organization without thoroughly testing the impact that the policy has on accounts. Instead, create an OU that you can move your accounts into one at a time, or at least in small numbers, to ensure that you don't inadvertently lock users out of key services."
- lis 9y agoOut of curiosity, how do people handle communication between services that are in different accounts? VPC peering to connect all accounts? Public Internet and OAuth? Another method?
- yeukhon 9y agoYou need to define “between services”. You mentioned oauth and VPC peering. One is authentication, the other is networking accessibility. There is no one solution. Generally you have: VPC peering between accounts, Network Access Control List (NACL) for VPC port control, security groups between instances (and some AWS services which uses SG to limit port access), IAM roles to authenticate and authorized certain AWS services to do things (e.gz Lambda to read S3 bucket) - but IAM and policies (bucket policies, SQS policies) govern authentication and authorization. Finally there is also organization service which allows you to control what AWS services are allowrd in a group of AWS accounts. Sorry, on mobile so I can’t make a prettier list. I am generally disappointed at the complexity of authentication and authorization mechanisms exists for AWS services to be really honest.
- lis 9y agoOh, sorry, absolutely. I'm actually talking about services that talk to each other using HTTP. One option I've though of is using the public internet to connect them and use OAuth to make sure that they are actually allowed to communicate, solving the security issues on the application level. Another option would be to allow only access from the internal network, but then you have to connect them somehow.
- sokoloff 9y agoI'd recommend against the latter in any medium or large organization (and small as well, but sometimes one needs to make trade offs). That's putting too much trust in the network. You don't want one phished internal user to expose the entire set of services that developers all think of as "on a trusted network".
- yeukhon 9y ago
- stiveridibla 9y agoExcellent article thanks for sharing. It's awesome to see someone documenting their experiences with multi-account strategy. You said you manage account with Terraform, are there any other resource types that you don't give the dev teams access to such as networking? Also, how are you deploying Terraform at scale across all of your accounts?
- jbergknoff 9y agoWhat are best practices for sharing build artifacts (e.g. images in ECR, files in S3) among AWS accounts? Cross-account IAM policies, explicit promotion process to move artifacts between accounts, or something else?
- ejcx 9y agoWe use ECR and have all of our images in our hub account, and make them readable from the prod/stage/dev accounts using a cross account policy. For S3, it's more complicated because of object permissions and all the insanity that comes with cross-account writing etc. Edit for more info: No explicit promotion process for containers. Engineers build images in our CI pipeline when their pull request is approved and merged to master, which is pushed to the hub account.
- posnet 9y agoAtlassian open sourced a tool to help with CLI access when using many AWS accounts through federated roles. https://developer.atlassian.com/blog/2017/12/introducing-cloudtoken/ https://developer.atlassian.com/blog/2017/12/introducing-clo...
- 7ewis 9y agoWe created something similar in house. Users authenticate to an internal website with ADFS (including MFA) and are then presented with a list of roles where they can either click through to the website assuming a role in that account for an hour, or click an option to access temporary credentials. The AWS roles are deployed from our CI/CD pipeline to all of our AWS accounts, so we don't have to have user accounts anywhere and can still deploy features from our Pipeline without logging in. We are also in the process of setting up automated account provisioning from our HR system, Workday. Based on the users team and job title, they'll be added into an Active Directory group which would then give them access to the resources required for their role. Once complete, this will save the support team a lot of time!
- bringtheaction 9y agoThat’s neat would your company consider open sourcing it if I understand correctly that you made it for your own use not to sell it anyway.
- 7ewis 9y agoIt has actually been considered. I'll bring it up again. Personally I don't see why not!
- robax 9y agoThis is really similar to what Amazon uses in-house to manage aws roles. Not sure if there’s a commercial or open-source alternative but this could be a fruitful project for someone!
- cyberferret 9y agoI am interested in the thought processes behind setting up separate AWS accounts "for GDPR compliance" as opposed to having one AWS account and running multi regions (via VPC etc.) under that account? Is it because teams in different locations (US, EU etc.) can run semi-independently? How do you manage shared resources (e.g. S3 or RDS SQL servers) that need to be accessed from multiple regions, yet still maintain GDPR compliance?
- ejcx 9y agoAWS has a lot of resources that are not per region. The GDPR line is about improving our least privilege access story. There's a lot more to GDPR than that, though
- foodbaby 9y agoTemporary security credentials from AssumeRole are valid for up to 3600 secs (1 hour). Given that, how do folks handle long running jobs/sessions?