4 ms·
Here's an example of an effective setup: * Have a separate AWS account for each developer that allows for experimentation without risking shared environments.
by bilalq 4y ago
Here's an example of an effective setup:
* Have a separate AWS account for each developer that allows for experimentation without risking shared environments. Engineers can have admin access on their own account to enable rapid prototyping and experimentation.
* Have separate AWS accounts for separate services. Ideally a separate account per service/stage pairing. If you're even more mature in your cloud operational journey, you may go for one account per service/stage/region or per cell in a cellular architecture if you're really advanced. If you're just a small startup, maybe just go for a separate account for beta and prod stages.
* Because running services have their own account, the blast radius of an engineer modifying permissions is quite limited already. However, changes to infra (including IAM changes) should all be done through a CI/CD pipeline. Engineers can only change infra by submitting a PR to update the IaC definition and passing through your peer review and automated checks. Tools like AWS CDK with self-mutating pipelines where the pipeline itself is modeled via IaC are great for this.
* Use a higher level of abstraction to manage IAM permissions like AWS CDK. Manually figuring out the permissions needed is a nightmare and an exercise in frustration.
* Try to keep AWS credentials ephemeral with whatever third-party services you're working with.
* If you're using AWS CodePipelines, run all your pipelines in a separate DevOps account where all the pipelines live. Set a role that engineers can use on this account that lets them debug pipelines without mutating. If you're using Github actions or something else, you obviously need to guard permissioning of that other service and try to use decent practices for getting AWS credentials. In the case of Github, for example, use OIDC to assume ephemeral credentials to an IAM role rather than saving secrets of an access/secret key pair.
- techdragon 4y agoMajor caveat. You’re going to want to have at least usage monitoring tools and ideally usage limiting tools setup before you give developers their own individual AWS accounts… at the sort of scale where sharing a dev account stops being viable, accidentally creating and leaving expensive resources around to rack up a bill becomes far too easy
- TheNewsIsHere 4y agoThis is the way. I’ve seen this happen countless times. It’s happened to me too. It’s happened to colleagues. The worst case I’m aware of from first-hand knowledge was a large cluster of resources that got deployed for a product demo by a sales engineer and forgotten about. Turned into a nice ~$100,000 surprise in the quarterly budget. Netflix built a tool for managing IAM permission requests as an auditable workflow, called ConsoleMe: https://github.com/Netflix/consoleme https://github.com/Netflix/consoleme
- deleted 4y ago[deleted]
- bilalq 4y agoYup. The scope of the discussion was around permissioning/security, so I didn't get into billing, but you're absolutely right. You should have CloudTrail, billing alarms, and dashboards all setup. It may also be a good idea to setup automatic spring cleaning that nukes resources every two weeks or so unless they have special tags to mark retention.