15 ms·

Have lots of AWS accounts

228 points by rcrowley 4y ago

hgh 4y ago
AWS gets many things more right, but I think GCP wins on how they handle Projects and Managed Instance Groups for auto-scaling.
- remram 4y ago
  Easily separating billing accounts is definitely a must.
- robohoe 4y ago
  GCP organization structure is such a breath of fresh air after dealing with AWS. AWS Orgs and all the complexity with VPC and DNS management at a scale of hundreds of accounts is just a complete pain in the arse. GCP makes it far easier with Shared VPC and org/folder/project structure.
dijit 4y ago
One of the things I love most about google cloud is that "projects" are easy to create and easy to link to other projects. Roles and service accounts can even reference across projects, though I'm not sure I'd recommend doing that. No more faffing about with special accounts, passwords and difficult to configure shared VPCs, it all becomes so easy. Even managing the different accounts is difficult without browser extensions such as this one: https://chrome.google.com/webstore/detail/aws-extend-switch-roles/jpmkfafbacpgapdghgdpembnojdlgkdl?hl=en https://chrome.google.com/webstore/detail/aws-extend-switch-...
- rcrowley 4y ago
  Author of the article here: I agree. GCP projects are a better abstraction. I still think AWS is, on balance, a better cloud.
  - dijit 4y ago
    I'd love to hear more about why you think that's the case! Maybe the next blog post?
    - simonw 4y ago
      The killer feature of AWS is that they've broken almost nothing since first launching over 15 years ago. Their commitment to keeping old stuff working is truly amazing. No other cloud hosting service comes close, because no other cloud service has had 15 years to prove themselves in the same way!
      - jimt1234 4y ago
        SimpleDB is still running!
- 0xbadcafebee 4y ago
  I really dislike the Google Cloud way of doing things, for example Projects, Folders and Orgs. There's a dozen different weird paradigms that you have to consider to securely and reliably manage a large number of different projects, accounts, tenants, business units, etc in Google Cloud; if you don't do things "The Google Cloud Way" you are screwed. AWS is much simpler and more straightforward. You don't have to think about anything to segregate infrastructure, networks, applications, users, data, etc. Just put it in a different account. If someone needs access, you need to explicitly add extra connections/grant that access. You can't just accidentally create one user that implicitly has access to hundreds of accounts. Honestly the entire security model of Google Cloud is frightening. It's like they wanted to make it easy to expose everything.
  - lbhdc 4y ago
    You can still work like that. You just don't have to bother changing your project id, and switch accounts to jump to other projects. I do this for some things I work on where there are account level controls and the accounts cant be shared across projects.
  - deleted 4y ago
    [deleted]
  - DangitBobby 4y ago
    AWS is decidely _not_ simpler here. You've got Stockholm syndrome. What's complicated about making two projects and picking between them? What's complicated about grouping projects under a single organization? What's complicated about grouping projects under a folder? What's complicated about adding someone to a project through IAM? The only way you can fuck that last part up is by granting overly broad permissions (which they warn you when you've done right there in the console). You can safely ignore the existence of Organizations and folders completely if you are so inclined. At least there's a sensible way to set default policies for your organization baked into the platform. Even so, literally 3 hours, tops, and you know everything about these abstractions you could ever care about. I cannot emphasize enough how much easier GCP is to learn and use than AWS.
    - 4y ago
- atonse 4y ago
  AWS SSO has made it incredibly easy for us to secure and manage access to (and switch between) all our AWS accounts in the org. I'm a huge fan and recommend it.
  - tpmx 4y ago
    They recently rebranded it to the super-catchy "AWS IAM Identity Center (successor to AWS SSO)". Gotta love Seattle-based tech marketing people, presumably trained at Microsoft. :) I guess it will soon be the default user management system, and 'proper' IAM will be the low-level one. I see this as a reaction towards GCP's IMHO superior UX/system design in this aspect. I don't think they can entirely catch up because of early and bad architectural decisions regarding projects/accounts. Also a fan, so far.
  - macintux 4y ago
    I was originally skeptical when we implemented it because for safety reasons I wanted it to be painfully obvious when I was switching contexts. It's definitely grown on me, but I frequently emphasize to my teammates that I'd much rather they specify their AWS_PROFILE for each command they run, instead of exporting it into their environment. Especially using Windows and setx seems like an open invitation for disaster.
- GauntletWizard 4y ago
  One reason to have your roles and service accounts reference cross-project is to give your CI builder access to your artifacts project; Have your prod and dev environments use containers from a third account that serves only as an archive of your build artifacts.
- psanford 4y ago
  I don't know. I've found it to be pretty difficult to answer the question "which of these 1000 gcp projects are running a production workload and which are random one offs created by a dev messing around or by a google sheet script?"
  - dijit 4y ago
    I solve that by using folders. But when you have many of anything it will become hard to reason with. It’s not worse than your filesystem, in theory, but speaking for myself: my filesystem is a mess, so…
  - DangitBobby 4y ago
    So the problem is the abstraction is so easy to use to get a project going that people just litter them everywhere... how is that a problem with the platform? At least they are actually in one searchable location. How do you find your production workloads if you have 1000 AWS accounts?
    - psanford 4y ago
      In AWS we divide things up into child accounts that roughly match our org tree. Each team or service gets dev, staging, and prod child accounts. Teams have access to their child accounts and its fairly obvious what they all do, or at least, who to talk to to find out. I think my main complaint with GCP is that it is often tied directly to an org's gsuite account. And doing things in gsuite (used to?) automatically create GCP projects behind the scenes. So you could easily get thousands of projects that the users themselves didn't even know that they had created. If I were starting from scratch with GCP I'd use a third party IdP and not let users access it via their gsuite accounts. I suspect that would avoid most of the issues I've run into.
      - DangitBobby 4y ago
        We haven't seen that gsuite project problem, but we also don't do really color outside the lines with gsuite. Email, chat, docs, AD, etc.
Sevii 4y ago
"Imagine you’re trying to create the kind of isolation necessary to deliver the security, reliability, and compliance that business customers demand in one AWS account." Got to be my favorite way to describe two nines ever!
- rcrowley 4y ago
  actual lol
Havoc 4y ago
Isn’t that against terms since it activates a free tier credit?
- rcrowley 4y ago
  I'm not honestly sure how AWS Organizations interacts with the AWS free tier. Rest assured, though, having lots of AWS accounts (and using AWS Organizations, their service designed to _help_ you use lots of AWS accounts) is _not_ against the terms of service.
  - brodouevencode 4y ago
    Each account within the organization gets the limits of the free tier, just as a single account would. I think the reasoning is that if you're going to go through the hassle of setting up orgs then you're probably an enterprise user slated to take the long haul anyway.
    - kondro 4y ago
      Not true, the free tier is applied at the Organization level (i.e. the billing account), not to each individual account.
      - brodouevencode 4y ago
        Hmmm according to https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-free-tier.html https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2... it reads that it should be, but the wording is opaque. Hopefully someone from AWS can answer.
- traspler 4y ago
  Multi Account is a normal pattern for AWS. They even have tools to handle that better like AWS Control Tower. Someone from AWS we talked to even mentioned it as a perk (to get the free tier on every account).
pid-1 4y ago
> My favorite way to create a network between all my services hosted in different AWS accounts is to share a VPC from a network account into all my service accounts and use security groups to authorize service-to-service communication. There’s no per-byte tax, zonal architectures are easy to reason about, and security groups work just like you expect. That's gold advice. I wish AWS RAM supported more services (like AWS EKS). A small complain: working with AWS SSO is a bit tedious. My current solution is to share my ~/aws/config with everyone so we all have the same profile names and scripts can work for everyone.
- rcrowley 4y ago
  I have thought a great deal about whether I also want EKS clusters to officially support nodes in multiple AWS accounts. On the one hand, having the option to create that additional low-level isolation would be lovely, even and maybe even especially if I didn't always take it. On the other hand, isolating two things from each other but then tying them to the same Kubernetes cluster upgrade schedule feels wrong. In the end, I decided that if I care about isolating two things enough to put them in separate AWS accounts, I'm willing to spend the $75 per month that it takes to have separate EKS clusters, too. (This opinion perhaps obviously doesn't fit will with hobby/side project budgets.)
- rcrowley 4y ago
  Synchronizing ~/aws/config gets harder and harder as your team grows because there are both more people who need to receive changes and more people making changes. I think the human-readable names for AWS accounts need to be part of the account, not part of the laptop. Substrate [1] does this so that you can type commands like `substrate assume-role -domain example -environment production -quality beta` [2] to get where you're going. [1] <https://src-bin.com/substrate/ https://src-bin.com/substrate/> [2] <https://src-bin.com/substrate/manual/moving-between-aws-accounts/ https://src-bin.com/substrate/manual/moving-between-aws-acco...>
  - nijave 4y ago
    One place I worked just committed it to the git repo (obviously just SSO and account numbers, no passwords or tokens)
tazjin 4y ago
DerekBickerton 4y ago
I thought this was an article about having separate compartmented Amazon accounts. So one for shopping, another for Amazon Associates, another for AWS, another for Prime Video and Alexa, etc Does anyone here do this?
mattpallissard 4y ago
> AWS accounts are the most complete form of isolation on offer. Because it's a pretty brutal namespace. While I usually wind up with separate accounts for multiple reasons, I strongly prefer to get rbac/iam properly implemented in a single account wherever possible.
- forgomonika 4y ago
  > I strongly prefer to get rbac/iam properly implemented in a single account wherever possible. This can really help manage complexity but it the more this account grows the riskier it becomes if a bad actor breaks into the account (ranging from fraudsters/hackers to disgruntled employees).
  - mattpallissard 4y ago
    Using accounts as the security boundary is easier to reason about up front but it's rather ham fisted. If you're centrally automating large swaths of infrastructure across many accounts you'll wind up paying for that in the long run. When a shop smaller, or dealing with multi-tenancy situations, multiple accounts is an easy trade off to make. The account boundary has its place, but it's not for everything. Some of these scaling pains with hard boundaries have gotten better with awssso and iam features over the years but you still run into them on occasion. As far as whatever you mean by "breaking into an account". I said originally, "rbac/iam properly implemented". If you fuck that up, it doesn't matter whether or not you have one or multiple accounts. Having a single account does mean that you have to give anyone the keys to the kingdom or that you don't separate your concerns.
malcolp 4y ago
I love this approach, although I'm yet to work anywhere that does this. I guess the million (thousand?) dollar questios now become where do you draw the boundary across accounts? Presumably there are many bad ways to slice accounts up. And what happens when accounts do need to communicate? I can imagine three major scenarios for cross account permissions: * Cross account iam policies (painful in my experience) * Adding trust policies to enable cross account assuming roles (better but limited) * Limiting cross account policies to specific easy to configure services. E.g. S3 (best option, I've seen but maybe too limited) Would love to hear the author's view.
- iLoveOncall 4y ago
  > Would love to hear the author's view. Instead you can hear AWS's view, which is to have one account per stage per region per service. I can't find a source but I work for Amazon and this is what was recommended to us by ProServe (the contracting branch of AWS) when we talked with them. I think it's idiotic though (because regions are 100% separated within an account, and it would easily triple the number of accounts to manage), and so did my team, so we stuck with one account per stage per service. That said, cross account permissions is really not an issue, it's very easy and straightforward to setup. You also should not need it in 90% of the cases if your application is properly split with the right ownership for each microservice. For my current team we manage probably more than a thousand AWS accounts, and permissions are never an issue. Neither is anything else actually. We aggregate metrics in a single account for the stuff that needs to be aggregated, we have small CLI scripts that automate tedious steps like requesting limit increases, etc.
  - brodouevencode 4y ago
    > Instead you can hear AWS's view, which is to have one account per stage per region per service. This is exactly correct. > I think it's idiotic though (because regions are 100% separated within an account But still bound to the same service limits, no?
    - iLoveOncall 4y ago
coding123 4y ago
This is one of those cases where the wrong thing gets optimized because of something stupid. ...Which is probably fine because that's pretty much how it always goes.
jackconsidine 4y ago
I have indeed found that it's difficult to silo access to specific resources with AWS IAM. I use these custom policies a lot, which give write access to a specific S3 Bucket [0], and give sending capabilities for a specific SES Identity [1] respectively: [0]https://koptional.notion.site/IAM-Policy-for-select-S3-Access-on-a-single-bucket-1d368c7eb45446b1bd653d1a0425042c https://koptional.notion.site/IAM-Policy-for-select-S3-Acces... [1] https://koptional.notion.site/IAM-Policy-for-email-sending-on-behalf-of-single-SES-identity-414f02acd0e4496eb994e171a74f3fdf https://koptional.notion.site/IAM-Policy-for-email-sending-o...
- simonw 4y ago
  I ended up writing a whole custom tool for generating credentials and policies for specific S3 buckets: - https://s3-credentials.readthedocs.io/ https://s3-credentials.readthedocs.io/ - https://simonwillison.net/2021/Nov/3/s3-credentials/ https://simonwillison.net/2021/Nov/3/s3-credentials/ - https://simonwillison.net/2022/Jan/18/weeknotes/ https://simonwillison.net/2022/Jan/18/weeknotes/
  - jackconsidine 4y ago
    Whoa this is awesome! Wish I knew about this. It's really polished
moralestapia 4y ago
Every serious project I engage on has its own: * domain (obviously) * emails also for every provider I have a different email like (google@domain, twilio@domain, etc...) * credit cards (my bank makes it super easy to just create new ones) * phone no. (I just buy a burner phone) It's a bit of a PITA but the benefits outweigh the cons. I have a clear understanding of how much each of them costs me, for instance. Plus, the ban hammer will not be able to destroy all my income in one blow.
- simonw 4y ago
  Which bank is that?
  - moralestapia 4y ago
    BBVA (operates in LATAM and Spain) They give you an app where you can create virtual credit cards. Quite neat, I literally waited for something like that for decades. Coincidentally, I just set up a Mercury account last week and I think they have a similar feature but haven't explored it yet.
  - twelve40 4y ago
    in the US, at least Citi (that i know of) lets you create virtual card numbers.
- tomrod 4y ago
  For phone, what are your thoughts on something like Google Voice instead? For emails, how do you handle email? Outlook/Google Workspace? Something else?
  - moralestapia 4y ago
    I've never used Google Voice so I cannot say about that. Where I'm at you can buy a cheap phone for like $20 and it comes with a prepaid SIM that never expires. I have about 10 of those on a drawer on my desk LOL, it goes well with the hacker vibe B). Funny thing is I don't have them labeled, so when I need it I have to turn on all of them to see which one gets an SMS. For email, my domain provider handles email for me as well, I have a Google Workspace account and first thing I do when I set up an email is to link it there so I have a single Inbox for everything. (Although my real Inbox is the Spam folder, where a lot of genuine emails land now).
simonw 4y ago
This is fascinating. One thing that worries me: billing. If I have six different AWS accounts will I have to update six different places any time my credit card expires?
- forgomonika 4y ago
  AWS Organizations (https://aws.amazon.com/organizations/ https://aws.amazon.com/organizations/) is supposed to solve this for you. If all your accounts are in the same Organization then you get one bill instead of six.
  - jackson1442 4y ago
    You can also use AWS SSO (IAM Identity Center now) in conjunction with Organizations to federate into your AWS accounts using any SAML IdP.
- rcrowley 4y ago
  As the other commenter notes, no, you still have just one bill. Better, though, that one bill is broken down by account so you can see where the money’s going.
- deleted 4y ago
  [deleted]
twblalock 4y ago
I've seen this tried. It required a considerable investment in tooling and people to run it, because the dev teams just want to run their apps and don't want to manage accounts. And that's just the management complexity -- cross-account network adds a ton of complexity to other stuff, including VPC management, transit gateways, and sometimes DNS. Compared to having fewer accounts, the difference in complexity is enormous.
- rcrowley 4y ago
  Substrate [1] is meant to lessen the investment required to use lots of AWS accounts. Would love to know how it looks to you. [1] <https://src-bin.com/substrate/ https://src-bin.com/substrate/>
015a 4y ago
I would argue that the optimal number of AWS Accounts is Zero.
gw99 4y ago
I disagree with this perspective. You should have multiple accounts but only if your organisation requires it for isolation or data protection reasons and only enough to perform the task. Every other reason here is because you fucked up. You have poor architecture, poor tagging, poor VPC design, poor IAM policy and role modelling or don't know what you are doing to start with. And some of the stuff doesn't even make sense, particularly the point about EKS upgrades (I run three different versions of EKS in the same account). There are many negative effects of multiple accounts including billing aggregation is very difficult, having to dig through several accounts worth of Cloudwatch logs, unexpected egress traffic costs and the overall complexity is much higher. If you have to switch to an org account and set up SSO you then have an administrative cost which is immense. My favourite thing doing is spending 2 days opening support tickets in 10 different accounts to get a limit raised and then tracking the state of all the tickets and limit changes...
- forgomonika 4y ago
  Is billing aggregation a problem and do you need to open up 10 different support tickets if all the accounts are part of an AWS Organization? As for keeping the number of accounts down, I've seen that blow up significantly if you have someone break into an account or have a disgruntled employee that can get into a single account and wreak massive damage. Or if your software runs in multiple regions and you need to meet compliance requirements like GDPR, etc.
  - gw99 4y ago
    None of these concerns require multiple accounts. An account is a container for resources and resources do have tangible isolation between them if configured properly and properly delegated credentials should be issued to staff which are specific and limited in capability. Same with assumed roles. Same with VPC configurations. If you didn't do that, you fucked up. Adding more accounts doesn't guarantee that you didn't fuck up. And yes you need to open 10 tickets, even if you have enough spend to have a direct line to AWS internal staff with multiple enterprise accounts...
    - 4y ago
traspler 4y ago
At the company I work they just built their „Landing Zone“ as AWS calls it with all the on-prem connectivity, shared VPCs, Product Catalogs, IAM restrictions to the moon and so on and every team gets their own account linked to that one. It‘s an enormous amount of work to get all of that to work nicely together but when it works it‘s very nice and allows for a great way to partition responsibilities. For smaller deployments I think you will be in a mess pretty fast when you start doing cross-account things without enough planning ahead. But per-project accounts? Sure!
- rcrowley 4y ago
  Substrate [1] is meant to help folks not make a mess of lots of AWS accounts. Would love to know if it feels less enormous. [1] <https://src-bin.com/substrate/ https://src-bin.com/substrate/>
yonixw 4y ago
My anecdote on how we do it: - We have AWS Org - Each account has no root IAM and cost/pricing goes through root AWS Org Account - You move between accounts with AWS SSO (now IAM Federation) - No more password per account - AWS SSO standardizes boundaries across account with IAM policies, like eu-centeral-1 only for dev IAM etc. - Inside Account more granular access with IAM Assume Roles - Each account Cloudtrail to a central S3 for audit (Same region pricing works for diff accounts!) - Each account is `project.env` like micro_service1.dev or company1_k8s+s3.prod - That way, we have pricing per project built in where tag support ends (and it do have limits) We came to this structure realization after we saw Azure Resource Groups and Google Projects. We also saw the 5 VPC soft limit AWS has per region and think it's kind of a clue from amazon: "pss.. this account soft/hard limits is sized for 1 project/deployment" The only problems come from automating stuff, like Route53 records that can point automatically to load balancers in that account or as mentioned in the article, VPC2VPC, although we use serverless stuff like lambda and s3 and experience less of that. But as time go on we realized, like the VPC example in the article, that those problems forced us to structure our stuff in a more SOLID way, which became a feature for us. Just like moving docker-compose to k8s is not creating app-mesh and ops problem, but REVEALING them. So the solution for the Route53 example is to have a separate subdomain zone in each account for automation and ADDING another account `main_route53.prod` with a root zone pointing to them. Hope this helps for the curious.
- zimbatm 4y ago
  AWS Control Tower is great to set all of this up. It's basically a layer on top of AWS SSO, AWS Org, Cloudtrail, AWS Config. With some sane default security policies.
  - rcrowley 4y ago
    Control Tower is cool if the problem is “I need lots of AWS accounts.” Substrate [1] is cool if the problem is “I need to accomplish something and I’m cool with using lots of AWS accounts to do it.” [1] <https://src-bin.com/substrate/ https://src-bin.com/substrate/>
- pan69 4y ago
deleted 4y ago
[deleted]
felipelalli 4y ago
There is a practical problem: AWS has a very restrictive policy for deleting sub-accounts. From: https://docs.aws.amazon.com/organizations/latest/APIReference/API_CloseAccount.html https://docs.aws.amazon.com/organizations/latest/APIReferenc... "You can only close 10% of active member accounts within a rolling 30 day period. This quota is not bound by a calendar month, but starts when you close an account. Within 30 days of that initial account closure, you can't exceed the 10% account closure limit." I have create so many accounts that when I had to erase them I was stuck in this stupid limit.
- rcrowley 4y ago
  This is a bummer, yes. I haven’t looked but I wonder if that 10% is a soft limit. At any rate, this is a good reason to use accounts for architectural divisions, not teams, and certainly not individual engineers.
- nijave 4y ago
  Somewhere I worked used to just run awsnuke and move them to a "deleteme" OU. I guess that's an option if you're okay with a bunch of empty accounts (they have a few hundred in there)
ManuelKiessling 4y ago
Here‘s a very detailed step-by-step tutorial on how to realize resource-separation without giving up single sign-on with AWS: https://manuel.kiessling.net/2020/12/29/single-sign-on-and-resource-separation-on-aws/ https://manuel.kiessling.net/2020/12/29/single-sign-on-and-r... Only one of several ways to achieve that, but one that works really well for me.
epberry 4y ago
Multiple AWS accounts is definitely a best practice in larger engineering orgs. We implemented a CloudFormation StackSet to ingest them into our billing tool and lay them out appropriately. This proved to be a very slick solution from AWS so it made me believe that they want their larger customers to use multiple accounts too. If you are smaller I would not recommend it. Many things become a little more difficult, as others have pointed out. Oftentimes a devops or platform engineering org will paper over these things.
- nijave 4y ago
  AWS recommends a multi account strategy to help achieve their Well Architected framework. https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/organizing-your-aws-environment.html https://docs.aws.amazon.com/whitepapers/latest/organizing-yo...
woojae 4y ago
I've seen really stupid things done whenever there are multiple accounts. Very few developers know how to assume a role properly. This leads to assume-role permissions that are too permissive to make things work.
issa 4y ago
I am fairly certain this article could either be read as totally serious or complete satire. It works both ways.
sameg14 4y ago
Multiple AWS accounts makes life a living nightmare. We have 38 AWS accounts and it is incredibly difficult to maintain each one of them. IAM and even worse cross account IAM is horrible to author and maintain! Keeping track of resource limits and billing sucks. When using SSO, which we do, you cannot have more than one account open in the same browser at the same time. Use GCP instead, segregate your infra by project, use cloud identity/IAM and keep the head of hair you had in your 20s! You've been warned...
- rcrowley 4y ago
  Curious / product research: Are your 38 accounts all in the same organization? Do you have any human IAM users left or is it all IdP, all the time? Do you use Terraform or anything like it? Also, yes, a pox on the single-player AWS Console. I’ve at least found a way to logout from one account and login to another in the same motion but it’s still a poor experience.
  - sameg14 4y ago
    Yeah all accounts are in the same OU. We do have human IAM users but those are "legacy". Nowadays Okta has been the preferred method of accessing AWS console and CLI. We do use terraform but that is also fragmented since each team has the freedom to innovate in their own way. People use CDK, SAM, CloudFormation, Terraform etc. This fracturing of IaC techniques has been a natural consequence of having too many silos aka. accounts and has made it hard to enforce consistency. I think having 2 or 3 accounts is probably ok for a small to medium size org. We are 96 humans so far.
    - rcrowley 4y ago
      Interesting. Thanks for the detailed response. Another, positive way to look at one aspect of your architecture is that the AWS account boundary prevents most cases of dueling configuration management, with two tools changing the same resource back and forth forever.
  - MayeulC 4y ago
    I guess you could use Firefox container tabs?
jesuspiece 4y ago
Only if you're gonna use AWS Orgs/AWS Control Tower to help manage everything otherwise it gets wild. Internally, Amazonians also use AWS accounts per "app" basically.
- redditor98654 4y ago
  In fact per app per region that the AWS service exists in. My AWS service exists in 24 regions so that is 24 “prod” accounts, 8 “gamma” and 6 “beta” and 1 “alpha” account. Some newer teams go even further. If the app is made out of different micro services, then each micro service gets its own account per region. Typically a medium sized AWS service will have like 30 micro services behind it (not really micro; these tend to hold lots of code and APIs) so that would be 24 regions * 30 micro-service accounts. Tooling is important without which you cannot manage so many accounts.
  - libria 4y ago
    https://www.lastweekinaws.com/blog/the-aws-service-i-hate-the-most/ https://www.lastweekinaws.com/blog/the-aws-service-i-hate-th... > The fact that it’s the internal service used to provision AWS accounts means that AWS engineers building AWS are insulated from the way that the rest of the world manages AWS accounts–or should I say, ways. They don’t have to deal in the same way with AWS Organizations or Landing Zones or Control Tower or AWS SSO (an absolute hidden gem of a service, by the way). And that’s the crux of my beef with the service. Apparently, /u/quinnypig dislikes this account-manager-thing specifically because AWS has been hoarding it to themselves. His point is valid: When is AWS going to advertise account per region per service as an official default policy and bring AWS Orgs up to par with it? They think it's the right way, why not for the customer?
tobyjsullivan 4y ago
Define “lots.” Because the default limit in AWS is 10 accounts per org. Quota increases can be requested but the default limit tells us what AWS thinks normal usage should be for most use cases. https://docs.aws.amazon.com/organizations/latest/userguide/orgs_reference_limits.html https://docs.aws.amazon.com/organizations/latest/userguide/o... Perhaps the author meant “more than one”?
- psanford 4y ago
  > Quota increases can be requested but the default limit tells us what AWS thinks normal usage should be for most use cases. This is certainly not true for a lot of AWS quotas. If you hit an ec2 quota for an instance type, that isn't AWS telling you that you are using too much compute. Its there as a speedbump to make sure you have some idea that you know what you are doing and to make sure AWS can actually service your requests. AWS will happily let you have hundreds of child accounts. In fact, if you are talking to them about your architecture they will even encourage it (assuming that it is actually appropriate for the scale of your organization).
- encryptluks2 4y ago
  I think this just tells us author is using a method counterintuitive to what AWS would recommend, so while you may try it out that it probably isn't best practice and so when you screw up then you will just be left pointing to some blog as to why you chose the direction you did.
jiggawatts 4y ago
A related problem is multi-cloud orgs trying to copy this advice into other clouds. For example, at $dayjob, the cloud guy is trying to replicate the AWS account structure into Azure, where it's largely unnecessary. In Azure the equivalent of an account is a subscription, but then there's a second hierarchy level in the form of resource groups. Combined with tagging, this makes it very easy to do internal charge-backs or RBAC. They've even added a feature to redistribute the costs of shared resources: https://azure.microsoft.com/blog/simplify-financial-reporting-with-cost-allocation-now-in-preview/ https://azure.microsoft.com/blog/simplify-financial-reportin... Splitting things across accounts or subscriptions is complex and results in all sorts of strange limitations. For example, some resources can't be connected to each other across subscription boundaries, so you have to duplicate them. Another example is Kubernetes (EKS or AKS). The typical thing to do is to create a single cluster with multiple node pools, all in the same account or subscription. Then this is split using namespaces. All of this will be in one account, and can't be "associated" with other accounts or subscriptions. The second a shared platform like this is introduced, the fine-grained account model breaks down.
manv1 4y ago
Having multiple accounts reduces the potential blast radius tremendously. Anyone who hasn't accidentally deleted "stuff" in the wrong account is someone who hasn't been doing production work for long. One big downside is that it makes sharing resources between accounts more difficult...which I suppose might also be an upside, since that dependency needs to be explicit in the various permissions. It also makes tooling more awkward. But, it also allows you to completely automate deployment of consistent environments, providing a IT/CI/Testing nirvana for not that much work.
mongro1 4y ago
Lots of AWS accounts doesn't scale. Accounts are heavy items in terms of governance, manageability and cost. On your way to 100 accounts you'll be rearchitecting security and networking and will find yourself in a strange limbo of architecture models. Once over 100 you'll be drowning in the tech debt of a complex environment with increasing friction. Accounts can be made lightweight by using shared VPC/subnets but then you'll be in the realm of niche user, hampered by AWS's poor support for RAM service support with poor documentation if you plan on using anything off the highway of bread and butter services. IMO a balance needs to be struck with sensible boundaries built on business units or ownership. Shared VPC's are inherently unstable and should be avoided where possible. Build a good delegated IAM model and hammer people to use it properly.
- rcrowley 4y ago
  I’d love to hear more about your experience with shared VPCs. What’s inherently unstable about them?
throwayyy479087 4y ago
This is how AWS runs projects internally. Tons of dedicated, limited scope accounts.
thayne 4y ago
I agree with a lot of this, but it's missing any discussion of downsides to having a lot of accounts. Some of these include: * Granting permissions to resources in other accounts is complicated. Even where there is first class support, such as for s3 and kms, it involves multiple steps, and familiarity with confusing terminology. * Using the web console or cli is more complicated. In the cli you'll have to manage a bunch of profiles, and probably figure out a way to distribute that aws config to your team. And in the web console, switching between accounts is a huge pain unless you use a third party browser plugin to automate assuming a role (which normally requires knowing the account id). And even then, you need to give that plugin a mapping between names and account ids. * Several products charge per AWS account. Using a lot of accounts can make those products very expensive. * Having to assume a role in another account can complicate code, especially if you may or may not have to assume a role depending on the circumstances. I say this as someone with experience with working with several accounts, and who thinks that we should have more accounts. Even with these downsides, once you reach a certain size or complexity, the benefits outweigh the detriments. But the detriments are still there. I wish AWS did more to make working with a lot of accounts easier.
- vladvasiliu 4y ago
  > * Having to assume a role in another account can complicate code, especially if you may or may not have to assume a role depending on the circumstances. One workaround I usually implement for this is to always have to assume a role. Meaning that, basically, your base permissions only allow you to assume roles. To actually do the job, you need a separate one, in whichever account. This way, it's also easier to standardize roles across accounts. A action requires R role.
hatware 4y ago
In my experience, no business is able to shuffle its people around efficiently enough for this choice to make a difference. I can see situations where it would help, and I can see situations where it would absolutely hurt to have multiple accounts. Either way, people politics are going to get in the way more than any decision you make here. You can plan a pretty picnic but you can't predict the weather.
xwowsersx 4y ago
Has anyone here used Control Tower? [0] If so, I'm curious to hear your experience and whether it's worth it. [0] https://aws.amazon.com/controltower/ https://aws.amazon.com/controltower/
scumola 4y ago
I've used AWS for 12+ years. In the old days, we had one jumbo account that we separated by tags for projects and billing. It was a pain. Then about 5 years ago, AWS suggested that we use the multi-account strategy. I changed jobs and decided to go with Control Tower, AWS Orgs and the multi-account strategy. I spent more time writing automation to supplement and write terraform and python code to build supporting infrastructure to tie accounts together, enable the proper services and networking in all accounts. The automation for build a new account grew and grew and became a behemoth. There's no API for Control Tower and the process of setting up a new account with MFA enabled and all of the bells and whistles enabled is such a pain in the ass that I consider the AWS multi-account strategy not worth it AT ALL anymore I don't care how much it reduces the "blast zone" it's a monumentally stupid idea. You should try to put all of your eggs into one basket and manage access to your resources inside of IAM instead. Use tags smartly and you'll thank me in the long run.
- coredog64 4y ago
  Control Tower now has a limited API and it’s reasonable to expect additional capability in the future. Until then, there’s things like Account Factory for Terraform. Or, if you’re really burned out on Control Tower, you can check out OrgFormation.
nokya 4y ago
Sorry to ask but...what is an "account" in AWS? From what I read in the comments and the article, it seems to be everything but what I naively consider to be an "account". Could someone come with some analogy please?

228 points by rcrowley 4y ago

hgh 4y ago

AWS gets many things more right, but I think GCP wins on how they handle Projects and Managed Instance Groups for auto-scaling.

remram 4y ago
Easily separating billing accounts is definitely a must.
robohoe 4y ago
GCP organization structure is such a breath of fresh air after dealing with AWS. AWS Orgs and all the complexity with VPC and DNS management at a scale of hundreds of accounts is just a complete pain in the arse. GCP makes it far easier with Shared VPC and org/folder/project structure.

dijit 4y ago

One of the things I love most about google cloud is that "projects" are easy to create and easy to link to other projects. Roles and service accounts can even reference across projects, though I'm not sure I'd recommend doing that. No more faffing about with special accounts, passwords and difficult to configure shared VPCs, it all becomes so easy. Even managing the different accounts is difficult without browser extensions such as this one: https://chrome.google.com/webstore/detail/aws-extend-switch-roles/jpmkfafbacpgapdghgdpembnojdlgkdl?hl=en https://chrome.google.com/webstore/detail/aws-extend-switch-...

rcrowley 4y ago
Author of the article here: I agree. GCP projects are a better abstraction. I still think AWS is, on balance, a better cloud.
- dijit 4y ago
  I'd love to hear more about why you think that's the case! Maybe the next blog post?
  - simonw 4y ago
    The killer feature of AWS is that they've broken almost nothing since first launching over 15 years ago. Their commitment to keeping old stuff working is truly amazing. No other cloud hosting service comes close, because no other cloud service has had 15 years to prove themselves in the same way!
    - jimt1234 4y ago
      SimpleDB is still running!
0xbadcafebee 4y ago
I really dislike the Google Cloud way of doing things, for example Projects, Folders and Orgs. There's a dozen different weird paradigms that you have to consider to securely and reliably manage a large number of different projects, accounts, tenants, business units, etc in Google Cloud; if you don't do things "The Google Cloud Way" you are screwed. AWS is much simpler and more straightforward. You don't have to think about anything to segregate infrastructure, networks, applications, users, data, etc. Just put it in a different account. If someone needs access, you need to explicitly add extra connections/grant that access. You can't just accidentally create one user that implicitly has access to hundreds of accounts. Honestly the entire security model of Google Cloud is frightening. It's like they wanted to make it easy to expose everything.
- lbhdc 4y ago
  You can still work like that. You just don't have to bother changing your project id, and switch accounts to jump to other projects. I do this for some things I work on where there are account level controls and the accounts cant be shared across projects.
- deleted 4y ago
  [deleted]
- DangitBobby 4y ago
  AWS is decidely _not_ simpler here. You've got Stockholm syndrome. What's complicated about making two projects and picking between them? What's complicated about grouping projects under a single organization? What's complicated about grouping projects under a folder? What's complicated about adding someone to a project through IAM? The only way you can fuck that last part up is by granting overly broad permissions (which they warn you when you've done right there in the console). You can safely ignore the existence of Organizations and folders completely if you are so inclined. At least there's a sensible way to set default policies for your organization baked into the platform. Even so, literally 3 hours, tops, and you know everything about these abstractions you could ever care about. I cannot emphasize enough how much easier GCP is to learn and use than AWS.
  - 4y ago
atonse 4y ago
AWS SSO has made it incredibly easy for us to secure and manage access to (and switch between) all our AWS accounts in the org. I'm a huge fan and recommend it.
- tpmx 4y ago
  They recently rebranded it to the super-catchy "AWS IAM Identity Center (successor to AWS SSO)". Gotta love Seattle-based tech marketing people, presumably trained at Microsoft. :) I guess it will soon be the default user management system, and 'proper' IAM will be the low-level one. I see this as a reaction towards GCP's IMHO superior UX/system design in this aspect. I don't think they can entirely catch up because of early and bad architectural decisions regarding projects/accounts. Also a fan, so far.
- macintux 4y ago
  I was originally skeptical when we implemented it because for safety reasons I wanted it to be painfully obvious when I was switching contexts. It's definitely grown on me, but I frequently emphasize to my teammates that I'd much rather they specify their AWS_PROFILE for each command they run, instead of exporting it into their environment. Especially using Windows and setx seems like an open invitation for disaster.
GauntletWizard 4y ago
One reason to have your roles and service accounts reference cross-project is to give your CI builder access to your artifacts project; Have your prod and dev environments use containers from a third account that serves only as an archive of your build artifacts.
psanford 4y ago
I don't know. I've found it to be pretty difficult to answer the question "which of these 1000 gcp projects are running a production workload and which are random one offs created by a dev messing around or by a google sheet script?"
- dijit 4y ago
  I solve that by using folders. But when you have many of anything it will become hard to reason with. It’s not worse than your filesystem, in theory, but speaking for myself: my filesystem is a mess, so…
- DangitBobby 4y ago
  So the problem is the abstraction is so easy to use to get a project going that people just litter them everywhere... how is that a problem with the platform? At least they are actually in one searchable location. How do you find your production workloads if you have 1000 AWS accounts?
  - psanford 4y ago
    In AWS we divide things up into child accounts that roughly match our org tree. Each team or service gets dev, staging, and prod child accounts. Teams have access to their child accounts and its fairly obvious what they all do, or at least, who to talk to to find out. I think my main complaint with GCP is that it is often tied directly to an org's gsuite account. And doing things in gsuite (used to?) automatically create GCP projects behind the scenes. So you could easily get thousands of projects that the users themselves didn't even know that they had created. If I were starting from scratch with GCP I'd use a third party IdP and not let users access it via their gsuite accounts. I suspect that would avoid most of the issues I've run into.
    - DangitBobby 4y ago
      We haven't seen that gsuite project problem, but we also don't do really color outside the lines with gsuite. Email, chat, docs, AD, etc.

Sevii 4y ago

"Imagine you’re trying to create the kind of isolation necessary to deliver the security, reliability, and compliance that business customers demand in one AWS account." Got to be my favorite way to describe two nines ever!

rcrowley 4y ago
actual lol

Havoc 4y ago

Isn’t that against terms since it activates a free tier credit?

rcrowley 4y ago
I'm not honestly sure how AWS Organizations interacts with the AWS free tier. Rest assured, though, having lots of AWS accounts (and using AWS Organizations, their service designed to _help_ you use lots of AWS accounts) is _not_ against the terms of service.
- brodouevencode 4y ago
  Each account within the organization gets the limits of the free tier, just as a single account would. I think the reasoning is that if you're going to go through the hassle of setting up orgs then you're probably an enterprise user slated to take the long haul anyway.
  - kondro 4y ago
    Not true, the free tier is applied at the Organization level (i.e. the billing account), not to each individual account.
    - brodouevencode 4y ago
      Hmmm according to https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-free-tier.html https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2... it reads that it should be, but the wording is opaque. Hopefully someone from AWS can answer.
traspler 4y ago
Multi Account is a normal pattern for AWS. They even have tools to handle that better like AWS Control Tower. Someone from AWS we talked to even mentioned it as a perk (to get the free tier on every account).

pid-1 4y ago

> My favorite way to create a network between all my services hosted in different AWS accounts is to share a VPC from a network account into all my service accounts and use security groups to authorize service-to-service communication. There’s no per-byte tax, zonal architectures are easy to reason about, and security groups work just like you expect. That's gold advice. I wish AWS RAM supported more services (like AWS EKS). A small complain: working with AWS SSO is a bit tedious. My current solution is to share my ~/aws/config with everyone so we all have the same profile names and scripts can work for everyone.

rcrowley 4y ago
I have thought a great deal about whether I also want EKS clusters to officially support nodes in multiple AWS accounts. On the one hand, having the option to create that additional low-level isolation would be lovely, even and maybe even especially if I didn't always take it. On the other hand, isolating two things from each other but then tying them to the same Kubernetes cluster upgrade schedule feels wrong. In the end, I decided that if I care about isolating two things enough to put them in separate AWS accounts, I'm willing to spend the $75 per month that it takes to have separate EKS clusters, too. (This opinion perhaps obviously doesn't fit will with hobby/side project budgets.)
rcrowley 4y ago
Synchronizing ~/aws/config gets harder and harder as your team grows because there are both more people who need to receive changes and more people making changes. I think the human-readable names for AWS accounts need to be part of the account, not part of the laptop. Substrate [1] does this so that you can type commands like `substrate assume-role -domain example -environment production -quality beta` [2] to get where you're going. [1] <https://src-bin.com/substrate/ https://src-bin.com/substrate/> [2] <https://src-bin.com/substrate/manual/moving-between-aws-accounts/ https://src-bin.com/substrate/manual/moving-between-aws-acco...>
- nijave 4y ago
  One place I worked just committed it to the git repo (obviously just SSO and account numbers, no passwords or tokens)

tazjin 4y ago

DerekBickerton 4y ago

I thought this was an article about having separate compartmented Amazon accounts. So one for shopping, another for Amazon Associates, another for AWS, another for Prime Video and Alexa, etc Does anyone here do this?

mattpallissard 4y ago

> AWS accounts are the most complete form of isolation on offer. Because it's a pretty brutal namespace. While I usually wind up with separate accounts for multiple reasons, I strongly prefer to get rbac/iam properly implemented in a single account wherever possible.

forgomonika 4y ago
> I strongly prefer to get rbac/iam properly implemented in a single account wherever possible. This can really help manage complexity but it the more this account grows the riskier it becomes if a bad actor breaks into the account (ranging from fraudsters/hackers to disgruntled employees).
- mattpallissard 4y ago
  Using accounts as the security boundary is easier to reason about up front but it's rather ham fisted. If you're centrally automating large swaths of infrastructure across many accounts you'll wind up paying for that in the long run. When a shop smaller, or dealing with multi-tenancy situations, multiple accounts is an easy trade off to make. The account boundary has its place, but it's not for everything. Some of these scaling pains with hard boundaries have gotten better with awssso and iam features over the years but you still run into them on occasion. As far as whatever you mean by "breaking into an account". I said originally, "rbac/iam properly implemented". If you fuck that up, it doesn't matter whether or not you have one or multiple accounts. Having a single account does mean that you have to give anyone the keys to the kingdom or that you don't separate your concerns.

malcolp 4y ago

I love this approach, although I'm yet to work anywhere that does this. I guess the million (thousand?) dollar questios now become where do you draw the boundary across accounts? Presumably there are many bad ways to slice accounts up. And what happens when accounts do need to communicate? I can imagine three major scenarios for cross account permissions: * Cross account iam policies (painful in my experience) * Adding trust policies to enable cross account assuming roles (better but limited) * Limiting cross account policies to specific easy to configure services. E.g. S3 (best option, I've seen but maybe too limited) Would love to hear the author's view.

iLoveOncall 4y ago
> Would love to hear the author's view. Instead you can hear AWS's view, which is to have one account per stage per region per service. I can't find a source but I work for Amazon and this is what was recommended to us by ProServe (the contracting branch of AWS) when we talked with them. I think it's idiotic though (because regions are 100% separated within an account, and it would easily triple the number of accounts to manage), and so did my team, so we stuck with one account per stage per service. That said, cross account permissions is really not an issue, it's very easy and straightforward to setup. You also should not need it in 90% of the cases if your application is properly split with the right ownership for each microservice. For my current team we manage probably more than a thousand AWS accounts, and permissions are never an issue. Neither is anything else actually. We aggregate metrics in a single account for the stuff that needs to be aggregated, we have small CLI scripts that automate tedious steps like requesting limit increases, etc.
- brodouevencode 4y ago
  > Instead you can hear AWS's view, which is to have one account per stage per region per service. This is exactly correct. > I think it's idiotic though (because regions are 100% separated within an account But still bound to the same service limits, no?
  - iLoveOncall 4y ago

coding123 4y ago

This is one of those cases where the wrong thing gets optimized because of something stupid. ...Which is probably fine because that's pretty much how it always goes.

jackconsidine 4y ago

I have indeed found that it's difficult to silo access to specific resources with AWS IAM. I use these custom policies a lot, which give write access to a specific S3 Bucket [0], and give sending capabilities for a specific SES Identity [1] respectively: [0]https://koptional.notion.site/IAM-Policy-for-select-S3-Access-on-a-single-bucket-1d368c7eb45446b1bd653d1a0425042c https://koptional.notion.site/IAM-Policy-for-select-S3-Acces... [1] https://koptional.notion.site/IAM-Policy-for-email-sending-on-behalf-of-single-SES-identity-414f02acd0e4496eb994e171a74f3fdf https://koptional.notion.site/IAM-Policy-for-email-sending-o...

simonw 4y ago
I ended up writing a whole custom tool for generating credentials and policies for specific S3 buckets: - https://s3-credentials.readthedocs.io/ https://s3-credentials.readthedocs.io/ - https://simonwillison.net/2021/Nov/3/s3-credentials/ https://simonwillison.net/2021/Nov/3/s3-credentials/ - https://simonwillison.net/2022/Jan/18/weeknotes/ https://simonwillison.net/2022/Jan/18/weeknotes/
- jackconsidine 4y ago
  Whoa this is awesome! Wish I knew about this. It's really polished

moralestapia 4y ago

Every serious project I engage on has its own: * domain (obviously) * emails also for every provider I have a different email like (google@domain, twilio@domain, etc...) * credit cards (my bank makes it super easy to just create new ones) * phone no. (I just buy a burner phone) It's a bit of a PITA but the benefits outweigh the cons. I have a clear understanding of how much each of them costs me, for instance. Plus, the ban hammer will not be able to destroy all my income in one blow.

simonw 4y ago
Which bank is that?
- moralestapia 4y ago
  BBVA (operates in LATAM and Spain) They give you an app where you can create virtual credit cards. Quite neat, I literally waited for something like that for decades. Coincidentally, I just set up a Mercury account last week and I think they have a similar feature but haven't explored it yet.
- twelve40 4y ago
  in the US, at least Citi (that i know of) lets you create virtual card numbers.
tomrod 4y ago
For phone, what are your thoughts on something like Google Voice instead? For emails, how do you handle email? Outlook/Google Workspace? Something else?
- moralestapia 4y ago
  I've never used Google Voice so I cannot say about that. Where I'm at you can buy a cheap phone for like $20 and it comes with a prepaid SIM that never expires. I have about 10 of those on a drawer on my desk LOL, it goes well with the hacker vibe B). Funny thing is I don't have them labeled, so when I need it I have to turn on all of them to see which one gets an SMS. For email, my domain provider handles email for me as well, I have a Google Workspace account and first thing I do when I set up an email is to link it there so I have a single Inbox for everything. (Although my real Inbox is the Spam folder, where a lot of genuine emails land now).

simonw 4y ago

This is fascinating. One thing that worries me: billing. If I have six different AWS accounts will I have to update six different places any time my credit card expires?

forgomonika 4y ago
AWS Organizations (https://aws.amazon.com/organizations/ https://aws.amazon.com/organizations/) is supposed to solve this for you. If all your accounts are in the same Organization then you get one bill instead of six.
- jackson1442 4y ago
  You can also use AWS SSO (IAM Identity Center now) in conjunction with Organizations to federate into your AWS accounts using any SAML IdP.
rcrowley 4y ago
As the other commenter notes, no, you still have just one bill. Better, though, that one bill is broken down by account so you can see where the money’s going.
deleted 4y ago
[deleted]

twblalock 4y ago

I've seen this tried. It required a considerable investment in tooling and people to run it, because the dev teams just want to run their apps and don't want to manage accounts. And that's just the management complexity -- cross-account network adds a ton of complexity to other stuff, including VPC management, transit gateways, and sometimes DNS. Compared to having fewer accounts, the difference in complexity is enormous.

rcrowley 4y ago
Substrate [1] is meant to lessen the investment required to use lots of AWS accounts. Would love to know how it looks to you. [1] <https://src-bin.com/substrate/ https://src-bin.com/substrate/>

015a 4y ago

I would argue that the optimal number of AWS Accounts is Zero.

gw99 4y ago

I disagree with this perspective. You should have multiple accounts but only if your organisation requires it for isolation or data protection reasons and only enough to perform the task. Every other reason here is because you fucked up. You have poor architecture, poor tagging, poor VPC design, poor IAM policy and role modelling or don't know what you are doing to start with. And some of the stuff doesn't even make sense, particularly the point about EKS upgrades (I run three different versions of EKS in the same account). There are many negative effects of multiple accounts including billing aggregation is very difficult, having to dig through several accounts worth of Cloudwatch logs, unexpected egress traffic costs and the overall complexity is much higher. If you have to switch to an org account and set up SSO you then have an administrative cost which is immense. My favourite thing doing is spending 2 days opening support tickets in 10 different accounts to get a limit raised and then tracking the state of all the tickets and limit changes...

forgomonika 4y ago
Is billing aggregation a problem and do you need to open up 10 different support tickets if all the accounts are part of an AWS Organization? As for keeping the number of accounts down, I've seen that blow up significantly if you have someone break into an account or have a disgruntled employee that can get into a single account and wreak massive damage. Or if your software runs in multiple regions and you need to meet compliance requirements like GDPR, etc.
- gw99 4y ago
  None of these concerns require multiple accounts. An account is a container for resources and resources do have tangible isolation between them if configured properly and properly delegated credentials should be issued to staff which are specific and limited in capability. Same with assumed roles. Same with VPC configurations. If you didn't do that, you fucked up. Adding more accounts doesn't guarantee that you didn't fuck up. And yes you need to open 10 tickets, even if you have enough spend to have a direct line to AWS internal staff with multiple enterprise accounts...
  - 4y ago

traspler 4y ago

At the company I work they just built their „Landing Zone“ as AWS calls it with all the on-prem connectivity, shared VPCs, Product Catalogs, IAM restrictions to the moon and so on and every team gets their own account linked to that one. It‘s an enormous amount of work to get all of that to work nicely together but when it works it‘s very nice and allows for a great way to partition responsibilities. For smaller deployments I think you will be in a mess pretty fast when you start doing cross-account things without enough planning ahead. But per-project accounts? Sure!

rcrowley 4y ago
Substrate [1] is meant to help folks not make a mess of lots of AWS accounts. Would love to know if it feels less enormous. [1] <https://src-bin.com/substrate/ https://src-bin.com/substrate/>

yonixw 4y ago

My anecdote on how we do it: - We have AWS Org - Each account has no root IAM and cost/pricing goes through root AWS Org Account - You move between accounts with AWS SSO (now IAM Federation) - No more password per account - AWS SSO standardizes boundaries across account with IAM policies, like eu-centeral-1 only for dev IAM etc. - Inside Account more granular access with IAM Assume Roles - Each account Cloudtrail to a central S3 for audit (Same region pricing works for diff accounts!) - Each account is `project.env` like micro_service1.dev or company1_k8s+s3.prod - That way, we have pricing per project built in where tag support ends (and it do have limits) We came to this structure realization after we saw Azure Resource Groups and Google Projects. We also saw the 5 VPC soft limit AWS has per region and think it's kind of a clue from amazon: "pss.. this account soft/hard limits is sized for 1 project/deployment" The only problems come from automating stuff, like Route53 records that can point automatically to load balancers in that account or as mentioned in the article, VPC2VPC, although we use serverless stuff like lambda and s3 and experience less of that. But as time go on we realized, like the VPC example in the article, that those problems forced us to structure our stuff in a more SOLID way, which became a feature for us. Just like moving docker-compose to k8s is not creating app-mesh and ops problem, but REVEALING them. So the solution for the Route53 example is to have a separate subdomain zone in each account for automation and ADDING another account `main_route53.prod` with a root zone pointing to them. Hope this helps for the curious.

zimbatm 4y ago
AWS Control Tower is great to set all of this up. It's basically a layer on top of AWS SSO, AWS Org, Cloudtrail, AWS Config. With some sane default security policies.
- rcrowley 4y ago
  Control Tower is cool if the problem is “I need lots of AWS accounts.” Substrate [1] is cool if the problem is “I need to accomplish something and I’m cool with using lots of AWS accounts to do it.” [1] <https://src-bin.com/substrate/ https://src-bin.com/substrate/>
pan69 4y ago

deleted 4y ago

[deleted]

felipelalli 4y ago

There is a practical problem: AWS has a very restrictive policy for deleting sub-accounts. From: https://docs.aws.amazon.com/organizations/latest/APIReference/API_CloseAccount.html https://docs.aws.amazon.com/organizations/latest/APIReferenc... "You can only close 10% of active member accounts within a rolling 30 day period. This quota is not bound by a calendar month, but starts when you close an account. Within 30 days of that initial account closure, you can't exceed the 10% account closure limit." I have create so many accounts that when I had to erase them I was stuck in this stupid limit.

rcrowley 4y ago
This is a bummer, yes. I haven’t looked but I wonder if that 10% is a soft limit. At any rate, this is a good reason to use accounts for architectural divisions, not teams, and certainly not individual engineers.
nijave 4y ago
Somewhere I worked used to just run awsnuke and move them to a "deleteme" OU. I guess that's an option if you're okay with a bunch of empty accounts (they have a few hundred in there)

ManuelKiessling 4y ago

Here‘s a very detailed step-by-step tutorial on how to realize resource-separation without giving up single sign-on with AWS: https://manuel.kiessling.net/2020/12/29/single-sign-on-and-resource-separation-on-aws/ https://manuel.kiessling.net/2020/12/29/single-sign-on-and-r... Only one of several ways to achieve that, but one that works really well for me.

epberry 4y ago

Multiple AWS accounts is definitely a best practice in larger engineering orgs. We implemented a CloudFormation StackSet to ingest them into our billing tool and lay them out appropriately. This proved to be a very slick solution from AWS so it made me believe that they want their larger customers to use multiple accounts too. If you are smaller I would not recommend it. Many things become a little more difficult, as others have pointed out. Oftentimes a devops or platform engineering org will paper over these things.

nijave 4y ago
AWS recommends a multi account strategy to help achieve their Well Architected framework. https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/organizing-your-aws-environment.html https://docs.aws.amazon.com/whitepapers/latest/organizing-yo...

woojae 4y ago

I've seen really stupid things done whenever there are multiple accounts. Very few developers know how to assume a role properly. This leads to assume-role permissions that are too permissive to make things work.

issa 4y ago

I am fairly certain this article could either be read as totally serious or complete satire. It works both ways.

sameg14 4y ago

Multiple AWS accounts makes life a living nightmare. We have 38 AWS accounts and it is incredibly difficult to maintain each one of them. IAM and even worse cross account IAM is horrible to author and maintain! Keeping track of resource limits and billing sucks. When using SSO, which we do, you cannot have more than one account open in the same browser at the same time. Use GCP instead, segregate your infra by project, use cloud identity/IAM and keep the head of hair you had in your 20s! You've been warned...

rcrowley 4y ago
Curious / product research: Are your 38 accounts all in the same organization? Do you have any human IAM users left or is it all IdP, all the time? Do you use Terraform or anything like it? Also, yes, a pox on the single-player AWS Console. I’ve at least found a way to logout from one account and login to another in the same motion but it’s still a poor experience.
- sameg14 4y ago
  Yeah all accounts are in the same OU. We do have human IAM users but those are "legacy". Nowadays Okta has been the preferred method of accessing AWS console and CLI. We do use terraform but that is also fragmented since each team has the freedom to innovate in their own way. People use CDK, SAM, CloudFormation, Terraform etc. This fracturing of IaC techniques has been a natural consequence of having too many silos aka. accounts and has made it hard to enforce consistency. I think having 2 or 3 accounts is probably ok for a small to medium size org. We are 96 humans so far.
  - rcrowley 4y ago
    Interesting. Thanks for the detailed response. Another, positive way to look at one aspect of your architecture is that the AWS account boundary prevents most cases of dueling configuration management, with two tools changing the same resource back and forth forever.
- MayeulC 4y ago
  I guess you could use Firefox container tabs?

jesuspiece 4y ago

Only if you're gonna use AWS Orgs/AWS Control Tower to help manage everything otherwise it gets wild. Internally, Amazonians also use AWS accounts per "app" basically.

redditor98654 4y ago
In fact per app per region that the AWS service exists in. My AWS service exists in 24 regions so that is 24 “prod” accounts, 8 “gamma” and 6 “beta” and 1 “alpha” account. Some newer teams go even further. If the app is made out of different micro services, then each micro service gets its own account per region. Typically a medium sized AWS service will have like 30 micro services behind it (not really micro; these tend to hold lots of code and APIs) so that would be 24 regions * 30 micro-service accounts. Tooling is important without which you cannot manage so many accounts.
- libria 4y ago
  https://www.lastweekinaws.com/blog/the-aws-service-i-hate-the-most/ https://www.lastweekinaws.com/blog/the-aws-service-i-hate-th... > The fact that it’s the internal service used to provision AWS accounts means that AWS engineers building AWS are insulated from the way that the rest of the world manages AWS accounts–or should I say, ways. They don’t have to deal in the same way with AWS Organizations or Landing Zones or Control Tower or AWS SSO (an absolute hidden gem of a service, by the way). And that’s the crux of my beef with the service. Apparently, /u/quinnypig dislikes this account-manager-thing specifically because AWS has been hoarding it to themselves. His point is valid: When is AWS going to advertise account per region per service as an official default policy and bring AWS Orgs up to par with it? They think it's the right way, why not for the customer?

tobyjsullivan 4y ago

Define “lots.” Because the default limit in AWS is 10 accounts per org. Quota increases can be requested but the default limit tells us what AWS thinks normal usage should be for most use cases. https://docs.aws.amazon.com/organizations/latest/userguide/orgs_reference_limits.html https://docs.aws.amazon.com/organizations/latest/userguide/o... Perhaps the author meant “more than one”?

psanford 4y ago
> Quota increases can be requested but the default limit tells us what AWS thinks normal usage should be for most use cases. This is certainly not true for a lot of AWS quotas. If you hit an ec2 quota for an instance type, that isn't AWS telling you that you are using too much compute. Its there as a speedbump to make sure you have some idea that you know what you are doing and to make sure AWS can actually service your requests. AWS will happily let you have hundreds of child accounts. In fact, if you are talking to them about your architecture they will even encourage it (assuming that it is actually appropriate for the scale of your organization).
encryptluks2 4y ago
I think this just tells us author is using a method counterintuitive to what AWS would recommend, so while you may try it out that it probably isn't best practice and so when you screw up then you will just be left pointing to some blog as to why you chose the direction you did.

jiggawatts 4y ago

A related problem is multi-cloud orgs trying to copy this advice into other clouds. For example, at $dayjob, the cloud guy is trying to replicate the AWS account structure into Azure, where it's largely unnecessary. In Azure the equivalent of an account is a subscription, but then there's a second hierarchy level in the form of resource groups. Combined with tagging, this makes it very easy to do internal charge-backs or RBAC. They've even added a feature to redistribute the costs of shared resources: https://azure.microsoft.com/blog/simplify-financial-reporting-with-cost-allocation-now-in-preview/ https://azure.microsoft.com/blog/simplify-financial-reportin... Splitting things across accounts or subscriptions is complex and results in all sorts of strange limitations. For example, some resources can't be connected to each other across subscription boundaries, so you have to duplicate them. Another example is Kubernetes (EKS or AKS). The typical thing to do is to create a single cluster with multiple node pools, all in the same account or subscription. Then this is split using namespaces. All of this will be in one account, and can't be "associated" with other accounts or subscriptions. The second a shared platform like this is introduced, the fine-grained account model breaks down.

manv1 4y ago

Having multiple accounts reduces the potential blast radius tremendously. Anyone who hasn't accidentally deleted "stuff" in the wrong account is someone who hasn't been doing production work for long. One big downside is that it makes sharing resources between accounts more difficult...which I suppose might also be an upside, since that dependency needs to be explicit in the various permissions. It also makes tooling more awkward. But, it also allows you to completely automate deployment of consistent environments, providing a IT/CI/Testing nirvana for not that much work.

mongro1 4y ago

Lots of AWS accounts doesn't scale. Accounts are heavy items in terms of governance, manageability and cost. On your way to 100 accounts you'll be rearchitecting security and networking and will find yourself in a strange limbo of architecture models. Once over 100 you'll be drowning in the tech debt of a complex environment with increasing friction. Accounts can be made lightweight by using shared VPC/subnets but then you'll be in the realm of niche user, hampered by AWS's poor support for RAM service support with poor documentation if you plan on using anything off the highway of bread and butter services. IMO a balance needs to be struck with sensible boundaries built on business units or ownership. Shared VPC's are inherently unstable and should be avoided where possible. Build a good delegated IAM model and hammer people to use it properly.

rcrowley 4y ago
I’d love to hear more about your experience with shared VPCs. What’s inherently unstable about them?

throwayyy479087 4y ago

This is how AWS runs projects internally. Tons of dedicated, limited scope accounts.

thayne 4y ago

I agree with a lot of this, but it's missing any discussion of downsides to having a lot of accounts. Some of these include: * Granting permissions to resources in other accounts is complicated. Even where there is first class support, such as for s3 and kms, it involves multiple steps, and familiarity with confusing terminology. * Using the web console or cli is more complicated. In the cli you'll have to manage a bunch of profiles, and probably figure out a way to distribute that aws config to your team. And in the web console, switching between accounts is a huge pain unless you use a third party browser plugin to automate assuming a role (which normally requires knowing the account id). And even then, you need to give that plugin a mapping between names and account ids. * Several products charge per AWS account. Using a lot of accounts can make those products very expensive. * Having to assume a role in another account can complicate code, especially if you may or may not have to assume a role depending on the circumstances. I say this as someone with experience with working with several accounts, and who thinks that we should have more accounts. Even with these downsides, once you reach a certain size or complexity, the benefits outweigh the detriments. But the detriments are still there. I wish AWS did more to make working with a lot of accounts easier.

vladvasiliu 4y ago
> * Having to assume a role in another account can complicate code, especially if you may or may not have to assume a role depending on the circumstances. One workaround I usually implement for this is to always have to assume a role. Meaning that, basically, your base permissions only allow you to assume roles. To actually do the job, you need a separate one, in whichever account. This way, it's also easier to standardize roles across accounts. A action requires R role.

hatware 4y ago

In my experience, no business is able to shuffle its people around efficiently enough for this choice to make a difference. I can see situations where it would help, and I can see situations where it would absolutely hurt to have multiple accounts. Either way, people politics are going to get in the way more than any decision you make here. You can plan a pretty picnic but you can't predict the weather.

xwowsersx 4y ago

Has anyone here used Control Tower? [0] If so, I'm curious to hear your experience and whether it's worth it. [0] https://aws.amazon.com/controltower/ https://aws.amazon.com/controltower/

scumola 4y ago

I've used AWS for 12+ years. In the old days, we had one jumbo account that we separated by tags for projects and billing. It was a pain. Then about 5 years ago, AWS suggested that we use the multi-account strategy. I changed jobs and decided to go with Control Tower, AWS Orgs and the multi-account strategy. I spent more time writing automation to supplement and write terraform and python code to build supporting infrastructure to tie accounts together, enable the proper services and networking in all accounts. The automation for build a new account grew and grew and became a behemoth. There's no API for Control Tower and the process of setting up a new account with MFA enabled and all of the bells and whistles enabled is such a pain in the ass that I consider the AWS multi-account strategy not worth it AT ALL anymore I don't care how much it reduces the "blast zone" it's a monumentally stupid idea. You should try to put all of your eggs into one basket and manage access to your resources inside of IAM instead. Use tags smartly and you'll thank me in the long run.

coredog64 4y ago
Control Tower now has a limited API and it’s reasonable to expect additional capability in the future. Until then, there’s things like Account Factory for Terraform. Or, if you’re really burned out on Control Tower, you can check out OrgFormation.

nokya 4y ago

Sorry to ask but...what is an "account" in AWS? From what I read in the comments and the article, it seems to be everything but what I naively consider to be an "account". Could someone come with some analogy please?