8 ms·
AWS Identity service handles 400M API calls every second
- lukeramsden 5y agoHave AWS / employees ever spoken about the stack they run for IAM? Sounds fascinating
- onlyrealcuzzo 5y ago400M qps seems mind boggling. What other things exist at this scale?? As people have pointed out, something like Dynamo is likely FAR more complex - but my understanding is that it's also 3-4 orders of magnitude smaller.
- nicoburns 5y agoThese are staggering numbers, but my understanding is that IAM roles are wholly contained within the boundaries of an AWS account, so it seems like something that could relatively easily be partitioned horizontally with no shared state required between instances.
- esotericimpl 5y agoPolicies can shared across accounts with orgs, in addition arns can be set to allow roles access from other accounts. Boundary is mainly self contained but there can be interactions outside, sometimes significantly. Having said that, under the hood who knows .
- Pxtl 5y agoImho, the fact that this is an impressive achievement speaks to how much overhead the web costs. "400m operations per second? Wow, that's almost 5% of the number of operations per second a typical consumer processor can do!" I realize this is an apples to Buicks comparison, I just get bothered by how millions of anything might be impressive when we have PCs that are designed with billions of everything.
- asdfasgasdgasdg 5y ago> I just get bothered by how millions of anything might be impressive when we have PCs that are designed with billions of everything. That is a strange thing to be bothered by. Are you similarly unimpressed by a $400M lottery jackpot, since CPUs can execute billions of instructions per second?
- orf 5y agoBig numbers. Are there any talks or papers out there about how IAM is implemented under the hood?
- bpicolo 5y agoNot sure on IAM, but Google put out a paper on Zanzibar. https://research.google/pubs/pub48190/ https://research.google/pubs/pub48190/
- kimburgess 5y agoAuth0 are also working on a Zanzibar inspired system. Some interesting notes on their exploration via: https://twitter.com/auth0lab https://twitter.com/auth0lab
- p_l 5y agoI can confirm that IAM works definitely not like Zanzibar - in last job we have started to regularly hit issues where various AWS components would refuse to work because IAM Policy got too big, so it's definitely not querying another API for specific permissions but somehow goes around with the whole resulting policy :-|
- jzelinskie 5y agoWhat kind of policies were you writing? IAM is structured in a hierarchical fashion to limit policy complexity so that performance is fairly consistent at the cost of expressiveness. Zanzibar-like systems tend not to limit the complexity/expressiveness, by instead relying on denormalized views of the data to keep queries fast. Disclaimer: I'm a founder of authzed (YC W21), a productized Zanzibar implementation
- p_l 5y agoWhat we know is that we started to have increasing amounts of "policy is too big" errors in AWS APIs. Not IAM APIs. But things like Sagemaker being unable to start an EC2 instance because the networking system couldn't evaluate IAM policy (the error messages were annoying in both being quite specific and not helpful ultimately). The computational complexity of the policies wasn't high (in fact, they were pretty much key-value things, as the only time we had something more complex was in AssumeRoleUsingWebIdentity calls - btw, that API sucks, embarassingly so - everything else was plain list of what a role could access).
- amelius 5y agoHow often do identities or access settings change? Not often, I guess. So it seems (to me) that they are accessing almost static data at a very high speed, which is not really a surprise. Am I missing something?
- artwr 5y agoA lot of the calls in the organizations I have worked for use assume role and are usually associated with temporary credentials. Those are often shortish lived and not static
- orf 5y agoThe policy documents don't change that much I think, but there is more than that that goes into evaluating a policy. You've got resource + org policies, all kinds of tags, complex conditionals, per-assumption stuff etc etc. Fascinating problem space.
- refulgentis 5y agoYes - their claim has no relation to time-to-response, so static data, etc. is irrelevant. Thought experiment: if it suddenly took 10 seconds to respond to the calls, the number of responses still equals the number of requests, so the request velocity mentioned in the headline remains the same.
- Spooky23 5y agoPeople and things need access tokens all of the time. Off the shelf services can handle most loads - Even things like AD can scale out to millions of users. Used to work on 20M+ user identity service that didn’t see volume anything like this, but when utilization went up usually it was just a matter of adding more replica databases and auth servers.
- CalRobert 5y agoSo three calls per minute for every human being on the planet.
- new_realist 5y agoAgreed, this is nothing to brag about.
- jstx1 5y agoSay what you want about Amazon and how they treat their developers, from the outside this looks like something really cool to work on.
- deleted 5y ago[deleted]
- electricshampo1 5y agoSeems 400M is aggregated qps worldwide. Wonder what avg qps looks like per iam server (and size of server).
- the_arun 5y agoAlso, what about peak t/qps per iam server? Average usually flattens the graph.
- anurag 5y agoWhile this is an impressive technical achievement, it's worth taking a step back to ask why IAM needs to serve 400M API calls every second when AWS has between one and two million active users. How would this number change if IAM were less complex? Edit: I understand how every machine needs to invoke IAM APIs and how temporary credentials and other uses increase the number super-linearly with every active user. Still, 400M RPS (nearly 35B requests/day) could be reduced significantly by improving the underlying object model so it scales down better. Right now, even a simple Lambda function that needs to access other AWS resources requires 3 API calls: create a policy, create a role, and connect the two.
- orf 5y ago> it's worth taking a step back to ask why IAM needs to serve 400M API calls every second when AWS has between one and two million active users The thought process that would lead to such a statement intrigues me. IAM activity scales with total aggregate activity across all workloads that run on AWS, which I would assume scales with the total activity of the billions of people who interact with AWS directly or indirectly each day.
- sofixa 5y agoIAM isn't only used for users. Every API call needs to do auth/authz, including cross-service.
- snak 5y agoJust for clarity: 3.456 × 10^13 34 560 000 000 000 ~35 long-scale, european billions (10^12) ~35000 short-scale, american/english billions (10^9)
- elango 5y agowould be interesting to see how much each server can handle, that is engineering
- ram_rar 5y ago400M RPS is mind boggling, even if one considers most of the data to be embarrassingly shard-ed. Kudos to engineers who built it. Otoh, it bothers me that every single service call needs to go to IAM to check for permissions. Has anyone explored other architectures/designs to circumvent centralized auth?