4 ms·
Even if the response changes only very infrequently, permissions are not what you want to be serving stale responses for. Can you confidently invalidate the rig
by bastawhiz 5y ago
Even if the response changes only very infrequently, permissions are not what you want to be serving stale responses for. Can you confidently invalidate the right caches in a system serving 400M QPS? I would bet that relatively little is cached in IAM today.
- idunno246 5y agoiam is eventually consistent...because its caching so much. we can see changes easily take a minute to propagate
- bastawhiz 5y agoReplication delay and stale caches are two different problems with vastly different causes
- josephh 5y agoCan you elaborate more? Is it not possible to have replication delay due to stale caches?
- bastawhiz 5y agoI'm not aware of a data store which populates its replicas with data from a cache. Surely there are caches within any given data store, but the whole point of a replica is to have the data replicated (for scaling reads, redundancy in case your leader fails). If you're using caches, you're deliberately sending old data to the replicas. I can't think of why you'd do that. I guess you could have replicas of your caches, but I'm referring specifically to data store replicas.
- nicoburns 5y agoIs a replica not a cache of sorts?
- bastawhiz 5y agoNot really. Replicas are constantly replicating live data from your leader. All of it, in its raw form. A query against a replica isn't inherently faster than a query against the leader, the data is likely just a bit old. A cache stores the output of an operation to avoid needing to perform the same operation again. You deliberately put data into a cache to avoid needing to create that data a second time—it won't magically appear there.
- isbvhodnvemrwvn 5y agoNot very stale, but there is definitely a period where the policy is unstable, e.g. if you allow some action, for a relatively short while (usually less than a minute) you can see stale responses where the action is blocked, sometimes interleaved with allowed access.
- asdfasgasdgasdg 5y agoIt depends on what exactly they're providing. Are they providing bounded staleness for certain types of reads? And is that the case for all reads, or do they also cover the new enemy problem? I.e. do they prevent people who previously had access to a resource from viewing changes to that resource?
- amelius 5y ago> Even if the response changes only very infrequently, permissions are not what you want to be serving stale responses for. But perhaps the permission updates are slow. Like, you change a permission, wait 5 seconds, and the system tells you that they are updated across all servers.
- bastawhiz 5y agoThe delay is either in replicating the data to all of the sources of truth, or invalidating the cache. Invalidate the cache too soon and you're recaching stale data. Caches only increase the delay more.
- jeffbee 5y agoYou don't always need to shoot down caches for consistency. See the Zanzibar paper for how one system maintains consistency without cache invalidation.