I think it's wise to minimize the number of synchronous reads necessary for correct operation. Pervasive usage of immutable data and (expiring) cryptographic c

I think it's wise to minimize the number of synchronous reads necessary for correct operation. Pervasive usage of immutable data and (expiring) cryptographic capability tokens makes this much easier. Typically, the use of immutable data results in slower writes, but simpler caching logic, which can often result in faster reads in distributed systems. Luckily, many workloads are very read-heavy. If your CI/CD pipeline uses capability-based permissions, utilizing delegatable expiring cryptographic tokens (e.g. Google's Macaroons), then a stale read of the ACL (presenting a token incompatible with the latest ACL) would allow the CI/CD pipeline to start running. Completing all non-local I/O (i.e. observable side-effects) would require synchronous reads of a cryptographic hash of the latest repository ACL and comparison with an ACL hash in the token (with a slow path of re-verifying the token could still be issued in case of ACL changes). Presumably, this ACL hash would be a part of the repository's immutable root node, so in the common case of CI/CD pipelines storing outputs into a repository, this synchronous read is "free" in the sense that it's part of the transaction to change the reference to the repository's immutable root node. Granted, you have to be careful and make sure all of the I/O that's not local to the CI/CD pipeline (i.e. all observable side-effects, e.g. releases, sending emails) goes through through an ACL check that involves a synchronized read of the repository root reference. The fast path is just checking the cryptographic hash of the ACL in the capability token against the hash in the repository's root node. The slow path (when the ACL has changed since the capability token was generated) involves re-checking the ACL and verifying that the user named in the token is still allowed to create the presented token. The downside is a racy read of the ACL (in the form of presenting a cryptographic token based on a revoked permission) when starting the CI/CD pipeline results in wasteful resource usage. If this is a worry, an additional synchronous read can be added at pipeline start time to avoid this race. If the cost of this synchronous read is prohibitive, a cache of <user, revoked_permission, revocation_time> can be added to the CI/CD start logic. The upsides are that you can cache ACL reads client-side (in the form of your cryptographic capability tokens), needing only synchronous ACL reads for non-local I/O (i.e. observable side-effects) and if an attacker starts a malicious CI/CD operation, locking out the attacker before the malicious I/O (releases, emails, etc.) completes results in the malicious non-local I/O (i.e. observable side-effects) getting discarded.