3 ms·

Permissioning/ACLs is something that you really want to get synchronous whenever possible, you don't want caching and pipeline delays if you can help it. Some

Permissioning/ACLs is something that you really want to get synchronous whenever possible, you don't want caching and pipeline delays if you can help it. Some maintainer is acting in bad faith, someone else quickly locks them out of all the repositories that they have not yet thought to corrupt, then it looks really really bad if they find out that they still have permission to run the CI/CD scripts, maybe with malicious substitutions, even though you blocked them from being able to push commits up. The other part of that is, it's not too hard to get right. See what GitHub was doing. People don't change ACLs that often, nowhere near as often as you read them. If they did, you could rate-limit them. And the rest of the problem is, you have one writer and many read copies in one place and you can enforce boundaries on how stale the data gets on the read copies. Two hard problems, first is cache invalidation. Of course, they have found out that they are big enough to shard, one would have hoped that they would have found that out in a gentler way, much sooner. But let's not pretend that their architecture makes no sense, it's a fine architecture, they just happened to outgrow it in a bad way.

KMag 5y ago
I think it's wise to minimize the number of synchronous reads necessary for correct operation. Pervasive usage of immutable data and (expiring) cryptographic capability tokens makes this much easier. Typically, the use of immutable data results in slower writes, but simpler caching logic, which can often result in faster reads in distributed systems. Luckily, many workloads are very read-heavy. If your CI/CD pipeline uses capability-based permissions, utilizing delegatable expiring cryptographic tokens (e.g. Google's Macaroons), then a stale read of the ACL (presenting a token incompatible with the latest ACL) would allow the CI/CD pipeline to start running. Completing all non-local I/O (i.e. observable side-effects) would require synchronous reads of a cryptographic hash of the latest repository ACL and comparison with an ACL hash in the token (with a slow path of re-verifying the token could still be issued in case of ACL changes). Presumably, this ACL hash would be a part of the repository's immutable root node, so in the common case of CI/CD pipelines storing outputs into a repository, this synchronous read is "free" in the sense that it's part of the transaction to change the reference to the repository's immutable root node. Granted, you have to be careful and make sure all of the I/O that's not local to the CI/CD pipeline (i.e. all observable side-effects, e.g. releases, sending emails) goes through through an ACL check that involves a synchronized read of the repository root reference. The fast path is just checking the cryptographic hash of the ACL in the capability token against the hash in the repository's root node. The slow path (when the ACL has changed since the capability token was generated) involves re-checking the ACL and verifying that the user named in the token is still allowed to create the presented token. The downside is a racy read of the ACL (in the form of presenting a cryptographic token based on a revoked permission) when starting the CI/CD pipeline results in wasteful resource usage. If this is a worry, an additional synchronous read can be added at pipeline start time to avoid this race. If the cost of this synchronous read is prohibitive, a cache of <user, revoked_permission, revocation_time> can be added to the CI/CD start logic. The upsides are that you can cache ACL reads client-side (in the form of your cryptographic capability tokens), needing only synchronous ACL reads for non-local I/O (i.e. observable side-effects) and if an attacker starts a malicious CI/CD operation, locking out the attacker before the malicious I/O (releases, emails, etc.) completes results in the malicious non-local I/O (i.e. observable side-effects) getting discarded.

KMag 5y ago

I think it's wise to minimize the number of synchronous reads necessary for correct operation. Pervasive usage of immutable data and (expiring) cryptographic capability tokens makes this much easier. Typically, the use of immutable data results in slower writes, but simpler caching logic, which can often result in faster reads in distributed systems. Luckily, many workloads are very read-heavy. If your CI/CD pipeline uses capability-based permissions, utilizing delegatable expiring cryptographic tokens (e.g. Google's Macaroons), then a stale read of the ACL (presenting a token incompatible with the latest ACL) would allow the CI/CD pipeline to start running. Completing all non-local I/O (i.e. observable side-effects) would require synchronous reads of a cryptographic hash of the latest repository ACL and comparison with an ACL hash in the token (with a slow path of re-verifying the token could still be issued in case of ACL changes). Presumably, this ACL hash would be a part of the repository's immutable root node, so in the common case of CI/CD pipelines storing outputs into a repository, this synchronous read is "free" in the sense that it's part of the transaction to change the reference to the repository's immutable root node. Granted, you have to be careful and make sure all of the I/O that's not local to the CI/CD pipeline (i.e. all observable side-effects, e.g. releases, sending emails) goes through through an ACL check that involves a synchronized read of the repository root reference. The fast path is just checking the cryptographic hash of the ACL in the capability token against the hash in the repository's root node. The slow path (when the ACL has changed since the capability token was generated) involves re-checking the ACL and verifying that the user named in the token is still allowed to create the presented token. The downside is a racy read of the ACL (in the form of presenting a cryptographic token based on a revoked permission) when starting the CI/CD pipeline results in wasteful resource usage. If this is a worry, an additional synchronous read can be added at pipeline start time to avoid this race. If the cost of this synchronous read is prohibitive, a cache of <user, revoked_permission, revocation_time> can be added to the CI/CD start logic. The upsides are that you can cache ACL reads client-side (in the form of your cryptographic capability tokens), needing only synchronous ACL reads for non-local I/O (i.e. observable side-effects) and if an attacker starts a malicious CI/CD operation, locking out the attacker before the malicious I/O (releases, emails, etc.) completes results in the malicious non-local I/O (i.e. observable side-effects) getting discarded.