I'm very puzzled by the consensus group load balancing section. The article emphasizes correctness of the Raft algorithm was super important (to the point that

I'm very puzzled by the consensus group load balancing section. The article emphasizes correctness of the Raft algorithm was super important (to the point that they skipped clear optimizations!!11), but, then immediately follows up with (as far as I can tell) a load-balancer wrapper approach for rebalancing and scaling. My "this feels like consensus bug city" detectors immediately went off. Consensus algorithms (including Raft and Paxos) are notoriously picky and hard to get right around cluster membership changes. If you try to end run around this by sharding to different clusters with a simple traffic director to choose which cluster, how does the traffic director achieve consensus with the clusters that the traffic is going to the right cluster? You haven't solved any consensus problem, you've just moved it to your load balancers. A solution for this problem (to agree on which cluster the data is owned by) is 2-phase commit on top of the consensus clusters. It didn't appear from the diagrams that that's what they did here, so either I missed something, or this wouldn't pass a Jepsen test. Did I miss something? [If you did build 2PC on top of these consensus clusters, you'd have built a significant portion of Spanner's architecture inside of a secure enclave. That's hilarious.]