(The other thing I forgot to mention was having a good consistent hashing strategy in the load balancer, so that the instances of each service consistently spea

(The other thing I forgot to mention was having a good consistent hashing strategy in the load balancer, so that the instances of each service consistently speak to the same instances of the other service, even as machines are spun up and down, but still maintains an even spread of load. This helps greatly in terms of allowing each instance to cache only the data for requests that target that instance. With a round-robin load balancer you end up with a lot more variation in your incoming requests, and thus require either more memory or expect more cache misses. That was probably the hardest part of the project, since the built-in load balancers don't have the specific functionality we needed).