Lack of backpressure is perhaps the most common newbie mistake in distributed systems. I've just seen so many failures because the servers were overwhelmed and

Lack of backpressure is perhaps the most common newbie mistake in distributed systems. I've just seen so many failures because the servers were overwhelmed and the clients were greedy/impatient (didn't react well to anything less than an immediate response even when they were doing everything they could to overwhelm). It should be a primary design consideration for any such system being built anew. > Backpressure Strategies: control, buffer, drop One of my favorite hacks for Gluster (at Facebook) was in this area. We were using bog-standard NFS as the connection protocol, so neither explicit control nor dropping were options. (People who have always had that luxury with their own thrift/grpc/whatever protocols and libraries don't realize how much of a pain it is not to have it.) That left buffering, which led to memory pressure and latency bubbles. What I ended up implementing was implicit control. We'd just stop reading on a particular client's socket for increasing numbers of milliseconds if overall latency was too high and its queue was too long. This leveraged information we already had, and "borrowed" TCP's flow/buffer control as our own. Clients couldn't ignore it; they'd block in send() or somewhere equivalent. It worked really well to smooth out many of the worst latency spikes. If you're ever stuck in a similar situation, I highly recommend this approach.