Yeah, the trade here is lower tail latency for increased median latency (ideally the mean is the same but mightn’t be the case). For many systems this is desir

Yeah, the trade here is lower tail latency for increased median latency (ideally the mean is the same but mightn’t be the case). For many systems this is desirable. It’s also the reason that incremental and concurrent gc were invented even though you could just have a gc where the mutator runs until it can’t allocate anymore, then you do a big mark and sweep, then you resume the mutator. Decreasing tail latency can generally improve downstream systems. Let’s say latency for requests is iid and I have some intermediate system which receives a request, needs to send 20 requests to your server, wait for them all to complete, then responds. Then the median latency of my system will look like the 95th %ile latency of your system that it gets data from. If your system were a web server and immune a web browser then you could pay yourself on the back for having low median latency while your users actually experience your 95th %ile latency waiting for all the resources to load.