3 ms·
What about this is a surprising result? Optimize as directly as possible for load balancing the thing(s) that matter most to your users (e.g. latency), not the
by layla5alive 2y ago
What about this is a surprising result? Optimize as directly as possible for load balancing the thing(s) that matter most to your users (e.g. latency), not the thing that indirectly correlates to the thing(s) that matter (CPU Utilization). I've been making this exact argument at work for a while.
Funny to read this article today, just after myself and two others at work also just saved the company we work at far more than our combined expected lifetime gross earnings with a single (different) optimization.
- froh 2y agoyou got me curious. can you share?
- aseipp 2y agoThe paper reiterates -- multiple times -- that optimizing for latency is obviously the right choice; the other major signal is instantaneous requests-in-flight (RIF). They explicitly mention, on the very first page, that their contributions are not that "low latency is good, aim for that", it's that A) they use a particular combination of RIF and latency to select replicas called their "hot-cold lexicographic rule" which they find works really well, and B) their "probes" (requests by the balancer to the backends, to discover what their current state is) are collected asynchronously rather than in the critical request serving path, to help further drive utilization. Most of the ink in the paper in Section 4 explains the design choices they made around probing as a result. I'm not done with the paper yet, but the basics are in fact written on page one.