4 ms·

Even though each cluster was single AZ the whole system wasn't, so we weren't bound by the capacity of a single AZ. Most of the situations where we needed to d

Even though each cluster was single AZ the whole system wasn't, so we weren't bound by the capacity of a single AZ. Most of the situations where we needed to drastically scale up were known ahead of time as well (e.g. campaign from customer), and we would preallocate instances or even more clusters. I may be forcing my memory, but if I'm not mistaken, our auto scaling was setup in a way that the system could handle sudden load increases of ~50% without noticeable disruption. Spikes bigger than this could lead to increased latency and/or error rate.

kccqzy 3y ago
That's another way of saying your typical utilization ratio is 66%. Which is on the low side honestly. That said, it's a trade off between efficiency and load spike tolerance. I trust that the trade off is made with informed decision.
- rewmie 3y ago
  > That said, it's a trade off between efficiency and load spike tolerance. I trust that the trade off is made with informed decision. I don't think that relatively low utilization rates is the scenario that requires "informed decision". The only tradeoff in low utilization rate scenarios is cost, which might be outright cheaper and irrelevant once you do the math on the tradeoffs of using reserved instances vs the cost of scaling up with on-demand instances. You need to make a damn good case to chronically underprovision your system and expect it to autoscale your way into nickle-and-dime savings.
- ec109685 3y ago
  66% isn’t low utilization. You’re always going to have micro spikes, and you never want to clip, so keeping some headroom around feels smart. Unless you co-mingle online and offline (batch) traffic on same hosts, flat response times and high utilization aren’t compatible.
  - sitkack 3y ago
    High utilization means high variability and low resiliency and the last k-percentage of utilization causes highly non-linear effects.
    - ec109685 3y ago
      Smart budgeted retries can help smooth those peaks.
  - kccqzy 3y ago
    While I agree there's always going to be micro spikes and keeping some headroom is smart, 33% may be too much of a headroom for all but the most latency sensitive RPC services. Personally I aim for only 20% headroom.

kccqzy 3y ago

That's another way of saying your typical utilization ratio is 66%. Which is on the low side honestly. That said, it's a trade off between efficiency and load spike tolerance. I trust that the trade off is made with informed decision.

rewmie 3y ago
> That said, it's a trade off between efficiency and load spike tolerance. I trust that the trade off is made with informed decision. I don't think that relatively low utilization rates is the scenario that requires "informed decision". The only tradeoff in low utilization rate scenarios is cost, which might be outright cheaper and irrelevant once you do the math on the tradeoffs of using reserved instances vs the cost of scaling up with on-demand instances. You need to make a damn good case to chronically underprovision your system and expect it to autoscale your way into nickle-and-dime savings.
ec109685 3y ago
66% isn’t low utilization. You’re always going to have micro spikes, and you never want to clip, so keeping some headroom around feels smart. Unless you co-mingle online and offline (batch) traffic on same hosts, flat response times and high utilization aren’t compatible.
- sitkack 3y ago
  High utilization means high variability and low resiliency and the last k-percentage of utilization causes highly non-linear effects.
  - ec109685 3y ago
    Smart budgeted retries can help smooth those peaks.
- kccqzy 3y ago
  While I agree there's always going to be micro spikes and keeping some headroom is smart, 33% may be too much of a headroom for all but the most latency sensitive RPC services. Personally I aim for only 20% headroom.