3 ms·
> Timeouts, retries, and resilience At a previous employer I was responsible for a critical service that was starting to show strain as traffic ramped up. It u
by cratermoon 5y ago
> Timeouts, retries, and resilience
At a previous employer I was responsible for a critical service that was starting to show strain as traffic ramped up. It used Hystrix as the circuit breaker for calls to backend services, including the DB, and at peak times the thread pool would fill and start rejecting additional requests. I was tasked with fixing that.
There's a very simple formula for tuning the number of threads:
> requests per second at peak when healthy × 99th percentile latency in seconds + some breathing room
The catch is that getting good RPS and latency numbers is in a distributed system deployed across three geographically separated datacenters is the opposite of simple. In particular, the legacy of the system meant that we had one write instance of the DB, in one datacenter, meaning that latency was different depending on which DC was the source of the call, so there was no one setting that worked for all instances.