3 ms·
I'm not sure I'd advocate for this design... The traffic sharing and backoff seems crude... The "10% per minute" doesn't prevent sudden increases in load kil
by londons_explore 7y ago
I'm not sure I'd advocate for this design... The traffic sharing and backoff seems crude... The "10% per minute" doesn't prevent sudden increases in load killing both providers.
I would design it like this:
Put all requests into a distributed queue, for example persistent pubsub.
Have workers take work from that queue.
Each worker should send a new request to a provider if the rate of requests sent in the past minute is < 2x the rate of requests in the previous minute, and the number of in flight requests is < 10x the average of the past minute, and the rate of errors, including timeouts, is <1%. If both providers are eligible, send to whichever has had the fewest requests in 24h.
This prevents flooding/DoSing a badly configured provider (a well configured provider would have ingress ratelimiting, and you could do away with all the above logic).
Have alerting on the age of the oldest item in the queue, and a monitoring dashboard showing dispatch rate to each provider, with response error codes.
All the state is local to the worker, and doesn't need persisting. If a worker crashes, the item doesn't get acknowledged to pubsub, and will be retried. If you like, you can autoscale the number of workers based on their cpu utilization.
I'd expect the above to scale to 10k qps per worker, and 5Mqps for 1000 workers before needing a redesign.