4 ms·

> it would be really nice if load balancers used realtime performance metrics to balance traffic at a millisecond level With Kubernetes this is trivial. Horizo

> it would be really nice if load balancers used realtime performance metrics to balance traffic at a millisecond level With Kubernetes this is trivial. Horizontal Pod Autoscaling integrates with Prometheus so can spin up a new instance of your service based on whatever custom metrics you like. And it is proven to work without requiring some "creative solution" that will be buggier, less secure and inevitably less maintained than something that is industry standard.

deleted 4y ago
[deleted]
remram 4y ago
Prometheus+Kubernetes needs in the order of 30-60s to scale up, not a good match with GP's "millisecond level" ask. (scrape interval is 15s, pod creation requires pulling images)
- pyth0 4y ago
  > (scrape interval is 15s, pod creation requires pulling images) You can reduce the scrape interval and depending on the cluster provider the image pulling should be extremely fast once cached but indeed it would be on the order of many seconds rather than milliseconds.
HALtheWise 4y ago
I don't think you understood what I was suggesting here, I'm referring to the load balancing system not auto scaling. Imagine a cluster with two nodes, that gets three requests. A round robin load balancer will distribute the first and third to node 1, and the second to node 2. However, if request 1 requires a lot of work for some reason (either it's an intrinsically complicated request or triggers gc or something), that means node 1 ends up heavily loaded and node 2 is underutilized. A smarter load balancer could notice that node 1 is still busy, and redirect request 3 to the node that has capacity available at the instant it comes in. Given large enough numbers of small requests, it should all average out in the long run, but that requires over provisioning by enough that servers can still handle their "fair share" of requests even while running GC or otherwise dealing with unusually difficult requests. Realtime smarter load balancing should require less overhead.

deleted 4y ago

[deleted]

remram 4y ago

Prometheus+Kubernetes needs in the order of 30-60s to scale up, not a good match with GP's "millisecond level" ask. (scrape interval is 15s, pod creation requires pulling images)

pyth0 4y ago
> (scrape interval is 15s, pod creation requires pulling images) You can reduce the scrape interval and depending on the cluster provider the image pulling should be extremely fast once cached but indeed it would be on the order of many seconds rather than milliseconds.

HALtheWise 4y ago

I don't think you understood what I was suggesting here, I'm referring to the load balancing system not auto scaling. Imagine a cluster with two nodes, that gets three requests. A round robin load balancer will distribute the first and third to node 1, and the second to node 2. However, if request 1 requires a lot of work for some reason (either it's an intrinsically complicated request or triggers gc or something), that means node 1 ends up heavily loaded and node 2 is underutilized. A smarter load balancer could notice that node 1 is still busy, and redirect request 3 to the node that has capacity available at the instant it comes in. Given large enough numbers of small requests, it should all average out in the long run, but that requires over provisioning by enough that servers can still handle their "fair share" of requests even while running GC or otherwise dealing with unusually difficult requests. Realtime smarter load balancing should require less overhead.