4 ms·

How do you set up capacity for each cluster? You'll reach a point where one cluster will be overloaded; then, it makes more sense to send some requests to a clu

by puzzle 8y ago

How do you set up capacity for each cluster? You'll reach a point where one cluster will be overloaded; then, it makes more sense to send some requests to a cluster in another region, rather than melting down the local one. GSLB supports that.

alpb 8y ago
(I am not 100% sure, but) I think this is done automatically. When you add an instance group behind a load balancer (https://cloud.google.com/compute/docs/instance-groups/ https://cloud.google.com/compute/docs/instance-groups/) (which is what GKE Ingress does), the load balancer is aware of the health/CPU utilization of the backend VM instances. With this information, the load balancer knows about the RPS (queries per second), CPU utilization, and number of connections established to each backend. In Kubernetes/GKE case, each "node pool" of a cluster is an "instance group" and GKE Ingress controller configures the LB with instance groups and health checks. I believe the load balancing strategy (QPS, CPU, connections etc) is currently not configurable through Kubernetes since Ingress API does not have a good way of exposing cloud-specific configuration details, yet.
- puzzle 8y ago
  I see. Perhaps latency can be an alternate metric to divine capacity/maxqps, because you could have a backend that is not using a lot of CPU, but is still very slow, because it's just waiting on the local DB (which is the one melting down). If region X is Y ms away, you only start sending traffic there once the local backed is Y+Z ms slower than usual. The problem with such a metric is that it's hard to tell overload-induced latency from other kinds of high latencies. Health checks are good, but they tell you there's a problem seconds after you have already triggered it. :-) Better to have some concept of nominal capacity. Yeah, the Ingress API is not advanced at all and most people end up using controller-specific, non-portable annotations. I thought perhaps that was the case here. Hopefully there will be an improved API soon.
sbr464 8y ago
You can set a cpu usage % limit and/or request per second limit.
nikhiljindal 8y ago
Right. As others have pointed out Google Cloud balancer supports setting up max requests limits, but this is not exposed in kubernetes ingress API. If required, users can set these limits directly on GCLB after setting it up with kubemci. But they will be overwritten by kubemci update. A common scenario is to setup Pod autoscaler and cluster autoscaler to scale clusters or add more clusters in the same region rather than route requests to a different region. Disclaimer: I work on kubemci.
stealthmodeclan 8y ago
There are balancing modes in GSLB. One cluster is deployed on one instance group. So you can set max cpu utilization or RPS for a region. Requests flow to different region when you breach the defined threshold for the defined metric

by puzzle 8y ago

alpb 8y ago

(I am not 100% sure, but) I think this is done automatically. When you add an instance group behind a load balancer (https://cloud.google.com/compute/docs/instance-groups/ https://cloud.google.com/compute/docs/instance-groups/) (which is what GKE Ingress does), the load balancer is aware of the health/CPU utilization of the backend VM instances. With this information, the load balancer knows about the RPS (queries per second), CPU utilization, and number of connections established to each backend. In Kubernetes/GKE case, each "node pool" of a cluster is an "instance group" and GKE Ingress controller configures the LB with instance groups and health checks. I believe the load balancing strategy (QPS, CPU, connections etc) is currently not configurable through Kubernetes since Ingress API does not have a good way of exposing cloud-specific configuration details, yet.

puzzle 8y ago
I see. Perhaps latency can be an alternate metric to divine capacity/maxqps, because you could have a backend that is not using a lot of CPU, but is still very slow, because it's just waiting on the local DB (which is the one melting down). If region X is Y ms away, you only start sending traffic there once the local backed is Y+Z ms slower than usual. The problem with such a metric is that it's hard to tell overload-induced latency from other kinds of high latencies. Health checks are good, but they tell you there's a problem seconds after you have already triggered it. :-) Better to have some concept of nominal capacity. Yeah, the Ingress API is not advanced at all and most people end up using controller-specific, non-portable annotations. I thought perhaps that was the case here. Hopefully there will be an improved API soon.

sbr464 8y ago

You can set a cpu usage % limit and/or request per second limit.

nikhiljindal 8y ago

Right. As others have pointed out Google Cloud balancer supports setting up max requests limits, but this is not exposed in kubernetes ingress API. If required, users can set these limits directly on GCLB after setting it up with kubemci. But they will be overwritten by kubemci update. A common scenario is to setup Pod autoscaler and cluster autoscaler to scale clusters or add more clusters in the same region rather than route requests to a different region. Disclaimer: I work on kubemci.

stealthmodeclan 8y ago

There are balancing modes in GSLB. One cluster is deployed on one instance group. So you can set max cpu utilization or RPS for a region. Requests flow to different region when you breach the defined threshold for the defined metric