6 ms·

Thanks for posting! How did you work around issues with GRPC on AWS/ELB?

spenczar5 9y ago
We tried grpc-gateway a little bit. But mostly, we wrote Twirp as a gRPC alternative because it was hard to work with ELBs.
justinko 9y ago
We had to stick to layer 4/TCP via an ELB (as opposed to an ALB).
- Xorlev 9y ago
  gRPC connections are persistent, so a single client will always talk to a single backend in that configuration. It's fine if you have lots of clients but can easily lead to load imbalances. That's why projects such as Envoy exist. I'd link it, but I'm on mobile. Keep an eye on it.
  - puzzle 9y ago
    You can use round robin load balancing in gRPC without Envoy.
    - tacticus 9y ago
      And it's not difficult to swap it out for a consistent hash balancer or other solution.
- johnbellone 9y ago
  You are doing L4 load balancing with the ELB? So you can’t do sticky session management? Is that a problem you run into, if so, how did you handle it?
cam-stitt 9y ago
When I implemented GRPC on ELB, we used a multiplexer to re-route the GRPC requests. See: https://github.com/soheilhy/cmux/ https://github.com/soheilhy/cmux/. The only other issue we had was that ELB's would not let connections live longer than 30 seconds.
- pebers 9y ago
  You can increase that connection timeout, although there's an upper limit of an hour or so which it will eventually exceed.
  - cam-stitt 9y ago
    I completely forgot this. Makes it a lot more feasible to use it.
weitzj 9y ago
What works for us (grpc) Elb with layer 4 and proxy protocol enabled. Behind Elb sits nghttpx (not nginx) doing TLS termination and request forwarding to gRPC. Proxy protocol is used to keep the source IP. This setup is all done with Kubernetes using kops for the cluster setup, nghttpx-ingress-lb as the ingress controller. Also we have multiple namespaces/environments in Kubernetes (staging/dev..), so nghttpx does routing based on the hostname. We tried linkerd before but somehow failed using it as an ingress controller doing TLS termination and upstream HTTP2. Doing the other way of routing everything through a dedicated linkerd port and a dtab worked, but mixing in TLS termination + upstream HTTP2 in a single dtab stopped us. So for now we keep this simpler setup and we probably are going to check out Istio/Heptio/Envoy

We tried grpc-gateway a little bit. But mostly, we wrote Twirp as a gRPC alternative because it was hard to work with ELBs.

justinko 9y ago

We had to stick to layer 4/TCP via an ELB (as opposed to an ALB).

Xorlev 9y ago
gRPC connections are persistent, so a single client will always talk to a single backend in that configuration. It's fine if you have lots of clients but can easily lead to load imbalances. That's why projects such as Envoy exist. I'd link it, but I'm on mobile. Keep an eye on it.
- puzzle 9y ago
  You can use round robin load balancing in gRPC without Envoy.
  - tacticus 9y ago
    And it's not difficult to swap it out for a consistent hash balancer or other solution.
johnbellone 9y ago
You are doing L4 load balancing with the ELB? So you can’t do sticky session management? Is that a problem you run into, if so, how did you handle it?

cam-stitt 9y ago

When I implemented GRPC on ELB, we used a multiplexer to re-route the GRPC requests. See: https://github.com/soheilhy/cmux/ https://github.com/soheilhy/cmux/. The only other issue we had was that ELB's would not let connections live longer than 30 seconds.

pebers 9y ago
You can increase that connection timeout, although there's an upper limit of an hour or so which it will eventually exceed.
- cam-stitt 9y ago
  I completely forgot this. Makes it a lot more feasible to use it.

weitzj 9y ago

What works for us (grpc) Elb with layer 4 and proxy protocol enabled. Behind Elb sits nghttpx (not nginx) doing TLS termination and request forwarding to gRPC. Proxy protocol is used to keep the source IP. This setup is all done with Kubernetes using kops for the cluster setup, nghttpx-ingress-lb as the ingress controller. Also we have multiple namespaces/environments in Kubernetes (staging/dev..), so nghttpx does routing based on the hostname. We tried linkerd before but somehow failed using it as an ingress controller doing TLS termination and upstream HTTP2. Doing the other way of routing everything through a dedicated linkerd port and a dtab worked, but mixing in TLS termination + upstream HTTP2 in a single dtab stopped us. So for now we keep this simpler setup and we probably are going to check out Istio/Heptio/Envoy