8 ms·

Envoy Proxy at Reddit

136 points by runesoerensen 8y ago

ckwang 8y ago
Hello! Blog post writer here. Happy to answer any questions.
- shay_ker 8y ago
  Hi Courtney, thanks for the post. Others have mentioned that there are some gotchas with Envoy, and you mention a few about the migration bumps. Did you encounter other gotchas? And do you have any suggestions on how to avoid/mitigate their impact?
  - ckwang 8y ago
    Let me start by saying: The gotchas that we've experienced have been far eclipsed by the benefits. Aside from what what discussed, there weren't many things with Envoy specifically. We did have a few minor issues with the Thrift tooling, e.g. https://github.com/envoyproxy/envoy/commit/a3c744294bca2cbf612a5c8dc8d6ea13c63f3f38 https://github.com/envoyproxy/envoy/commit/a3c744294bca2cbf6... but as the above indicates, they were resolved _very_ quickly. The most important thing when making a transition like this is to have as much monitoring and observability as possible without the new tech. We were able to quickly identify and respond to issues we had with Envoy based on existing application and system instrumentation that weren't directly provided by Envoy, along with the vigilance of our engineering team.
- gbrayut 8y ago
  Awesome blog post! I very much enjoy hearing how large web properties implement these technologies and any issues they experience along the way. Are you using envoy at all in your main http ingress path? You mentioned haproxy and AWS ELBs, but it wasn't clear if envoy is also being considered for public ingress traffic. Keep up the great work!
  - ckwang 8y ago
    We have not yet put Envoy in our main HTTP ingress path, but internally we have designs and implementation paths ready to go, and it's definitely being considered for public ingress traffic. As we noted in the last "teaser" section of the post we'd really like to leverage Envoy's routing functionality to facilitate migrating client-facing APIs in the backend without affecting frontend interfaces. Our HAProxy layer that routes ingress traffic to the core backend infrastructure has considerable routing logic that can be moved to Envoy and then further extended. We'd love to explore that path in the coming months.
    - gbrayut 8y ago
      I look forward to hearing more about your plans for ingress and how the various pieces fit together (CDN, L4/L7 LBs, TLS termination, Geo/policy DNS balancing). Especially regarding the performance and new features available using Envoy. I've use HAProxy before and it was great for simple routing/reverse proxy but not so great at complex/dynamic configuration or cert management.
      - ckwang 8y ago
        Yes! HAProxy is a terrific piece of tech, and has been awesome for our use cases so far. We do quite a bit with it for our main ingress routing and it was basically flawless as our data plane in SmartStack. I'm really excited about what we're building out for next year and can't wait to share as well. Feel free to reach out on reddit (u/wangofchung) or directly at courtney.wang@reddit.com for more in-depth discussion!
      - rogerdonut 8y ago
        HAProxy supports quite complex configurations. We've actually found that many of our users are only realizing extremely basic capabilities so we have been working on increasing our blog content to help them take advantage of some of the more complex configurations that can be done. We've even found that many users are not aware that HAProxy now supports Hitless Reloads [1]. Quite a bit of complex routing and dynamic configurations can be provided by map files [2] and these and many other settings can be updated directly from the Runtime API [3]. With that said -- we are actively working to make things even better and intend to introduce support for updating SSL certificates/keys directly through the Runtime API as well as introducing a Data Plane API for HAProxy. We have a new release coming any day now and this will lay the foundation that will allow us to continue to provide best-in-class performance while accelerating cutting edge feature delivery. [1] https://www.haproxy.com/blog/hitless-reloads-with-haproxy-howto/ https://www.haproxy.com/blog/hitless-reloads-with-haproxy-ho... [2] https://www.haproxy.com/blog/introduction-to-haproxy-maps/ https://www.haproxy.com/blog/introduction-to-haproxy-maps/ [3] https://www.haproxy.com/blog/dynamic-configuration-haproxy-runtime-api/ https://www.haproxy.com/blog/dynamic-configuration-haproxy-r...
- cube2222 8y ago
  Hey, I myself am planning to introduce envoy into an existing mixed kubernetes/bare-metal architecture, having the same "one service at a time" considerations. Have you been thinking about adopting istio? If yes, why didn't you?
  - ckwang 8y ago
    We're currently evaluating the pieces that comprise Istio, both within Kubernetes and outside of it in our existing infrastructure. We didn't do so immediately because we did not want to immediately update all of our technology at once and felt that a piece-wise migration would be both the least disruptive to our infrastructure and safest. I think of Istio as like Smartstack in that it's not actually a complete "thing" so much as a suite of technologies that can be individually evaluated and deployed. It's very easy to fall into the trap of wanting to do everything at once, and we opted to make small progressive steps for this initiative.
    - cube2222 8y ago
      Yeah, I'm thinking of going with only the minimal istio installation (pilot only), not to roll our own, and setting it up with our existing consul service Discovery.
      - ckwang 8y ago
        That's pretty close to our next immediate steps as well :)
- pm90 8y ago
  This was a nice blog post! 1. Have you considered/are considering ISTIO control plane for your Envoy fleet? Why or why not? 2. Did you containerize your applications before using Envoy? The blog post talks about running them on autoscaled ec2 instances but its not clear if you're running application binaries on those vms or serving from containers
  - ckwang 8y ago
    1. We are considering Istio! This is especially true for our Kubernetes environment. We are already planning to deploy Pilot for the first iteration of our control plane in our non-K8s environment, so the other pieces that comprise Istio is a natural place for us to continue exploring. 2. We have not containerized prior to Envoy. We're running application binaries provisioned with Puppet on EC2 for most of our infrastructure still.
    - cube2222 8y ago
      2. Do you have multiple non-containerized services per machine? If yes, did you go with one proxy per machine or service instance?
      - ckwang 8y ago
        We run one proxy per machine, even when there are multiple services running. The proxy is just an abstraction to the downstream dependencies. Even if there are multiple services per machine, they can still reach downstream services via the same proxy path.
        cube2222 8y ago
        Thanks for all the answers, I really appreciate it! I've got one more question. In the period you had parts of your system with envoy and parts without, have your routed the outbound traffic from envoy-equiped services through their local proxy before reaching its envoy-less destination? Or did you omit envoy then?
yourduskquibble 8y ago
I see a lot of technical jargon but wondering how all of this relates and fits in with (if at all) the redesign?
- soonbesleeping 8y ago
  Agreed, it was hard to read from all the jargon. Also seems overly complex.
  - pm90 8y ago
    The original architecture seemed overly complex IMO. Maintaining a fleet of HAProxys and their configs seems daunting... it seems like they're gaining both flexibility (maintaining configs easily) and observability (request/response metrics for thirft requests)
    - ckwang 8y ago
      Per-service proxy deployments are a bit complex for the infrastructure but provide a nice abstraction for the service and service developers themselves. The configuration scheme is indeed daunting, which is what we're hoping Envoy and its xDS APIs + centralized configs can help us solve for developer teams.
  - rhizome 8y ago
    Like the redesign.
  - dalyons 8y ago
    if you arent familiar with the "jargon" in this post then you're probably not in a position to reasonably judge whether its "overly complex". Most everything in here would be familiar to anyone with experience working on modern high volume web properties.
  - ojhughes 8y ago
    Regardless of whether Service Mesh is overly complex, the industry seems to have entered an era of "Complexity Worship". I was speaking to some engineers at a small startup the other day with only a handful of customers. They have invested significant resource building there own K8S cluster, ensuring it runs on multi-cloud etc, sounds a lot like premature optimisation.
ojhughes 8y ago
The more logic we push out of band into sidecars, the harder application issues become to debug. For example, let's say an Envoy config change is made centrally and all of sudden my app breaks because an HTTP header has stopped being set. Before I would easily be able to write a unit test to fix such a thing. Now I would need to replicate the envoy config in a test environment etc.
- confiq 8y ago
  But I think that's the issue with ANY big software that you build. This specific is not a job for unittest but for integration test.
  - ojhughes 8y ago
    The sidecar model is different to integrating with a 3rd party API as it is designed to operate transparently. My Integration test might be passing but when running with the sidecar, traffic can be mutated etc
wlll 8y ago
Meta: "operationalizing", I really wish people wouldn't do this.

136 points by runesoerensen 8y ago

ckwang 8y ago

Hello! Blog post writer here. Happy to answer any questions.

shay_ker 8y ago
Hi Courtney, thanks for the post. Others have mentioned that there are some gotchas with Envoy, and you mention a few about the migration bumps. Did you encounter other gotchas? And do you have any suggestions on how to avoid/mitigate their impact?
- ckwang 8y ago
  Let me start by saying: The gotchas that we've experienced have been far eclipsed by the benefits. Aside from what what discussed, there weren't many things with Envoy specifically. We did have a few minor issues with the Thrift tooling, e.g. https://github.com/envoyproxy/envoy/commit/a3c744294bca2cbf612a5c8dc8d6ea13c63f3f38 https://github.com/envoyproxy/envoy/commit/a3c744294bca2cbf6... but as the above indicates, they were resolved _very_ quickly. The most important thing when making a transition like this is to have as much monitoring and observability as possible without the new tech. We were able to quickly identify and respond to issues we had with Envoy based on existing application and system instrumentation that weren't directly provided by Envoy, along with the vigilance of our engineering team.
gbrayut 8y ago
Awesome blog post! I very much enjoy hearing how large web properties implement these technologies and any issues they experience along the way. Are you using envoy at all in your main http ingress path? You mentioned haproxy and AWS ELBs, but it wasn't clear if envoy is also being considered for public ingress traffic. Keep up the great work!
- ckwang 8y ago
  We have not yet put Envoy in our main HTTP ingress path, but internally we have designs and implementation paths ready to go, and it's definitely being considered for public ingress traffic. As we noted in the last "teaser" section of the post we'd really like to leverage Envoy's routing functionality to facilitate migrating client-facing APIs in the backend without affecting frontend interfaces. Our HAProxy layer that routes ingress traffic to the core backend infrastructure has considerable routing logic that can be moved to Envoy and then further extended. We'd love to explore that path in the coming months.
  - gbrayut 8y ago
    I look forward to hearing more about your plans for ingress and how the various pieces fit together (CDN, L4/L7 LBs, TLS termination, Geo/policy DNS balancing). Especially regarding the performance and new features available using Envoy. I've use HAProxy before and it was great for simple routing/reverse proxy but not so great at complex/dynamic configuration or cert management.
    - ckwang 8y ago
      Yes! HAProxy is a terrific piece of tech, and has been awesome for our use cases so far. We do quite a bit with it for our main ingress routing and it was basically flawless as our data plane in SmartStack. I'm really excited about what we're building out for next year and can't wait to share as well. Feel free to reach out on reddit (u/wangofchung) or directly at courtney.wang@reddit.com for more in-depth discussion!
    - rogerdonut 8y ago
      HAProxy supports quite complex configurations. We've actually found that many of our users are only realizing extremely basic capabilities so we have been working on increasing our blog content to help them take advantage of some of the more complex configurations that can be done. We've even found that many users are not aware that HAProxy now supports Hitless Reloads [1]. Quite a bit of complex routing and dynamic configurations can be provided by map files [2] and these and many other settings can be updated directly from the Runtime API [3]. With that said -- we are actively working to make things even better and intend to introduce support for updating SSL certificates/keys directly through the Runtime API as well as introducing a Data Plane API for HAProxy. We have a new release coming any day now and this will lay the foundation that will allow us to continue to provide best-in-class performance while accelerating cutting edge feature delivery. [1] https://www.haproxy.com/blog/hitless-reloads-with-haproxy-howto/ https://www.haproxy.com/blog/hitless-reloads-with-haproxy-ho... [2] https://www.haproxy.com/blog/introduction-to-haproxy-maps/ https://www.haproxy.com/blog/introduction-to-haproxy-maps/ [3] https://www.haproxy.com/blog/dynamic-configuration-haproxy-runtime-api/ https://www.haproxy.com/blog/dynamic-configuration-haproxy-r...
cube2222 8y ago
Hey, I myself am planning to introduce envoy into an existing mixed kubernetes/bare-metal architecture, having the same "one service at a time" considerations. Have you been thinking about adopting istio? If yes, why didn't you?
- ckwang 8y ago
  We're currently evaluating the pieces that comprise Istio, both within Kubernetes and outside of it in our existing infrastructure. We didn't do so immediately because we did not want to immediately update all of our technology at once and felt that a piece-wise migration would be both the least disruptive to our infrastructure and safest. I think of Istio as like Smartstack in that it's not actually a complete "thing" so much as a suite of technologies that can be individually evaluated and deployed. It's very easy to fall into the trap of wanting to do everything at once, and we opted to make small progressive steps for this initiative.
  - cube2222 8y ago
    Yeah, I'm thinking of going with only the minimal istio installation (pilot only), not to roll our own, and setting it up with our existing consul service Discovery.
    - ckwang 8y ago
      That's pretty close to our next immediate steps as well :)
pm90 8y ago
This was a nice blog post! 1. Have you considered/are considering ISTIO control plane for your Envoy fleet? Why or why not? 2. Did you containerize your applications before using Envoy? The blog post talks about running them on autoscaled ec2 instances but its not clear if you're running application binaries on those vms or serving from containers
- ckwang 8y ago
  1. We are considering Istio! This is especially true for our Kubernetes environment. We are already planning to deploy Pilot for the first iteration of our control plane in our non-K8s environment, so the other pieces that comprise Istio is a natural place for us to continue exploring. 2. We have not containerized prior to Envoy. We're running application binaries provisioned with Puppet on EC2 for most of our infrastructure still.
  - cube2222 8y ago
    2. Do you have multiple non-containerized services per machine? If yes, did you go with one proxy per machine or service instance?
    - ckwang 8y ago
      We run one proxy per machine, even when there are multiple services running. The proxy is just an abstraction to the downstream dependencies. Even if there are multiple services per machine, they can still reach downstream services via the same proxy path.
      - cube2222 8y ago
        Thanks for all the answers, I really appreciate it! I've got one more question. In the period you had parts of your system with envoy and parts without, have your routed the outbound traffic from envoy-equiped services through their local proxy before reaching its envoy-less destination? Or did you omit envoy then?

yourduskquibble 8y ago

I see a lot of technical jargon but wondering how all of this relates and fits in with (if at all) the redesign?

soonbesleeping 8y ago
Agreed, it was hard to read from all the jargon. Also seems overly complex.
- pm90 8y ago
  The original architecture seemed overly complex IMO. Maintaining a fleet of HAProxys and their configs seems daunting... it seems like they're gaining both flexibility (maintaining configs easily) and observability (request/response metrics for thirft requests)
  - ckwang 8y ago
    Per-service proxy deployments are a bit complex for the infrastructure but provide a nice abstraction for the service and service developers themselves. The configuration scheme is indeed daunting, which is what we're hoping Envoy and its xDS APIs + centralized configs can help us solve for developer teams.
- rhizome 8y ago
  Like the redesign.
- dalyons 8y ago
  if you arent familiar with the "jargon" in this post then you're probably not in a position to reasonably judge whether its "overly complex". Most everything in here would be familiar to anyone with experience working on modern high volume web properties.
- ojhughes 8y ago
  Regardless of whether Service Mesh is overly complex, the industry seems to have entered an era of "Complexity Worship". I was speaking to some engineers at a small startup the other day with only a handful of customers. They have invested significant resource building there own K8S cluster, ensuring it runs on multi-cloud etc, sounds a lot like premature optimisation.

ojhughes 8y ago

The more logic we push out of band into sidecars, the harder application issues become to debug. For example, let's say an Envoy config change is made centrally and all of sudden my app breaks because an HTTP header has stopped being set. Before I would easily be able to write a unit test to fix such a thing. Now I would need to replicate the envoy config in a test environment etc.

confiq 8y ago
But I think that's the issue with ANY big software that you build. This specific is not a job for unittest but for integration test.
- ojhughes 8y ago
  The sidecar model is different to integrating with a 3rd party API as it is designed to operate transparently. My Integration test might be passing but when running with the sidecar, traffic can be mutated etc

wlll 8y ago

Meta: "operationalizing", I really wish people wouldn't do this.