3 ms·

What are y'all using for an egress gateway? I haven't been able to find one that meets my needs. Typically I use Envoy, because it supports Opentracing and Pr

What are y'all using for an egress gateway? I haven't been able to find one that meets my needs. Typically I use Envoy, because it supports Opentracing and Prometheus metrics, which is what I'm looking for for an egress proxy. (I also like circuit breakers for external APIs; if Slack is down, there's no point in sending them another request for a while and it should just fail fast.) But Envoy does not handle dynamic HTTPS proxying particularly well (you can't just set $http_proxy in your environment and have it Just Work; you would need to create a cluster and route for every upstream), so doesn't work too well for the egress gateway case. (Things like Istio claim to provide this, but they actually just configure Envoy to act as a TCP proxy and can't extract your Opentracing identifiers or response codes. So you can get metrics like "you sent 2398473 bytes to 13.249.46.27" but not "100% of requests to api.slack.com timed out". It's something, but not useful enough to be worth the configuration overhead for me.) I have also looked at some of the traditional egress proxies, like Apache Traffic Server. It seems like it was invented before anyone ever heard of distributed tracing or metrics, and doesn't support those. So it doesn't work for me. My ultimate goal is to configure every application I use to make all outgoing requests through a proxy, and then use network policies to physically prevent them from talking to any namespace that isn't the egress proxy or something it's authorized to talk to. That way, there can be no unexpected (and unaudited) communication with the outside world; if some container I use starts including a coin miner or a traffic sniffer or something, I'll know about it and can shut it down. But... nobody seems to care except me. So I might have to write my own, which is a pain.

therein 7y ago
I am biased because I contributed actively to Apache Traffic Server when I worked at a past employer. It has its quirks. I played a major role in migrating our company from ATS4 to ATS5 and then ATS6. There are many developers eager to help, especially if you're hosting or able to show up at the ATS summits. Envoy has native middleware support but as far as I know you have to build the middleware with the server. With ATS, you can use TS API or atscppapi and create shared objects that are dynamically loaded middleware. We had considered Facebook's proxygen at one point too but we already had way too many middleware targeting ATS that it didn't make much sense. However the hooks that they provide match almost 1-to-1 so it would be not that challenging. Even to this day, my past employer uses ATS in their PoPs. And then at the next layer of reverse proxies, and then the layer after that. So imagine PoP (level 0) to pick the right colo -> Level 1 to pick the right service cluster -> Level 2 to front the service instances are all ATS. After that you get to the origin servers that are discovered by a service discovery plugin loaded into ATS, just like all the higher levels. Each level discovers the downstream hosts over zookeeper back then and I had migrated that system to use Consul, so Consul now. For some context, we were probably handling 8000qps on each ATS, with extremely heavy use of middleware plugins to do everything from auth to dispatch to spam filtering with dynamic rulesets pushed from internal hosts. I benchmarked ATS with and without plugins. With no plugins, our configuration could push over 80000qps. We had ~36 PoPs, 20-30 instances at each PoP. That's around 8.6Mqps handled with ATS. Part of this was realtime bidding for ads [0], and the rest was actual user traffic, part of it static content, part of it dynamic. We had our own CDN but also had Akamai on the side to fallback to. AFAIK from my conversations with people in these summits, iTunes is behind ATS. Comcast also pipes your unencrypted traffic through an ATS instance of their own with some custom middleware. [0] - https://support.google.com/google-ads/answer/6366577?hl=en https://support.google.com/google-ads/answer/6366577?hl=en

therein 7y ago

I am biased because I contributed actively to Apache Traffic Server when I worked at a past employer. It has its quirks. I played a major role in migrating our company from ATS4 to ATS5 and then ATS6. There are many developers eager to help, especially if you're hosting or able to show up at the ATS summits. Envoy has native middleware support but as far as I know you have to build the middleware with the server. With ATS, you can use TS API or atscppapi and create shared objects that are dynamically loaded middleware. We had considered Facebook's proxygen at one point too but we already had way too many middleware targeting ATS that it didn't make much sense. However the hooks that they provide match almost 1-to-1 so it would be not that challenging. Even to this day, my past employer uses ATS in their PoPs. And then at the next layer of reverse proxies, and then the layer after that. So imagine PoP (level 0) to pick the right colo -> Level 1 to pick the right service cluster -> Level 2 to front the service instances are all ATS. After that you get to the origin servers that are discovered by a service discovery plugin loaded into ATS, just like all the higher levels. Each level discovers the downstream hosts over zookeeper back then and I had migrated that system to use Consul, so Consul now. For some context, we were probably handling 8000qps on each ATS, with extremely heavy use of middleware plugins to do everything from auth to dispatch to spam filtering with dynamic rulesets pushed from internal hosts. I benchmarked ATS with and without plugins. With no plugins, our configuration could push over 80000qps. We had ~36 PoPs, 20-30 instances at each PoP. That's around 8.6Mqps handled with ATS. Part of this was realtime bidding for ads [0], and the rest was actual user traffic, part of it static content, part of it dynamic. We had our own CDN but also had Akamai on the side to fallback to. AFAIK from my conversations with people in these summits, iTunes is behind ATS. Comcast also pipes your unencrypted traffic through an ATS instance of their own with some custom middleware. [0] - https://support.google.com/google-ads/answer/6366577?hl=en https://support.google.com/google-ads/answer/6366577?hl=en