4 ms·

Ideally all egress network connections should go via NAT Gateway & filters at NAT Gateway should have policies to say which request can go out. Failures do happ

Ideally all egress network connections should go via NAT Gateway & filters at NAT Gateway should have policies to say which request can go out. Failures do happen in programming, we need to have better guardrails ensuring security.

xxpor 5y ago
NAT != firewalling. People conflating the two leads to policies like needing NAT for ipv6.
jrockway 5y ago
What sort of software are you using for the NAT gateways? (I know that's an AWS term, but not what I'm looking for. I've used that to provide a consistent outbound IP for production traffic so that upstream providers can allowlist an IP to talk to their API. Dumb security model when TLS client certificates exist, but... easy to set up I guess.) I've always wanted to defend against the attack where an application routinely talks to some API hosted on AWS, but an attacker starts using the application to exfiltrate data to another API hosted on AWS. If you just look at outgoing IP addresses, you'll think something like "oh, that API just started another replica" and not "my app is doing something weird". I want to do MITM on my applications so that every outgoing payload can be inspected. (I come at this from a security angle, but I'm really more interested in debugging in production. "Page foo that talks to the bar API has stopped working" "Oh, here's the JSON it started returning instead of text/plain." Bug fixed in 5 minutes.) I know this sort of pattern is common for, say, corporate firewalls, but I haven't seen any good projects for doing it to arbitrary applications running in production. I looked at Apache Traffic Server which might be the right thing, but seems super old and doesn't support any integrations I'd want (OpenTracing, Prometheus). I also tried configuring Envoy to do what I want, but it also didn't work. (Things like Istio's Egress Gateways seem to not intercept TLS, so you just get a list of IP addresses requested, not URLs.) I was thinking of just writing something to do this, but I know that everyone on Earth wants the same thing, so I figure I'm just missing the obvious out of the box solution. I'm 100% OK with fail closed (applications must be configured to only egress through a known-trusted IP of the proxy, all other network connections fail), reconfiguring my applications (happy to use some library for this, or inject TLS certificates for authenting the MITM proxy to the application), and this only working with HTTP. Suggestions?
- samus 5y ago
  There is Blue Coat. The technology to MITM TLS connections is widespread, but I'd recommend against it. You would create a single point of failure for your whole infrastructure and a prime attack target. Also, if the proxy fails to correctly validate the TLS connection, every service will end up talking to the wrong endpoints. Possible alternative: a dedicated API facade that forwards to the real APIs, and blocking of all other connections. There might be a tool that generates one from Swagger specifications. Also, require a client certificate or a password for each service. This way, you get centralized logging and access control without breaking TLS.
tialaramex 5y ago
NAT is completely orthogonal to this problem. You say people could have network filters, they could, with or without NAT.
jiggawatts 5y ago
In the field this causes more issues that in solves. For example, overzealous secops admins block "port 80 outbound", because everyone knows that HTTP is "insecure" and HTTPS is "secure". Except that the SSL certificate on that HTTPS website needs OCSP on port 80 to be fully secured. If you block it, you get random 30 second timeouts and less security. In general, totally blocking the Internet and then punching holes is like playing whack-a-mole. At first, you'll block Windows Update because you should be "managing that" through something like WSUS or SCCM, right, right? No, the SCCM guys never did figure out how to manage the DMZ servers, so the most Internet exposed servers are not getting patched now. Congratulations! So you'll open up Windows Update. Except that you didn't, because you forgot that it also includes half a dozen vaguely related, infrequently used and undocumented additional URLs. Which are all CDNs. So they're a CNAME to a CNAME to a pool of A records approximately half the size of all of Azure. And there's the Linux machines. The BSD-based network appliances and their various call-home support features. And on, and on, and on. How exactly are you going to configure this in your "NAT"? You discover that you can't. You're going to have to use a HTTP web proxy. Congratulations, you've now blocked IPv6, HTTP/3, nerfed web performance for everyone everywhere, and introduced a terrifying single-point-of-failure that will even break basic authentication (Azure AD, Okta, SAML, etc...). You can lock out an entire network completely by accidentally powering off the web proxies, to the point that noone can log back on to turn them on. This is just the tip of the iceberg. I could rant for hours about how NAT and forced outbound proxies break more things than they solve. Famously, TLS v1.3 uses v1.2 in the protocol header because the morons that write web proxies don't understand how protocol versions work. Oh, okay, one more: Windows 11 and 2022 introduced new TCP optimisations (HyStart++) that improve performance on high bandwidth delay product links like modern cloud and 5G networks. Except that web proxies don't generally keep up with optimisations like this, so you gain nothing even if you upgrade the desktops and server to modern operating systems. (Same thing applies to Linux too)

xxpor 5y ago

NAT != firewalling. People conflating the two leads to policies like needing NAT for ipv6.

jrockway 5y ago

What sort of software are you using for the NAT gateways? (I know that's an AWS term, but not what I'm looking for. I've used that to provide a consistent outbound IP for production traffic so that upstream providers can allowlist an IP to talk to their API. Dumb security model when TLS client certificates exist, but... easy to set up I guess.) I've always wanted to defend against the attack where an application routinely talks to some API hosted on AWS, but an attacker starts using the application to exfiltrate data to another API hosted on AWS. If you just look at outgoing IP addresses, you'll think something like "oh, that API just started another replica" and not "my app is doing something weird". I want to do MITM on my applications so that every outgoing payload can be inspected. (I come at this from a security angle, but I'm really more interested in debugging in production. "Page foo that talks to the bar API has stopped working" "Oh, here's the JSON it started returning instead of text/plain." Bug fixed in 5 minutes.) I know this sort of pattern is common for, say, corporate firewalls, but I haven't seen any good projects for doing it to arbitrary applications running in production. I looked at Apache Traffic Server which might be the right thing, but seems super old and doesn't support any integrations I'd want (OpenTracing, Prometheus). I also tried configuring Envoy to do what I want, but it also didn't work. (Things like Istio's Egress Gateways seem to not intercept TLS, so you just get a list of IP addresses requested, not URLs.) I was thinking of just writing something to do this, but I know that everyone on Earth wants the same thing, so I figure I'm just missing the obvious out of the box solution. I'm 100% OK with fail closed (applications must be configured to only egress through a known-trusted IP of the proxy, all other network connections fail), reconfiguring my applications (happy to use some library for this, or inject TLS certificates for authenting the MITM proxy to the application), and this only working with HTTP. Suggestions?

samus 5y ago
There is Blue Coat. The technology to MITM TLS connections is widespread, but I'd recommend against it. You would create a single point of failure for your whole infrastructure and a prime attack target. Also, if the proxy fails to correctly validate the TLS connection, every service will end up talking to the wrong endpoints. Possible alternative: a dedicated API facade that forwards to the real APIs, and blocking of all other connections. There might be a tool that generates one from Swagger specifications. Also, require a client certificate or a password for each service. This way, you get centralized logging and access control without breaking TLS.

tialaramex 5y ago

NAT is completely orthogonal to this problem. You say people could have network filters, they could, with or without NAT.

jiggawatts 5y ago

In the field this causes more issues that in solves. For example, overzealous secops admins block "port 80 outbound", because everyone knows that HTTP is "insecure" and HTTPS is "secure". Except that the SSL certificate on that HTTPS website needs OCSP on port 80 to be fully secured. If you block it, you get random 30 second timeouts and less security. In general, totally blocking the Internet and then punching holes is like playing whack-a-mole. At first, you'll block Windows Update because you should be "managing that" through something like WSUS or SCCM, right, right? No, the SCCM guys never did figure out how to manage the DMZ servers, so the most Internet exposed servers are not getting patched now. Congratulations! So you'll open up Windows Update. Except that you didn't, because you forgot that it also includes half a dozen vaguely related, infrequently used and undocumented additional URLs. Which are all CDNs. So they're a CNAME to a CNAME to a pool of A records approximately half the size of all of Azure. And there's the Linux machines. The BSD-based network appliances and their various call-home support features. And on, and on, and on. How exactly are you going to configure this in your "NAT"? You discover that you can't. You're going to have to use a HTTP web proxy. Congratulations, you've now blocked IPv6, HTTP/3, nerfed web performance for everyone everywhere, and introduced a terrifying single-point-of-failure that will even break basic authentication (Azure AD, Okta, SAML, etc...). You can lock out an entire network completely by accidentally powering off the web proxies, to the point that noone can log back on to turn them on. This is just the tip of the iceberg. I could rant for hours about how NAT and forced outbound proxies break more things than they solve. Famously, TLS v1.3 uses v1.2 in the protocol header because the morons that write web proxies don't understand how protocol versions work. Oh, okay, one more: Windows 11 and 2022 introduced new TCP optimisations (HyStart++) that improve performance on high bandwidth delay product links like modern cloud and 5G networks. Except that web proxies don't generally keep up with optimisations like this, so you gain nothing even if you upgrade the desktops and server to modern operating systems. (Same thing applies to Linux too)