8 ms·

Using SNI proxying and IPv6 to share port 443 between webapps

138 points by petercooper 5y ago

karmanyaahm 5y ago
This is such a neat solution. As part of my effort to single-stack-v6 and minimum-effort-v4, for my self hosted services and projects, I've always wanted a way to avoid reverse proxy configuration for each hosted app; v6 can directly listen to a new address.
anderspitman 5y ago
This is great. I think SNI is currently one of the most pragmatic tools to work around ipv4 exhaustion. I like the approach described here, but in practice I prefer the convenience of having a reverse proxy to automatically handle TLS certs for me. That said, libraries like certmagic are making it more feasible for every app to manage its own certs.
- ignoramous 5y ago
  > I think SNI is currently one of the most pragmatic tools to work around ipv4 exhaustion. Pretty much: https://research.cloudflare.com/publications/Fayed2021/ https://research.cloudflare.com/publications/Fayed2021/ See also, DNS SVC (A/B) records (pseudo NAT at DNS layer); but not many deployments use it or understand it. Note that, SNI as a routing replacement works for TCP nicely without much (user-space) complication. With QUIC, transparently proxying connections isn't all that straight forward.
  - anderspitman 5y ago
    Thanks for the link. What exactly are the issues with QUIC? I've wondered if it's possible to do SNI routing with UDP but haven't looked into it yet.
    - ignoramous 5y ago
      QUIC hides and encrypts everything it possibly can, and that includes server and client identities (IPs are inconsequential to a QUIC session which is instead maintained through connection-id(s) exchanged under TLS encryption or is obfuscated away; that is, there is no way for a middleware to analyse a QUIC session / flow without actually MiTMing TLS). More: https://lwn.net/Articles/745590/ https://lwn.net/Articles/745590/
barbazoo 5y ago
Wow this is neat. > I have had it with standalone web servers: they're all over-complicated and I always end up with an awkward bifurcation of logic between my app's code and the web server's config Personally I've grown really fond of letting nginx terminate TLS and proxy to the web app. It's a clean separation of concerns, not very complicated and upgrading the cert is easy (certbot).
- francislavoie 5y ago
  FWIW, Caddy can replace nginx and certbot for this purpose, with less config and more robust ACME. Let me know if you need me to clarify, if you have any concerns.
  - iso1210 5y ago
    Caddy looks interesting, I currently use apache to proxy a few hundered sites and it works well enough, some are protected by client certificates, others by oidc, all then pass the authenticated user to the downstream server in a header, job done. I've managed to do this with openresty (nginx not supporting oidc out of the box), but it doesn't fill me with confidence, I guess it's all the lua. A quick glance at caddy shows it likewise doesn't support oidc integration out of the box, but instead I have to use another module that's no longer maintained ( https://github.com/thspinto/caddy-oidc https://github.com/thspinto/caddy-oidc )
    - francislavoie 5y ago
      Yeah, we defer to plugins to provide auth solutions, because it's... a whole thing. It's best maintained outside of the standard distribution, because there's so many ways to approach it. The caddy-oidc plugin you linked was written for Caddy v1, so it's no longer compatible. The most complete auth plugin for Caddy v2 is https://github.com/greenpau/caddy-security https://github.com/greenpau/caddy-security, and I think it probably does what you need.
  - barbazoo 5y ago
    Looks interesting. I'll check it out, thanks!!
mholt 5y ago
Nice, this is kind of why I made Project Conncept. It's a powerful TCP and UDP stream multiplexer based on Caddy: https://github.com/mholt/caddy-l4 https://github.com/mholt/caddy-l4 You can route raw TCP connections by using higher layer protocol matching logic like HTTP properties, SSH, TLS ClientHello info, and more, in composable routes that let you do nearly anything.
- rasengan 5y ago
  Thank you for this and Caddy.
  - mholt 5y ago
    You're welcome! Thanks for the nice comment. :) We have many contributors and several maintainers to thank as well.
- ignoramous 5y ago
  Neat. Kind of like a highly configurable https://github.com/inetaf/tcpproxy https://github.com/inetaf/tcpproxy > You can route raw TCP connections by using higher layer protocol matching logic like HTTP properties, SSH, TLS ClientHello info, and more, in composable routes that let you do nearly anything. How do you foresee such a setup handle QUIC? The encrypted connection-ids, 0RTT handshakes, and roaming client-ip and server-ips make it non trivial to proxy connections transparently.
  - mholt 5y ago
    Good question; I'm not really sure! Will need to look into it, or have people contribute some ideas. Feel free to start a discussion on the issue tracker if you're interested in this!
justsomehnguy 5y ago
Previous disc: https://news.ycombinator.com/item?id=31040667 https://news.ycombinator.com/item?id=31040667
klysm 5y ago
Well I guess I really have to learn IPv6 now
derefr 5y ago
> Meanwhile, my preferred language, Go, has a high-quality, memory-safe HTTPS server in the standard library that is well suited for direct exposure on the Internet. I know people do use Golang's http.Server for production use-cases. Does Google, though? Are services of Google customers ever actually directly talking to this Golang stack, without at least a minimal L7 WAF (as e.g. a default Nginx config does) in between? I ask because there are a number of weird connection latency, slowness, and "stuttering" problems I've experienced with services which I know do directly expose Golang servers — e.g. Docker Registry instances including Docker Hub; Minio instances; go-ethereum nodes; etc. — that I've never experienced with any Google service, or with any known non-Golang service. My hypothesis is that this is due to Golang's http.Server not having any upper limit on simultaneous connections (because just in CPU and memory terms, the Golang runtime can handle almost arbitrarily many), such that eventually the bottleneck actually becomes per-connection throughput, with clients becoming starved for space in the machine's network ring buffer; and because this is such an unusual bottleneck to have (usually it's only a thing with CDNs) — and because it causes no problems for the server, esp. if things like readiness checks are done through a separate internal NIC — the people running these servers don't even notice it's happening, and so lag far behind in horizontally scaling servers to spread out the demand for throughput. Or, to put that another way: the Golang http.Server isn't observable — exposing server-internal metrics — in the way that actual web servers like Nginx, or even web-app server frameworks like Jetty, are; and so it's very hard to know when things are silently going wrong for users, esp. in cases where the developers of a piece of software aren't themselves running it at scale and so never think to manually add observability for metrics that only become relevant at scale (which authors of generic web server software are usually aware of, but authors of application software usually aren't.) This leads me to think that, if Google themselves are using Golang services for anything at scale, and yet not rushing to implement such metrics into http.Server, then they must be observing these services in a very different way than we mere mortals do. Maybe calculating per-flow packet-wise QoS at the edge in their fancy LANai switches using historical statistical fingerprints of predictable flow patterns, or something.
- finnh 5y ago
  Seems like eBPF could be useful here as well, to get some external insight into per-connection counts and behavior.
twic 5y ago
Some lazy questions ... > Since IPv6 addresses are 128 bits long, but IPv4 addresses are only 32 bits, it's possible to embed IPv4 addresses in IPv6 addresses. snid embeds the client's IP address in the lower 32 bits of the source address which it uses to connect to the backend. How does this work? snid just makes up an IP address? What socket API calls do you make to do this? Just pick an address and bind, and the kernel is fine with that? And it all gets routed back and forth correctly? Do you have to configure this 64:ff9b:1::/48 prefix on the loopback interface? > Encrypted Client Hello doesn't actually encrypt the initial Client Hello message. It's still sent in the clear, but with a decoy SNI hostname. The actual Client Hello message, with the true SNI hostname, is encrypted and placed in an extension of the unencrypted Client Hello. To make Encrypted Client Hello work with snid, I just need to ensure that the decoy SNI hostname resolves to the IPv6 address of the backend server. snid will see this hostname and route the connection to the correct backend server, as usual. How does the decoy SNI hostname get chosen? This sounds like there needs to be a different decoy hostname for each backend service. Does that come from DNS somehow? The client doesn't just make it up at random?
- agwa 5y ago
  Great questions! > How does this work? snid just makes up an IP address? What socket API calls do you make to do this? Just pick an address and bind, and the kernel is fine with that? And it all gets routed back and forth correctly? Do you have to configure this 64:ff9b:1::/48 prefix on the loopback interface? First you have to set the IP_FREEBIND socket option, which allows binding to nonlocal/nonexistent IP address and then you call bind with whatever address you like. To ensure the packets get routed back properly, you need a local route for the 64:ff9b:1::/96 prefix, which can be added with: ip route add local 64:ff9b:1::/96 dev lo > How does the decoy SNI hostname get chosen? This sounds like there needs to be a different decoy hostname for each backend service. Does that come from DNS somehow? The client doesn't just make it up at random? The decoy hostname is specified in the ECHConfig struct[1], which is conveyed to the client via DNS in the HTTPS record[2]. It does indeed mean that each backend needs its own decoy hostname (which resolves to the IPv6 address of the backend). This means that ECH does not hide which backend is being connected to, but if a particular backend handles multiple hostnames, it can hide which of those hostnames the client is connecting to. [1] https://www.ietf.org/archive/id/draft-ietf-tls-esni-14.html#section-4 https://www.ietf.org/archive/id/draft-ietf-tls-esni-14.html#... [2] https://www.ietf.org/archive/id/draft-ietf-dnsop-svcb-https-08.html#section-10 https://www.ietf.org/archive/id/draft-ietf-dnsop-svcb-https-...
scott00 5y ago
I don't get how this works with ECH. Can anybody add some detail to what's in the article?
dcow 5y ago
If the decoy hostnames (as the author describes it) used in encrypted SNI are deterministic such that you can statically determine the real hostname the client wants, then what’s the point of encrypted SNI in the first place?
- agwa 5y ago
  ECH can't hide which backend the client wants, but if a particular backend handles multiple hostnames, it can hide which of those hostnames the client wants. I went into further detail here: https://news.ycombinator.com/item?id=31136335 https://news.ycombinator.com/item?id=31136335
peter_retief 5y ago
Great idea, I have been thinking of doing something similar but am struggling to get ISP's to support Ipv6. Some actually block Ipv6.
jraph 5y ago
I have a server that hosts several websites. I wanted some of them to be installed in a separate (systemd) container (because they belong to the same organization). I use nginx's ssl_preread module to proxy https requests to the container or to another port depending on the SNI. This is what snid does in the article if I understood correctly (without the DNS lookup because I don't need it, but it is able to do it too). It works well and it's good that the nginx at the front does not need to have the SSL certificates. In this setup, Nginx does not need to decode anything, it just does a pass-through, so this is quite light. It is also way simpler to setup than an actual HTTP reverse proxy.
agwa 5y ago
Since there are several questions about Encrypted Client Hello (ECH), and I kind of hand waved that section, I thought an example might be useful. Let's say the system is running two web server daemons: a multi-tenant blog hosting platform listening on 2001:db8::1, and a multi-tenant bug tracker listening on 2001:db8::2. snid is on 192.0.2.1. Your DNS records would look like this: blogs.example.com. A 192.0.2.1 blogs.example.com. AAAA 2001:db8::1 bugs.example.com. A 192.0.2.1 bugs.example.com. AAAA 2001:db8::2 The various tenants would be CNAMEd to one of these hostnames like: blog.domain1.example. CNAME blogs.example.com. bugs.domain2.example. CNAME bugs.example.com. The "decoy" hostnames (the "public_name" in ECH parlance) would be blogs.example.com or bugs.example.com. Thus, ECH would hide which tenant the client is connecting to, but would not hide the service. Note that if the client were connecting over IPv6, an eavesdropper would be able to determine the service anyways by looking at the destination IP address, which is unencrypted.
- gz5 5y ago
  Nice solution. You can got a step further if you have the need - your eavesdropper or malicious observer problem can be addressed by launching the network connections from inside the process space of your app, e.g. for golang: https://github.com/openziti/sdk-golang https://github.com/openziti/sdk-golang Similarly, this eliminates the IP address dependencies. Sample (Java in this case - see GitHub above for various language options): https://blogs.oracle.com/javamagazine/post/java-zero-trust-openziti https://blogs.oracle.com/javamagazine/post/java-zero-trust-o...
- cryptonector 5y ago
  Note that if you wanted to provide privacy protection for the encrypted SNI, you'd have to have a single external-facing IPv6 address too, not just IPv4. Then, of course, you couldn't steal enough bits to represent the client address in the source address presented to the backend. Tricky. You might have to implement the PROXY protocol after all. Another option might be to do something like identd for established TCP connections, and interpose on getpeername() on the backend side to call the identd at the proxy.
- bscphil 5y ago
  Would it be accurate to summarize this way? TLS + ECH encrypts all message content, including the client hello with the hostname, but it does not encrypt a specific message (an IP address + service identifier combination) that uniquely specifies which program will terminate the TLS connection. If you want information to be private about the host name you're connecting to, you need not only a single public IP address for many hosts, but also for that server to terminate TLS for all of those hosts.
zokier 5y ago
Clever. I think I'd prefer to do TLS termination at the proxy though, something akin to stunnel. Of course snid could be used together with stunnel, but I think it would lose the O(1) configuration then. Just terminating tls and not touching the http content would still avoid any of those http parsing issues mentioned.
- agwa 5y ago
  An earlier version did the TLS termination in the proxy. It's true you can avoid the HTTP parsing issues, but you lose the ability to do client certs or have backend-specific cipher/TLS version requirements. Also, I really like it that IPv6 clients can connect directly to the backend, bypassing any proxies.
thayne 5y ago
> To make Encrypted Client Hello work with snid, I just need to ensure that the decoy SNI hostname resolves to the IPv6 address of the backend server. Doesn't that sort of defeat the purpose of ECH?
GauntletWizard 5y ago
I'm a huge fan of this approach, but I would also combine it almost equally with standard http reverse proxies; there's a lot you can gain from having a proxy that can understand paths,buffer requests, etc.

138 points by petercooper 5y ago

This is such a neat solution. As part of my effort to single-stack-v6 and minimum-effort-v4, for my self hosted services and projects, I've always wanted a way to avoid reverse proxy configuration for each hosted app; v6 can directly listen to a new address.

anderspitman 5y ago

This is great. I think SNI is currently one of the most pragmatic tools to work around ipv4 exhaustion. I like the approach described here, but in practice I prefer the convenience of having a reverse proxy to automatically handle TLS certs for me. That said, libraries like certmagic are making it more feasible for every app to manage its own certs.

ignoramous 5y ago
> I think SNI is currently one of the most pragmatic tools to work around ipv4 exhaustion. Pretty much: https://research.cloudflare.com/publications/Fayed2021/ https://research.cloudflare.com/publications/Fayed2021/ See also, DNS SVC (A/B) records (pseudo NAT at DNS layer); but not many deployments use it or understand it. Note that, SNI as a routing replacement works for TCP nicely without much (user-space) complication. With QUIC, transparently proxying connections isn't all that straight forward.
- anderspitman 5y ago
  Thanks for the link. What exactly are the issues with QUIC? I've wondered if it's possible to do SNI routing with UDP but haven't looked into it yet.
  - ignoramous 5y ago
    QUIC hides and encrypts everything it possibly can, and that includes server and client identities (IPs are inconsequential to a QUIC session which is instead maintained through connection-id(s) exchanged under TLS encryption or is obfuscated away; that is, there is no way for a middleware to analyse a QUIC session / flow without actually MiTMing TLS). More: https://lwn.net/Articles/745590/ https://lwn.net/Articles/745590/

barbazoo 5y ago

Wow this is neat. > I have had it with standalone web servers: they're all over-complicated and I always end up with an awkward bifurcation of logic between my app's code and the web server's config Personally I've grown really fond of letting nginx terminate TLS and proxy to the web app. It's a clean separation of concerns, not very complicated and upgrading the cert is easy (certbot).

francislavoie 5y ago
FWIW, Caddy can replace nginx and certbot for this purpose, with less config and more robust ACME. Let me know if you need me to clarify, if you have any concerns.
- iso1210 5y ago
  Caddy looks interesting, I currently use apache to proxy a few hundered sites and it works well enough, some are protected by client certificates, others by oidc, all then pass the authenticated user to the downstream server in a header, job done. I've managed to do this with openresty (nginx not supporting oidc out of the box), but it doesn't fill me with confidence, I guess it's all the lua. A quick glance at caddy shows it likewise doesn't support oidc integration out of the box, but instead I have to use another module that's no longer maintained ( https://github.com/thspinto/caddy-oidc https://github.com/thspinto/caddy-oidc )
  - francislavoie 5y ago
    Yeah, we defer to plugins to provide auth solutions, because it's... a whole thing. It's best maintained outside of the standard distribution, because there's so many ways to approach it. The caddy-oidc plugin you linked was written for Caddy v1, so it's no longer compatible. The most complete auth plugin for Caddy v2 is https://github.com/greenpau/caddy-security https://github.com/greenpau/caddy-security, and I think it probably does what you need.
- barbazoo 5y ago
  Looks interesting. I'll check it out, thanks!!

mholt 5y ago

Nice, this is kind of why I made Project Conncept. It's a powerful TCP and UDP stream multiplexer based on Caddy: https://github.com/mholt/caddy-l4 https://github.com/mholt/caddy-l4 You can route raw TCP connections by using higher layer protocol matching logic like HTTP properties, SSH, TLS ClientHello info, and more, in composable routes that let you do nearly anything.

rasengan 5y ago
Thank you for this and Caddy.
- mholt 5y ago
  You're welcome! Thanks for the nice comment. :) We have many contributors and several maintainers to thank as well.
ignoramous 5y ago
Neat. Kind of like a highly configurable https://github.com/inetaf/tcpproxy https://github.com/inetaf/tcpproxy > You can route raw TCP connections by using higher layer protocol matching logic like HTTP properties, SSH, TLS ClientHello info, and more, in composable routes that let you do nearly anything. How do you foresee such a setup handle QUIC? The encrypted connection-ids, 0RTT handshakes, and roaming client-ip and server-ips make it non trivial to proxy connections transparently.
- mholt 5y ago
  Good question; I'm not really sure! Will need to look into it, or have people contribute some ideas. Feel free to start a discussion on the issue tracker if you're interested in this!

justsomehnguy 5y ago

Previous disc: https://news.ycombinator.com/item?id=31040667 https://news.ycombinator.com/item?id=31040667

klysm 5y ago

Well I guess I really have to learn IPv6 now

derefr 5y ago

> Meanwhile, my preferred language, Go, has a high-quality, memory-safe HTTPS server in the standard library that is well suited for direct exposure on the Internet. I know people do use Golang's http.Server for production use-cases. Does Google, though? Are services of Google customers ever actually directly talking to this Golang stack, without at least a minimal L7 WAF (as e.g. a default Nginx config does) in between? I ask because there are a number of weird connection latency, slowness, and "stuttering" problems I've experienced with services which I know do directly expose Golang servers — e.g. Docker Registry instances including Docker Hub; Minio instances; go-ethereum nodes; etc. — that I've never experienced with any Google service, or with any known non-Golang service. My hypothesis is that this is due to Golang's http.Server not having any upper limit on simultaneous connections (because just in CPU and memory terms, the Golang runtime can handle almost arbitrarily many), such that eventually the bottleneck actually becomes per-connection throughput, with clients becoming starved for space in the machine's network ring buffer; and because this is such an unusual bottleneck to have (usually it's only a thing with CDNs) — and because it causes no problems for the server, esp. if things like readiness checks are done through a separate internal NIC — the people running these servers don't even notice it's happening, and so lag far behind in horizontally scaling servers to spread out the demand for throughput. Or, to put that another way: the Golang http.Server isn't observable — exposing server-internal metrics — in the way that actual web servers like Nginx, or even web-app server frameworks like Jetty, are; and so it's very hard to know when things are silently going wrong for users, esp. in cases where the developers of a piece of software aren't themselves running it at scale and so never think to manually add observability for metrics that only become relevant at scale (which authors of generic web server software are usually aware of, but authors of application software usually aren't.) This leads me to think that, if Google themselves are using Golang services for anything at scale, and yet not rushing to implement such metrics into http.Server, then they must be observing these services in a very different way than we mere mortals do. Maybe calculating per-flow packet-wise QoS at the edge in their fancy LANai switches using historical statistical fingerprints of predictable flow patterns, or something.

finnh 5y ago
Seems like eBPF could be useful here as well, to get some external insight into per-connection counts and behavior.

twic 5y ago

Some lazy questions ... > Since IPv6 addresses are 128 bits long, but IPv4 addresses are only 32 bits, it's possible to embed IPv4 addresses in IPv6 addresses. snid embeds the client's IP address in the lower 32 bits of the source address which it uses to connect to the backend. How does this work? snid just makes up an IP address? What socket API calls do you make to do this? Just pick an address and bind, and the kernel is fine with that? And it all gets routed back and forth correctly? Do you have to configure this 64:ff9b:1::/48 prefix on the loopback interface? > Encrypted Client Hello doesn't actually encrypt the initial Client Hello message. It's still sent in the clear, but with a decoy SNI hostname. The actual Client Hello message, with the true SNI hostname, is encrypted and placed in an extension of the unencrypted Client Hello. To make Encrypted Client Hello work with snid, I just need to ensure that the decoy SNI hostname resolves to the IPv6 address of the backend server. snid will see this hostname and route the connection to the correct backend server, as usual. How does the decoy SNI hostname get chosen? This sounds like there needs to be a different decoy hostname for each backend service. Does that come from DNS somehow? The client doesn't just make it up at random?

agwa 5y ago
Great questions! > How does this work? snid just makes up an IP address? What socket API calls do you make to do this? Just pick an address and bind, and the kernel is fine with that? And it all gets routed back and forth correctly? Do you have to configure this 64:ff9b:1::/48 prefix on the loopback interface? First you have to set the IP_FREEBIND socket option, which allows binding to nonlocal/nonexistent IP address and then you call bind with whatever address you like. To ensure the packets get routed back properly, you need a local route for the 64:ff9b:1::/96 prefix, which can be added with: ip route add local 64:ff9b:1::/96 dev lo > How does the decoy SNI hostname get chosen? This sounds like there needs to be a different decoy hostname for each backend service. Does that come from DNS somehow? The client doesn't just make it up at random? The decoy hostname is specified in the ECHConfig struct[1], which is conveyed to the client via DNS in the HTTPS record[2]. It does indeed mean that each backend needs its own decoy hostname (which resolves to the IPv6 address of the backend). This means that ECH does not hide which backend is being connected to, but if a particular backend handles multiple hostnames, it can hide which of those hostnames the client is connecting to. [1] https://www.ietf.org/archive/id/draft-ietf-tls-esni-14.html#section-4 https://www.ietf.org/archive/id/draft-ietf-tls-esni-14.html#... [2] https://www.ietf.org/archive/id/draft-ietf-dnsop-svcb-https-08.html#section-10 https://www.ietf.org/archive/id/draft-ietf-dnsop-svcb-https-...

scott00 5y ago

I don't get how this works with ECH. Can anybody add some detail to what's in the article?

dcow 5y ago

If the decoy hostnames (as the author describes it) used in encrypted SNI are deterministic such that you can statically determine the real hostname the client wants, then what’s the point of encrypted SNI in the first place?

agwa 5y ago
ECH can't hide which backend the client wants, but if a particular backend handles multiple hostnames, it can hide which of those hostnames the client wants. I went into further detail here: https://news.ycombinator.com/item?id=31136335 https://news.ycombinator.com/item?id=31136335

peter_retief 5y ago

Great idea, I have been thinking of doing something similar but am struggling to get ISP's to support Ipv6. Some actually block Ipv6.

jraph 5y ago

I have a server that hosts several websites. I wanted some of them to be installed in a separate (systemd) container (because they belong to the same organization). I use nginx's ssl_preread module to proxy https requests to the container or to another port depending on the SNI. This is what snid does in the article if I understood correctly (without the DNS lookup because I don't need it, but it is able to do it too). It works well and it's good that the nginx at the front does not need to have the SSL certificates. In this setup, Nginx does not need to decode anything, it just does a pass-through, so this is quite light. It is also way simpler to setup than an actual HTTP reverse proxy.

agwa 5y ago

Since there are several questions about Encrypted Client Hello (ECH), and I kind of hand waved that section, I thought an example might be useful. Let's say the system is running two web server daemons: a multi-tenant blog hosting platform listening on 2001:db8::1, and a multi-tenant bug tracker listening on 2001:db8::2. snid is on 192.0.2.1. Your DNS records would look like this: blogs.example.com. A 192.0.2.1 blogs.example.com. AAAA 2001:db8::1 bugs.example.com. A 192.0.2.1 bugs.example.com. AAAA 2001:db8::2 The various tenants would be CNAMEd to one of these hostnames like: blog.domain1.example. CNAME blogs.example.com. bugs.domain2.example. CNAME bugs.example.com. The "decoy" hostnames (the "public_name" in ECH parlance) would be blogs.example.com or bugs.example.com. Thus, ECH would hide which tenant the client is connecting to, but would not hide the service. Note that if the client were connecting over IPv6, an eavesdropper would be able to determine the service anyways by looking at the destination IP address, which is unencrypted.

gz5 5y ago
Nice solution. You can got a step further if you have the need - your eavesdropper or malicious observer problem can be addressed by launching the network connections from inside the process space of your app, e.g. for golang: https://github.com/openziti/sdk-golang https://github.com/openziti/sdk-golang Similarly, this eliminates the IP address dependencies. Sample (Java in this case - see GitHub above for various language options): https://blogs.oracle.com/javamagazine/post/java-zero-trust-openziti https://blogs.oracle.com/javamagazine/post/java-zero-trust-o...
cryptonector 5y ago
Note that if you wanted to provide privacy protection for the encrypted SNI, you'd have to have a single external-facing IPv6 address too, not just IPv4. Then, of course, you couldn't steal enough bits to represent the client address in the source address presented to the backend. Tricky. You might have to implement the PROXY protocol after all. Another option might be to do something like identd for established TCP connections, and interpose on getpeername() on the backend side to call the identd at the proxy.
bscphil 5y ago
Would it be accurate to summarize this way? TLS + ECH encrypts all message content, including the client hello with the hostname, but it does not encrypt a specific message (an IP address + service identifier combination) that uniquely specifies which program will terminate the TLS connection. If you want information to be private about the host name you're connecting to, you need not only a single public IP address for many hosts, but also for that server to terminate TLS for all of those hosts.

zokier 5y ago

Clever. I think I'd prefer to do TLS termination at the proxy though, something akin to stunnel. Of course snid could be used together with stunnel, but I think it would lose the O(1) configuration then. Just terminating tls and not touching the http content would still avoid any of those http parsing issues mentioned.

agwa 5y ago
An earlier version did the TLS termination in the proxy. It's true you can avoid the HTTP parsing issues, but you lose the ability to do client certs or have backend-specific cipher/TLS version requirements. Also, I really like it that IPv6 clients can connect directly to the backend, bypassing any proxies.

thayne 5y ago

> To make Encrypted Client Hello work with snid, I just need to ensure that the decoy SNI hostname resolves to the IPv6 address of the backend server. Doesn't that sort of defeat the purpose of ECH?

GauntletWizard 5y ago

I'm a huge fan of this approach, but I would also combine it almost equally with standard http reverse proxies; there's a lot you can gain from having a proxy that can understand paths,buffer requests, etc.