5 ms·

It's interesting almost no web server provides an easy way to deal with multi-tenant multi-domain architectures in a good way that includes automatic SSL. Cadd

It's interesting almost no web server provides an easy way to deal with multi-tenant multi-domain architectures in a good way that includes automatic SSL. Caddy is the closest, but still not near enough. There is this small segment of the market that we operate in that requires thousands of TLS connected domains to be hosted behind a dynamic backend. It's services like Tumblr, Wordpress.com, or any other hosting service where you can get a "custom domain" to point to your own blog or site. NGINX - No. Apache - Nope. Caddy - Can do (but need lots of workarounds) Envoy - Nope. Everyone focuses on a few hand-coded domains and no automatic TLS. Maybe this part of the market is too small anyway. Sigh.

simplyinfinity 6y ago
We are using OpenResty with lua-auto-ssl for exactly this purpose, and it works like a charm.
mholt 6y ago
Several companies use Caddy for exactly this purpose. Fathom Analytics for example uses it for their custom domains feature. Caddy can even reactively provision certs during TLS handshakes. It's a native feature. Why does it require lots of workarounds?
- bschwindHN 6y ago
  Yeah I'm not sure what they're getting at, I've used Caddy as well for similar "custom domain" features, it was super easy. Thanks for creating it!
- eric4smith 6y ago
  Yes. Caddy is what we use, since not much else can do it as easily as Caddy can. And it's our go-to tool for several projects that require custom domains. And we really, really, appreciate it! I'm just saying that it's not something that is documented well or purpose built for that scenario.
- sladey 6y ago
  Is there any mature integration to achieve this with Kubernetes?
  - mholt 6y ago
    WIP: https://github.com/caddyserver/ingress/ https://github.com/caddyserver/ingress/
e12e 6y ago
Traefik makes it fairly easy (inasmuch as it makes anything easy). But it's just a proxy, not a web server.
- eric4smith 6y ago
  Traefik can work yes - documentation is really terrible though for anything approaching that use case. We had to really futz around with it and eventually went back to Caddy. Our use case is several thousand client domains that just proxy to some backends.
  - e12e 6y ago
    Did you move to caddy 2? Our initial use-case was ingress for docker swarm - after a fooray into k8s with the "traditional" nginx ingress with its rather hackish let's encrypt contraption. I briefly looked at caddy 2 - but wasn't able to find any out-of-the box tricks for listening to docker messages and dynamically configure sites in a sane way. Do you use custom code and configure caddy via the api?
    - mholt 6y ago
      > I briefly looked at caddy 2 - but wasn't able to find any out-of-the box tricks for listening to docker messages and dynamically configure sites in a sane way. Like this? (Am not a Docker user, but I know this is an insanely popular solution) https://github.com/lucaslorentz/caddy-docker-proxy https://github.com/lucaslorentz/caddy-docker-proxy There's also a WIP ingress controller: https://github.com/caddyserver/ingress/ https://github.com/caddyserver/ingress/
toast0 6y ago
I would think the way to do this would be to run a separate TLS daemon that handles the certificates (including acme challenges, presumably) and then pass the socket to your http server, either by proxying it (preferably to a unix socket), or like actually pass the FD with the session keys. I don't think hitch (formerly stud) supports acme challenges, but that's where I'd start.
xenonite 6y ago
Lighttpd seems to have solutions for that. Did you have a look at it? https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_SSL#SSL-on-multiple-domains https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_SSL... > A traditional problem with SSL in combination with name based virtual hosting has been that the SSL connection setup happens before the HTTP request. So at the moment lighttpd needs to send its certificate to the client, it does not know yet which domain the client will be requesting. This means it can only supply the default certificate (and use the corresponding key for encryption) and effectively, SSL can only be enabled for that default domain. There are a number of solutions to this problem, with varying levels of support by clients. Then, the best approach seems to be the following: > Server Name Indication (SNI) is a TLS extension to the TLS handshake that allows the client to send the name of the host it wants to contact. The server can then use this information to select the correct certificate.
elithrar 6y ago
You can definitely “lazy load” TLS certs into Envoy. The SDS (Secrets Discovery Service) supports this, and is touched on in TFA: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ssl#secret-discovery-service-sds https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv... You provide a gRPC service that can return the keypair needed for any host, with host config also being dynamic. https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret#sds-configuration https://www.envoyproxy.io/docs/envoy/latest/configuration/se...
andrenth 6y ago
Apache can do automatic TLS with mod_md.

simplyinfinity 6y ago

We are using OpenResty with lua-auto-ssl for exactly this purpose, and it works like a charm.

mholt 6y ago

Several companies use Caddy for exactly this purpose. Fathom Analytics for example uses it for their custom domains feature. Caddy can even reactively provision certs during TLS handshakes. It's a native feature. Why does it require lots of workarounds?

bschwindHN 6y ago
Yeah I'm not sure what they're getting at, I've used Caddy as well for similar "custom domain" features, it was super easy. Thanks for creating it!
eric4smith 6y ago
Yes. Caddy is what we use, since not much else can do it as easily as Caddy can. And it's our go-to tool for several projects that require custom domains. And we really, really, appreciate it! I'm just saying that it's not something that is documented well or purpose built for that scenario.
sladey 6y ago
Is there any mature integration to achieve this with Kubernetes?
- mholt 6y ago
  WIP: https://github.com/caddyserver/ingress/ https://github.com/caddyserver/ingress/

e12e 6y ago

Traefik makes it fairly easy (inasmuch as it makes anything easy). But it's just a proxy, not a web server.

eric4smith 6y ago
Traefik can work yes - documentation is really terrible though for anything approaching that use case. We had to really futz around with it and eventually went back to Caddy. Our use case is several thousand client domains that just proxy to some backends.
- e12e 6y ago
  Did you move to caddy 2? Our initial use-case was ingress for docker swarm - after a fooray into k8s with the "traditional" nginx ingress with its rather hackish let's encrypt contraption. I briefly looked at caddy 2 - but wasn't able to find any out-of-the box tricks for listening to docker messages and dynamically configure sites in a sane way. Do you use custom code and configure caddy via the api?
  - mholt 6y ago
    > I briefly looked at caddy 2 - but wasn't able to find any out-of-the box tricks for listening to docker messages and dynamically configure sites in a sane way. Like this? (Am not a Docker user, but I know this is an insanely popular solution) https://github.com/lucaslorentz/caddy-docker-proxy https://github.com/lucaslorentz/caddy-docker-proxy There's also a WIP ingress controller: https://github.com/caddyserver/ingress/ https://github.com/caddyserver/ingress/

toast0 6y ago

I would think the way to do this would be to run a separate TLS daemon that handles the certificates (including acme challenges, presumably) and then pass the socket to your http server, either by proxying it (preferably to a unix socket), or like actually pass the FD with the session keys. I don't think hitch (formerly stud) supports acme challenges, but that's where I'd start.

xenonite 6y ago

Lighttpd seems to have solutions for that. Did you have a look at it? https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_SSL#SSL-on-multiple-domains https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_SSL... > A traditional problem with SSL in combination with name based virtual hosting has been that the SSL connection setup happens before the HTTP request. So at the moment lighttpd needs to send its certificate to the client, it does not know yet which domain the client will be requesting. This means it can only supply the default certificate (and use the corresponding key for encryption) and effectively, SSL can only be enabled for that default domain. There are a number of solutions to this problem, with varying levels of support by clients. Then, the best approach seems to be the following: > Server Name Indication (SNI) is a TLS extension to the TLS handshake that allows the client to send the name of the host it wants to contact. The server can then use this information to select the correct certificate.

elithrar 6y ago

You can definitely “lazy load” TLS certs into Envoy. The SDS (Secrets Discovery Service) supports this, and is touched on in TFA: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ssl#secret-discovery-service-sds https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv... You provide a gRPC service that can return the keypair needed for any host, with host config also being dynamic. https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret#sds-configuration https://www.envoyproxy.io/docs/envoy/latest/configuration/se...

andrenth 6y ago

Apache can do automatic TLS with mod_md.