4 ms·

Just be mindful that any certs you issue in this way will be public information[1] so make sure the domain names don't give away any interesting facts about you

Just be mindful that any certs you issue in this way will be public information[1] so make sure the domain names don't give away any interesting facts about your infrastructure or future product ideas. I did this at my last job as well and I can still see them renewing them, including an unfortunate wildcard cert which wasn't me. [1] https://crt.sh/ https://crt.sh/

Helmut10001 2y ago
Just use wildcard certs and internal subdomains remain internal information.
- qmarchi 2y ago
  There's a larger risk that if someone breaches a system with a wildcard cert, then you can end up with them being able to impersonate _every_ part of your domain, not just the one application.
  - politelemon 2y ago
    It's the opposite - there is a risk, but not a larger risk. Environment traversal is easier through a certificate transparency log, there is almost zero work to do. Through a wildcard compromise, the environment is not immediately visible. It's much safer to do wildcard for certs for internal use.
    - ixfo 2y ago
      Environment visibility is easy to get. If you pwn a box which has foo.internal, you can now impersonate foo.internal. If you pwn a box which has *.internal, you can now impersonate super-secret.internal and everything else, and now you're a DNS change away from MITM across an entire estate. Security by obscurity while making the actual security of endpoints weaker is not an argument in favour of wildcards...
  - eru 2y ago
    Can't you have a limited wildcard? Something like *.for-testing-only.company.com?
    - kevincox 2y ago
      Yes, but then you are putting more information into the publically logged certificate. So it is a tradeoff between scope of certificate and data leak. I guess you can use a pattern like {human name}.{random}.internal but then you lose memoribility.
      - 8organicbits 2y ago
        I've considered building tools to manage decoy certificates, like it would register mail.example.com if you didn't have a mail server, but I couldn't justify polluting the cert transparency logs.
      - lacerrr 2y ago
        Made up problem, that approach is fine.
  - qwertox 2y ago
    I issue a wildcard cert for *.something.example.com. All subdomains which are meant for public consumption are at the first level, like www.example.com or blog.example.com, and the ones I use internally (or even privately accessible on the internet, like xmpp.something.example.com) are not up for discovery, as no public records exist. Everything at *.something.example.com, if it is supposed to be privately accessible on the internet, is resolved by a custom DNS server which does not respond to `ANY`-requests and logs every request. You'd need to know which subdomains exist. something.example.com has an `NS`-record entry with the domain name which points to the IP of that custom DNS server (ns.example.com). The intranet also has a custom DNS server which then serves the IPs of the subdomains which are only meant for internal consumption.
    - brewmarche 2y ago
      This is the DNS setup I’d have in mind as well. Regarding the certificates, if you don’t want to set up stuff on clients manually, the only drawback is the use of a wildcard certificate (which when compromised can be used to hijack everything under something.example.com). An intermediate CA with name constraints (can only sign certificates with names under something.example.com) sounds like a better solution if you deem the wildcard certificate too risky. Not sure which CA can issue it (letsencrypt is probably out) and how well supported it is
      - qwertox 2y ago
        I'm "ok" with that risk. It's less risky than other solutions, and there's also the issue that hijacked.something.example.com needs to be resolved by the internal DNS server. All of this would most likely need to be an inside job with some relatively big criminal energy. At that level you'd probably also have other attack vectors which you could consider.
        Helmut10001 2y ago
        This is also my thinking.. if someone compromises your VM that is responsible for retrieving wildcard certs from let's encrypt, then you're probably busted anyway. Such a machine would usually sit at the center of infrastructure, with limited need to be connected to from other machines.
- ivankuz 2y ago
  A fun tale about wildcard certificates for internal subdomains: The browser will gladly reuse an http2 connection with a resolved IP address. If you happen to have many subdomains pointing to a single ingress / reverse proxy that returns the same certificate for different Host headers, you can very well end up in a situation where the traffic will get messed up between services. To add to that - debugging that stuff becomes kind of wild, as it will keep reusing connections between browser windows (and maybe even different Chromium browsers) I might be messing up technical details, as it's been a long time since I've debugged some grpc Kubernetes mess. All I wanted to say is, that having an exact certificate instead of a wildcard is also a good way to ensure your traffic goes to the correct place internally.
  - nightpool 2y ago
    Sounds like you need to get better reverse proxies...? Making your site traffic RELY on the fact that you're using different certificates for different hosts sounds fragile as hell and it's just setting yourself up for even more pain in the future
    - ivankuz 2y ago
      It was the latest nginx at the time. I actually found a rather obscure issue on Github that touches on this problem, for those who are curious: https://github.com/kubernetes/ingress-nginx/issues/1681#issuecomment-543651358 https://github.com/kubernetes/ingress-nginx/issues/1681#issu... > We discovered a related issue where we have multiple ssl-passthrough upstreams that only use different hostnames. [...] nginx-ingress does not inspect the connection after the initial handshake - no matter if the HOST changes. That was 5-ish years ago though. I hope there are better ways than the cert hack now.
      - ploxiln 2y ago
        That's a misunderstanding in your use of this ingress-controller "ssl-passthrough" feature. > This feature is implemented by intercepting all traffic on the configured HTTPS port (default: 443) and handing it over to a local TCP proxy. This bypasses NGINX completely and introduces a non-negligible performance penalty. > SSL Passthrough leverages SNI and reads the virtual domain from the TLS negotiation So if you want multiple subdomains handled by the same ip address and using the same wildcard TLS cert, and chrome re-uses the connection for a different subdomain, nginx needs to handle/parse the http, and http-proxy to the backends. In this ssl-passthrough mode it can only look at the SNI host in the initial TLS handshake, and that's it, it can't look at the contents of the traffic. This is a limitation of http/tls/tcp, not of nginx.
zikduruqe 2y ago
I use https://github.com/FiloSottile/mkcert https://github.com/FiloSottile/mkcert for my internal stuff.
moontear 2y ago
I wish there was a way to remove public information such as this. Just like historical website ownership records. Maybe interesting for research purposes, but there is so much stuff in public records I don't want everyone to have access to. Should have thought about that before creating public records - but one may not be aware of all the ramifications of e.g. just creating an SSL cert with letsencrypt or registering a random domain name without privacy extensions.

Helmut10001 2y ago

Just use wildcard certs and internal subdomains remain internal information.

qmarchi 2y ago
There's a larger risk that if someone breaches a system with a wildcard cert, then you can end up with them being able to impersonate _every_ part of your domain, not just the one application.
- politelemon 2y ago
  It's the opposite - there is a risk, but not a larger risk. Environment traversal is easier through a certificate transparency log, there is almost zero work to do. Through a wildcard compromise, the environment is not immediately visible. It's much safer to do wildcard for certs for internal use.
  - ixfo 2y ago
    Environment visibility is easy to get. If you pwn a box which has foo.internal, you can now impersonate foo.internal. If you pwn a box which has *.internal, you can now impersonate super-secret.internal and everything else, and now you're a DNS change away from MITM across an entire estate. Security by obscurity while making the actual security of endpoints weaker is not an argument in favour of wildcards...
- eru 2y ago
  Can't you have a limited wildcard? Something like *.for-testing-only.company.com?
  - kevincox 2y ago
    Yes, but then you are putting more information into the publically logged certificate. So it is a tradeoff between scope of certificate and data leak. I guess you can use a pattern like {human name}.{random}.internal but then you lose memoribility.
    - 8organicbits 2y ago
      I've considered building tools to manage decoy certificates, like it would register mail.example.com if you didn't have a mail server, but I couldn't justify polluting the cert transparency logs.
    - lacerrr 2y ago
      Made up problem, that approach is fine.
- qwertox 2y ago
  I issue a wildcard cert for *.something.example.com. All subdomains which are meant for public consumption are at the first level, like www.example.com or blog.example.com, and the ones I use internally (or even privately accessible on the internet, like xmpp.something.example.com) are not up for discovery, as no public records exist. Everything at *.something.example.com, if it is supposed to be privately accessible on the internet, is resolved by a custom DNS server which does not respond to `ANY`-requests and logs every request. You'd need to know which subdomains exist. something.example.com has an `NS`-record entry with the domain name which points to the IP of that custom DNS server (ns.example.com). The intranet also has a custom DNS server which then serves the IPs of the subdomains which are only meant for internal consumption.
  - brewmarche 2y ago
    This is the DNS setup I’d have in mind as well. Regarding the certificates, if you don’t want to set up stuff on clients manually, the only drawback is the use of a wildcard certificate (which when compromised can be used to hijack everything under something.example.com). An intermediate CA with name constraints (can only sign certificates with names under something.example.com) sounds like a better solution if you deem the wildcard certificate too risky. Not sure which CA can issue it (letsencrypt is probably out) and how well supported it is
    - qwertox 2y ago
      I'm "ok" with that risk. It's less risky than other solutions, and there's also the issue that hijacked.something.example.com needs to be resolved by the internal DNS server. All of this would most likely need to be an inside job with some relatively big criminal energy. At that level you'd probably also have other attack vectors which you could consider.
      - Helmut10001 2y ago
        This is also my thinking.. if someone compromises your VM that is responsible for retrieving wildcard certs from let's encrypt, then you're probably busted anyway. Such a machine would usually sit at the center of infrastructure, with limited need to be connected to from other machines.
ivankuz 2y ago
A fun tale about wildcard certificates for internal subdomains: The browser will gladly reuse an http2 connection with a resolved IP address. If you happen to have many subdomains pointing to a single ingress / reverse proxy that returns the same certificate for different Host headers, you can very well end up in a situation where the traffic will get messed up between services. To add to that - debugging that stuff becomes kind of wild, as it will keep reusing connections between browser windows (and maybe even different Chromium browsers) I might be messing up technical details, as it's been a long time since I've debugged some grpc Kubernetes mess. All I wanted to say is, that having an exact certificate instead of a wildcard is also a good way to ensure your traffic goes to the correct place internally.
- nightpool 2y ago
  Sounds like you need to get better reverse proxies...? Making your site traffic RELY on the fact that you're using different certificates for different hosts sounds fragile as hell and it's just setting yourself up for even more pain in the future
  - ivankuz 2y ago
    It was the latest nginx at the time. I actually found a rather obscure issue on Github that touches on this problem, for those who are curious: https://github.com/kubernetes/ingress-nginx/issues/1681#issuecomment-543651358 https://github.com/kubernetes/ingress-nginx/issues/1681#issu... > We discovered a related issue where we have multiple ssl-passthrough upstreams that only use different hostnames. [...] nginx-ingress does not inspect the connection after the initial handshake - no matter if the HOST changes. That was 5-ish years ago though. I hope there are better ways than the cert hack now.
    - ploxiln 2y ago
      That's a misunderstanding in your use of this ingress-controller "ssl-passthrough" feature. > This feature is implemented by intercepting all traffic on the configured HTTPS port (default: 443) and handing it over to a local TCP proxy. This bypasses NGINX completely and introduces a non-negligible performance penalty. > SSL Passthrough leverages SNI and reads the virtual domain from the TLS negotiation So if you want multiple subdomains handled by the same ip address and using the same wildcard TLS cert, and chrome re-uses the connection for a different subdomain, nginx needs to handle/parse the http, and http-proxy to the backends. In this ssl-passthrough mode it can only look at the SNI host in the initial TLS handshake, and that's it, it can't look at the contents of the traffic. This is a limitation of http/tls/tcp, not of nginx.

zikduruqe 2y ago

I use https://github.com/FiloSottile/mkcert https://github.com/FiloSottile/mkcert for my internal stuff.

moontear 2y ago

I wish there was a way to remove public information such as this. Just like historical website ownership records. Maybe interesting for research purposes, but there is so much stuff in public records I don't want everyone to have access to. Should have thought about that before creating public records - but one may not be aware of all the ramifications of e.g. just creating an SSL cert with letsencrypt or registering a random domain name without privacy extensions.