12 ms·

Automatic SSL Certificates for internal IP's for home k8 setup using LetsEncrypt

156 points by gcds 6y ago

lgbr 6y ago
Cert-manager has great support for a number of providers[0] including AWS, CloudFlare, Google Cloud, and Azure. I recommend this not just for internal IP setups, for actually for all setups, since DNS verification is more robust than HTTP verification, particularly if you have issues with load balancers, or if Let's Encrypt decides to deprecate a protocol again [1]. [0] https://cert-manager.io/docs/configuration/acme/dns01/#supported-dns01-providers https://cert-manager.io/docs/configuration/acme/dns01/#suppo... [1] https://community.letsencrypt.org/t/upcoming-tls-sni-deprecation-in-certbot/76383 https://community.letsencrypt.org/t/upcoming-tls-sni-depreca...
- tialaramex 6y ago
  I like DNS challenges, but I don't see how it matters for deprecation of an ACME challenge type. The dns-01 challenge could just as easily for some reason need to be deprecated. The two likely reasons for such deprecation would apply just as well: 1. Updated Baseline Requirements or a programme policy requirement at any of the major root trust stores could forbid this challenge or require it to be substantially modified, obsoleting it in its current form. 2. The BRs don't change but Let's Encrypt finds they need to adjust this particular implementation in a non-compatible way so they deprecate the current challenge. It can be easier to do DNS challenges, but it can also be very rough, depending on all the moving parts in your system.
  - sleevi 6y ago
    Both of those are reasonable concerns, if all other factors were ignored. However, in practice, the DNS challenge (which demonstrates control over DNS) is greatly preferred over HTTP/TLS challenges (which demonstrate control over a single port). DNS is likely to be the only way to get a wildcard certificate, and HTTP/TLS will likely end up further restricted once SRVNames in certificates can be gracefully rolled out. As such, deploying the DNS based control is absolutely the best thing to do, and HTTP/TLS should be seen as legacy-compat fallbacks that may become more difficult in time. Either certificates become less scoped than “entire domain” (as they are today with dNSName SANs) or it becomes more difficult to use a single port to prove authorization for an entire domain.
    - sroussey 6y ago
      But can’t DNS queries be altered man in the middle style?
      - sleevi 6y ago
        I’m not sure your point? Any HTTP/ALPN request first begins with DNS, so if you’re trying to compare those, they all share the same base issue. In theory, this can be mitigated by DNSSEC, but that’s not relevant when comparing these validation methods. However, both the HTTP and ALPN methods only demonstrate control over a single port (or .well-known resource), while the DNS method demonstrates the full ability to alter any/all names.
        sroussey 6y ago
        Actually, I suppose DNS with DNSSEC or DNS over HTTPS would be better than any HTTP method.
- z3t4 6y ago
  Verification via DNS is not without issues. If you have more then one DNS server the verification record need to propagate to all servers. If you for example use anycast DNS you will run into issues. Letsencrypt uses Google name servers for lookup which is problematic because they do not behave, they will for example not try secondary dns servers if the first try fail, making the Letsencrypt verification also fail. And because of these issues and if you have many domains you will quickly reach Letsencrypt quota.
  - sleevi 6y ago
    Where have you seen Let’s Encrypt using Google’s servers? CAs are required to run full recursive resolvers, up to the root, and can’t just point at someone else’s DNS infrastructure. Which, if you think about it, is what you want: you don’t want the CA just trusting someone else is being honest, you want to go to the authoritative source.
  - radiowave 6y ago
    In which case it may be advisable to delegate the acme-challenge record to a different DNS provider, if doing so allows you to sidestep the anycast issues.
  - mercora 6y ago
    i had this issue but i just set the time to wait for propagation high enough to be somewhat certain and had no further issues since (its 10m i think). it does not really matter to me how long it takes as its an automated process... should be finished before expiration though ^^ having multiple nameservers is pretty standard and often mandatory requirement set by the NICs. Its just that my secondary is sometimes not fast enough to transfer the zone after a change notification which triggers this issue. Also, retrying after an authoritative nameserver said there is defacto no record seems pretty wrong to me... i doubt they use google DNS or anything really. in order to avoid caching issues they very likely resolve names recursively without (the usual) caching
  - throw0101a 6y ago
    > If you for example use anycast DNS you will run into issues. You're not wrong, but this assumes that you use the your 'service hostname' for verification as well, rather than using CNAMEs. So let us say you want to have "svc1.example.com" in your cert: you could put the ACME challenge under there, but if you have anycast delays that's a problem (as you mention). (A kludge is putting a 'sleep' somewhere to allow for propagation.) So instead what you can do is have "_acme-challenge.svc1.example.com" be a CNAME that points to (say) "_acme-challenge.svc1.dnsauth.example.com". This sub-domain is not anycast, and may actually be a single machine that is used solely for this purpose. The LE/ACME server goes to your main domain, finds a CNAME, and follows that to the real record and verification is achieved: * https://www.eff.org/deeplinks/2018/02/technical-deep-dive-securing-automation-acme-dns-challenge-validation https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se... * https://dan.langille.org/2019/02/01/acme-domain-alias-mode/ https://dan.langille.org/2019/02/01/acme-domain-alias-mode/ * https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mode https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo... The CNAME has to be set up initially, but can be left lying around otherwise. This is how $WORK deals with getting LE certs for internal domains: we create a CNAME record (but no A records) for the internal hostname in our external DNS that point to our "dnsauth" domain which gets updating by internal clients via an API.
- throw0101a 6y ago
  See also lexicon, which supports over 50 APIs: * https://github.com/AnalogJ/lexicon https://github.com/AnalogJ/lexicon
aforwardslash 6y ago
Keep in mind, adding local entries to your external DNS will expose internal details of your network, such as hostnames and IPs. Same goes for Let's Encrypt, due to Certificate Transparency logging.
- iso1210 6y ago
  While you'll get the hostnames leaked - you could register them as fake addresses (say an A record for 192.168.0.1 for every address), and have a local DNS server overriding with the real addresses. Whether this is worthwhile or not is debatable. Is the fact your internal server 'gubbins.mydomain.com' exists, or even that it exists on 10.0.41.43 really much use? The other option for internal certificates is to get a wildcard of *.internal.mydomain.com, and spread that wildcard certificate around your network. The final solution is run your own certificate authority and trust it on every browser. For some reason when you import a root certificate you can't typically allow that CA to only be used to authenticate a given subdomain. There are x509 constraints you can use in setting up the CA, but that's rare too, and I'm not sure every tool uses it. In any case, if you go for an internal DNS provision, make sure you set use-application-dns.net to NXDOMAIN on your internal dns server to override DoH too
  - cassianoleal 6y ago
    > Is the fact your internal server 'gubbins.mydomain.com' exists, or even that it exists on 10.0.41.43 really much use? Pretty much this. What does it matter if you know certain hostnames or internal IPs on my network? It's all firewalled anyway, and if it wasn't it would be trivial to find them out on your own...
    - Xylakant 6y ago
      It may be of interest for attackers that have no visibility into your network and make cross site attacks against you easier. For example, if I know that your router is available at router.network.internal, I might just try and see if your browser is logged in and send you a link to a page that starts making requests against that interface. Enumerating network resources can certainly be done via other ways, but DNS is a particularly easy one.
deleted 6y ago
[deleted]
guerby 6y ago
I haven't tried it yet but if you have control of your DNS and want automation: https://github.com/joohoi/acme-dns/ https://github.com/joohoi/acme-dns/ https://github.com/joohoi/acme-dns-certbot https://github.com/joohoi/acme-dns-certbot A simplified DNS server with a RESTful HTTP API to provide a simple way to automate ACME DNS challenges.
swiley 6y ago
I’ve tried to set up kubernetes at home a couple of times and I always freak out at the amount of layers and “just run this” style of tutorials. Am I crazy? I’ve heard guix has some kind of container management thing. I’ve been thinking about trying it anyway.
- Schwan 6y ago
  Ubuntu with Microk8s is very simple. Basically a few lines of commands, well documented and addons exist as well for most use cases. MetalLB for your loadbalancing needs, prometheus/grafana etc. https://microk8s.io/docs https://microk8s.io/docs And microk8s supports multiply nodes now as well.
- pas 6y ago
  k8s needs a control plane, that needs security, hence all the tokens, certs (which need internal and external IPs and FQDNs), also it needs to set up an overlay network (so you need to configure the CNI provider, sysctl stuff for ebtables and iptables/nftables to work correctly), and DNS, and a dashboard would be nice too. oh, and unless you use k3s or something that budles a container runtime (CRI provider) you need to setup one (eg docker). it's understandably complex, even if many parts are pretty standard (eg. the sysctl stuff, and installing dependencies is basically dnf/yum/apt/apk or exit and let the user do it). since the most error prone parts were/are setting up the TLS stuff that got automated first (in the form of kubeadm install), and the rest just remains in "run this" form. but the k3s installer is just a one liner call to a bash script. though then you have to make sure to include the magic env vars to get what you want.
  - DenseComet 6y ago
    The k3sup project [1] takes this a step further and makes installing k3s even easier. k3s has been the most useful piece of infra I run at home. It gives me all the benefits of k8s with none of the complexity. [1] https://github.com/alexellis/k3sup https://github.com/alexellis/k3sup
- achempion 6y ago
  "Kubernetes at home" called "Docker Swarm". I've been using it couple of years for many types of workloads and it's been a pleasant experience. Ansible + docker-compose.yml to manage apps on 1-3 servers. You just install docker and that's it. Swarm configuration is 95% similar to compose file you use locally if you're into developing with Docker. It has couple of edges if you try to use it for complex setups but for apps with < 5 devs and < 10 services it's really simple. You can still migrate to K8s when you hire dedicated DevOps team because you're running containers and all your setup documented in docker-compose file.
user5994461 6y ago
This page is raising a ton of security alerts: "NoScript detected a potential Cross-Site Scripting attack from https://www.techprowd.com https://www.techprowd.com to https://carbon.now.sh" https://carbon.now.sh" Images are failing to load too. Not sure what's going on. edit: Probably some misusing of DNS rather than actual attack but who knows. Author should fix the site.
- Xylakant 6y ago
  I think that's NoScript being overprotective. carbon.now.sh is a site that renders nice terminal sessions as html that you can include as iframes. - better than screenshots because you can actually copy the code. And as part of that request, the shell code is passed in the query string. I haven't investigated, but NoScript may be triggering on that.
- aurimasniekis 6y ago
  It's hosted on Ghost Pro serviced blog platform. https://carbon.now.sh https://carbon.now.sh is those code blocks u see in the article it's code highlighter
aasasd 6y ago
TLDR: - have a proper worldwide domain - obtain a certificate for that domain - point the domain to local IPs in your network and use the certificate on the local server. Doesn't change that you'd need to self-sign certs for .local or other funky domains.
- tialaramex 6y ago
  Probably just give things which need names actual globally unique names from the Internet's DNS hierarchy. More controversially I think you should give things globally unique addresses from the Internet's global address system but it's more important to at least give them names from the globally unique system even if you insist on using RFC1918 numbers.
Schwan 6y ago
Its TLS and not SSL. Its TLS for a long time now... And yes be aware that through this, it works fine but you are also exposing your internal infrastructure details through dns. I'm not seeing a big issue, just be aware of it.
- samgranieri 6y ago
  Just setup a wildcard cert, apply that to wherever, and you limit how much you expose your internal infra.
- teh_klev 6y ago
  > Its TLS and not SSL. Its TLS for a long time now... Sure, that's technically correct but a wee bit overly pedantic. When technical people speak about SSL/TLS certificates it's common parlance to say "SSL" and everyone usually knows what you're talking about, which includes TLS, and whatever other new acronym might come down the pipe in the future.
  - Schwan 6y ago
    You know, i do get this but we are in IT not in Marketing. My most used skill is to make sure i'm pedantic aka 'so we need to calculate this from that after this? and we need accuracy of 0.32? And what should that button do exactly?' It is not SSL its TLS and i don't expect everyone to get it but its still wrong. The weirdes thing in IT is, that i don't know any other word which is so missused then SSL.
- berbec 6y ago
  I know the difference, but common usage is to refer to the whole group of protocols as SSL. I have yet to apt-get install opentls on any webserver.
- darkarmani 6y ago
  Good point even if it is pedantic. I shift between terms depending on my audience, but try to use TLS in technical circles as an example. I worry that I'll forever have to use both terms, because while SSL is the term that communicates better to semi-literate audiences, I worry that some security expert will assume I don't know anything because I'm using the "wrong" term.
varbhat 6y ago
Instead of using hacky fragile methods , use https://github.com/FiloSottile/mkcert https://github.com/FiloSottile/mkcert to automate setting up local CA and making it trusted.
- Xylakant 6y ago
  mkcert may work fine if you're the only person using those network resources on a single machine, ever. Otherwise you've just traded yourself a trust management problem: Now you need to secure the key of that CA and distribute the CA certificate to all devices that should be trusting it. This may or may not be trivial. The fundamental problem is that this CA that you generated gets basically the same trust level as a public CA, but it's just sitting there on your machine. An attacker could use it to generate certificates for almost every site and your devices would trust them. That's probably ok if only your machine trusts that CA since if the attacker rooted your box to the point that they gained access to that CA key, all is lost anyways. In a network with other devices - maybe even not under your direct control - that tradeoff looks substantially different.
- samgranieri 6y ago
  I love that program and use that only on my laptop when I'm doing web development paired with nginx and dnsmasq. I do have to set an environment variable so nodejs trusts it.
jimueller 6y ago
split dns is typically the solution for this, is it not?
deleted 6y ago
[deleted]
Hitton 6y ago
It's not a certificate for internal ip address, it's a certificate for host name. Ip address is irrelevant here.
ttouch 6y ago
I did that, the very hard way (I didn’t know better at the time): https://whynot.fail/homelab/lets-encrypt-the-house/ https://whynot.fail/homelab/lets-encrypt-the-house/
phrygian 6y ago
I use step-ca [0] for these sort of things and it works brilliantly. I barely see the point of having external DNS servers resolving your internal infrastructure. [0] https://smallstep.com/certificates/ https://smallstep.com/certificates/
- samgranieri 6y ago
  I thought about that but passed because I didn't feel like telling all my browsers to trust that new CA. Yes, that's incredibly lazy. I bought a real domain name, told my UBNT USG that was the domain for my network, set up the dns servers to use digital ocean, used jetstack's cert-manager [0] to acquire the a wildcart cert using DNS01 instead of HTTP01, and use kubed [1] to synchronize the TLS cert across namespaces. One key thing to consider is that you really should ensure that you use the staging let's encrypt server to test out issuance and see your browser complain about warnings before you switch to production let's encrypt. Honestly, I don't mind that the cert requests for my domain show up in a CT log. [0] https://cert-manager.io/ https://cert-manager.io/ [1] https://cert-manager.io/docs/faq/kubed/ https://cert-manager.io/docs/faq/kubed/
aamsuzon 6y ago
Cloudfair is anothers solution i think.
danShumway 6y ago
I wrote a similar post about a year ago[0], but even at the time I wasn't the first to come up with this idea. As someone who doesn't have a lot of experiences with DNS security, seeing other people floating similar setups without significant pushback gives me more confidence that the core idea isn't horribly unsafe. I'm pretty happy/relieved to see other people playing around in the same space. My perspective was (and is) that for portable devices (phones/laptops) that are interfacing with locally hosted services, having SSL for those services is really important because your device probably isn't configured to check what network it's on before automatically pinging 192.168.1.x. This is doubly important if you have other people occasionally hopping onto your network and connecting to those same services. It's imo bad practice to ask everyone connecting to your network to install certificates or set up a certificate manager. I wouldn't do that for any of my personal devices if someone asked me to. To push this a step farther, I imagined a world where my services could handle not just renewing their own certificates, but also updating their addresses if they were moved to a different network/address. If I build a physical device to give to someone, I'd like them to be able to plug it into their network, go to a web URL, and have everything just work -- no messing around with their internal DNS settings or worrying about whether they're using DNS over HTTPS in Firefox. [0]: https://danshumway.com/blog/encrypting-internal-networks/ https://danshumway.com/blog/encrypting-internal-networks/
alexellisuk 6y ago
inlets with the inlets-operator [0] does this by using the HTTP01 challenge, and gives you a LoadBalancer just like you'd have on AWS. The benefit is that you get a real IP and routable traffic, there's no tricks required. It would also work with DNS01 if that's of interest. [0] https://github.com/inlets/inlets-operator https://github.com/inlets/inlets-operator Feel free to check it out in this tutorial: https://docs.inlets.dev/#/get-started/quickstart-ingresscontroller-cert-manager https://docs.inlets.dev/#/get-started/quickstart-ingresscont...
digitalsanctum 6y ago
Another alternative is inlets which automates all of the steps necessary and offers Layer 4 as well as Layer 7: https://docs.inlets.dev/#/ https://docs.inlets.dev/#/
berbec 6y ago
Why not just get a wildcard LE cert and not worry about it?
deleted 6y ago
[deleted]
viro 6y ago
Honestly this feels overly complex when you can just create a CA and add the CA to ur devices. Still cool tho.
- kakwa_ 6y ago
  At home, with just a few devices, it's doable, but adding a ca on all the devices of a corporate network is a huge pain. On one hand, you have varying levels of control (from none to total) on the devices. On the other hand adding a ca is a bit of a pain, with the ca needed to be added in various ca collection s for different softwares (ex in linux, ca-certificates, java certificates, and to add them system wide for browsers, you need to recompile libnss).
- throw0101a 6y ago
  We looked into this at $WORK, but it can be slightly annoying as you have to create a workflow for each operating system's trust store, but you also have to deal with many browsers independently as well, since many of them don't use the OS' trust store.
- darkarmani 6y ago
  You don't want to be running a CA. Because devices don't implement "name constraints", once you import your CA Root cert into a device, all HTTPS communications on that device can be subverted if someone gets ahold of your Root CA's cert/private key. I find it incredibly annoying that i can't tell chrome to use THIS CA Root Cert only for *.mydomain.com and not my banking domains, email, etc.
Naac 6y ago
I just created a wildcard with letsencrypt in the format of .internal.mydomain.com My public services all run out of .mydomain.com and all my internal services run out of .internal.mydomain.com I have my internal dns set to resolve any .internal calls to an internal load balancer which hosts the ca certs. The downside is that all internal services are ssl terminated at the load balancer, but this makes handling internal certs easy as they're rotated in a single location. This is Good Enough for my homelab.
windexh8er 6y ago
I do this with Traefik [0] internally in almost the same way. I use DNS-01 to get a Let's Encrypt wildcard cert and all my internal A records point to the ingress IP and Traefik happily proxies the communications to the appropriate service - container based and non-container based - which is the real win I was looking to solve for in my home environment. The thing I like about just using Traefik is it doesn't rely on a lot of extraneous tooling (can just use Docker without Swarm/K8s) and will automatically consume orchestration services if I'd like it to. But the reality is the majority of things I want valid certs for are static mappings. One config file update of a few new lines of boilerplate is all it takes to get a valid cert fronting any service. And then to get a dashboard of all my internal services I use Heimdall [1]. [0] https://docs.traefik.io/ https://docs.traefik.io/ [1] https://heimdall.site/ https://heimdall.site/
- j45 6y ago
  Appreciate this breakdown, I've been using an nginx proxy to hold a few things over and wanted to move in the direction of a managed service, which Traefik looks perfect for. As time goes on the value of having services running as appliances is becoming more and more valuable.
rackforms 6y ago
No association what so ever to 'em but I so dearly love what they do, I'd encourage users to donate to keep them going healthy and strong! https://letsencrypt.org/donate/ https://letsencrypt.org/donate/

156 points by gcds 6y ago

lgbr 6y ago

Cert-manager has great support for a number of providers[0] including AWS, CloudFlare, Google Cloud, and Azure. I recommend this not just for internal IP setups, for actually for all setups, since DNS verification is more robust than HTTP verification, particularly if you have issues with load balancers, or if Let's Encrypt decides to deprecate a protocol again [1]. [0] https://cert-manager.io/docs/configuration/acme/dns01/#supported-dns01-providers https://cert-manager.io/docs/configuration/acme/dns01/#suppo... [1] https://community.letsencrypt.org/t/upcoming-tls-sni-deprecation-in-certbot/76383 https://community.letsencrypt.org/t/upcoming-tls-sni-depreca...

tialaramex 6y ago
I like DNS challenges, but I don't see how it matters for deprecation of an ACME challenge type. The dns-01 challenge could just as easily for some reason need to be deprecated. The two likely reasons for such deprecation would apply just as well: 1. Updated Baseline Requirements or a programme policy requirement at any of the major root trust stores could forbid this challenge or require it to be substantially modified, obsoleting it in its current form. 2. The BRs don't change but Let's Encrypt finds they need to adjust this particular implementation in a non-compatible way so they deprecate the current challenge. It can be easier to do DNS challenges, but it can also be very rough, depending on all the moving parts in your system.
- sleevi 6y ago
  Both of those are reasonable concerns, if all other factors were ignored. However, in practice, the DNS challenge (which demonstrates control over DNS) is greatly preferred over HTTP/TLS challenges (which demonstrate control over a single port). DNS is likely to be the only way to get a wildcard certificate, and HTTP/TLS will likely end up further restricted once SRVNames in certificates can be gracefully rolled out. As such, deploying the DNS based control is absolutely the best thing to do, and HTTP/TLS should be seen as legacy-compat fallbacks that may become more difficult in time. Either certificates become less scoped than “entire domain” (as they are today with dNSName SANs) or it becomes more difficult to use a single port to prove authorization for an entire domain.
  - sroussey 6y ago
    But can’t DNS queries be altered man in the middle style?
    - sleevi 6y ago
      I’m not sure your point? Any HTTP/ALPN request first begins with DNS, so if you’re trying to compare those, they all share the same base issue. In theory, this can be mitigated by DNSSEC, but that’s not relevant when comparing these validation methods. However, both the HTTP and ALPN methods only demonstrate control over a single port (or .well-known resource), while the DNS method demonstrates the full ability to alter any/all names.
      - sroussey 6y ago
        Actually, I suppose DNS with DNSSEC or DNS over HTTPS would be better than any HTTP method.
z3t4 6y ago
Verification via DNS is not without issues. If you have more then one DNS server the verification record need to propagate to all servers. If you for example use anycast DNS you will run into issues. Letsencrypt uses Google name servers for lookup which is problematic because they do not behave, they will for example not try secondary dns servers if the first try fail, making the Letsencrypt verification also fail. And because of these issues and if you have many domains you will quickly reach Letsencrypt quota.
- sleevi 6y ago
  Where have you seen Let’s Encrypt using Google’s servers? CAs are required to run full recursive resolvers, up to the root, and can’t just point at someone else’s DNS infrastructure. Which, if you think about it, is what you want: you don’t want the CA just trusting someone else is being honest, you want to go to the authoritative source.
- radiowave 6y ago
  In which case it may be advisable to delegate the acme-challenge record to a different DNS provider, if doing so allows you to sidestep the anycast issues.
- mercora 6y ago
  i had this issue but i just set the time to wait for propagation high enough to be somewhat certain and had no further issues since (its 10m i think). it does not really matter to me how long it takes as its an automated process... should be finished before expiration though ^^ having multiple nameservers is pretty standard and often mandatory requirement set by the NICs. Its just that my secondary is sometimes not fast enough to transfer the zone after a change notification which triggers this issue. Also, retrying after an authoritative nameserver said there is defacto no record seems pretty wrong to me... i doubt they use google DNS or anything really. in order to avoid caching issues they very likely resolve names recursively without (the usual) caching
- throw0101a 6y ago
  > If you for example use anycast DNS you will run into issues. You're not wrong, but this assumes that you use the your 'service hostname' for verification as well, rather than using CNAMEs. So let us say you want to have "svc1.example.com" in your cert: you could put the ACME challenge under there, but if you have anycast delays that's a problem (as you mention). (A kludge is putting a 'sleep' somewhere to allow for propagation.) So instead what you can do is have "_acme-challenge.svc1.example.com" be a CNAME that points to (say) "_acme-challenge.svc1.dnsauth.example.com". This sub-domain is not anycast, and may actually be a single machine that is used solely for this purpose. The LE/ACME server goes to your main domain, finds a CNAME, and follows that to the real record and verification is achieved: * https://www.eff.org/deeplinks/2018/02/technical-deep-dive-securing-automation-acme-dns-challenge-validation https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se... * https://dan.langille.org/2019/02/01/acme-domain-alias-mode/ https://dan.langille.org/2019/02/01/acme-domain-alias-mode/ * https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mode https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo... The CNAME has to be set up initially, but can be left lying around otherwise. This is how $WORK deals with getting LE certs for internal domains: we create a CNAME record (but no A records) for the internal hostname in our external DNS that point to our "dnsauth" domain which gets updating by internal clients via an API.
throw0101a 6y ago
See also lexicon, which supports over 50 APIs: * https://github.com/AnalogJ/lexicon https://github.com/AnalogJ/lexicon

aforwardslash 6y ago

Keep in mind, adding local entries to your external DNS will expose internal details of your network, such as hostnames and IPs. Same goes for Let's Encrypt, due to Certificate Transparency logging.

iso1210 6y ago
While you'll get the hostnames leaked - you could register them as fake addresses (say an A record for 192.168.0.1 for every address), and have a local DNS server overriding with the real addresses. Whether this is worthwhile or not is debatable. Is the fact your internal server 'gubbins.mydomain.com' exists, or even that it exists on 10.0.41.43 really much use? The other option for internal certificates is to get a wildcard of *.internal.mydomain.com, and spread that wildcard certificate around your network. The final solution is run your own certificate authority and trust it on every browser. For some reason when you import a root certificate you can't typically allow that CA to only be used to authenticate a given subdomain. There are x509 constraints you can use in setting up the CA, but that's rare too, and I'm not sure every tool uses it. In any case, if you go for an internal DNS provision, make sure you set use-application-dns.net to NXDOMAIN on your internal dns server to override DoH too
- cassianoleal 6y ago
  > Is the fact your internal server 'gubbins.mydomain.com' exists, or even that it exists on 10.0.41.43 really much use? Pretty much this. What does it matter if you know certain hostnames or internal IPs on my network? It's all firewalled anyway, and if it wasn't it would be trivial to find them out on your own...
  - Xylakant 6y ago
    It may be of interest for attackers that have no visibility into your network and make cross site attacks against you easier. For example, if I know that your router is available at router.network.internal, I might just try and see if your browser is logged in and send you a link to a page that starts making requests against that interface. Enumerating network resources can certainly be done via other ways, but DNS is a particularly easy one.

deleted 6y ago

[deleted]

guerby 6y ago

I haven't tried it yet but if you have control of your DNS and want automation: https://github.com/joohoi/acme-dns/ https://github.com/joohoi/acme-dns/ https://github.com/joohoi/acme-dns-certbot https://github.com/joohoi/acme-dns-certbot A simplified DNS server with a RESTful HTTP API to provide a simple way to automate ACME DNS challenges.

swiley 6y ago

I’ve tried to set up kubernetes at home a couple of times and I always freak out at the amount of layers and “just run this” style of tutorials. Am I crazy? I’ve heard guix has some kind of container management thing. I’ve been thinking about trying it anyway.

Schwan 6y ago
Ubuntu with Microk8s is very simple. Basically a few lines of commands, well documented and addons exist as well for most use cases. MetalLB for your loadbalancing needs, prometheus/grafana etc. https://microk8s.io/docs https://microk8s.io/docs And microk8s supports multiply nodes now as well.
pas 6y ago
k8s needs a control plane, that needs security, hence all the tokens, certs (which need internal and external IPs and FQDNs), also it needs to set up an overlay network (so you need to configure the CNI provider, sysctl stuff for ebtables and iptables/nftables to work correctly), and DNS, and a dashboard would be nice too. oh, and unless you use k3s or something that budles a container runtime (CRI provider) you need to setup one (eg docker). it's understandably complex, even if many parts are pretty standard (eg. the sysctl stuff, and installing dependencies is basically dnf/yum/apt/apk or exit and let the user do it). since the most error prone parts were/are setting up the TLS stuff that got automated first (in the form of kubeadm install), and the rest just remains in "run this" form. but the k3s installer is just a one liner call to a bash script. though then you have to make sure to include the magic env vars to get what you want.
- DenseComet 6y ago
  The k3sup project [1] takes this a step further and makes installing k3s even easier. k3s has been the most useful piece of infra I run at home. It gives me all the benefits of k8s with none of the complexity. [1] https://github.com/alexellis/k3sup https://github.com/alexellis/k3sup
achempion 6y ago
"Kubernetes at home" called "Docker Swarm". I've been using it couple of years for many types of workloads and it's been a pleasant experience. Ansible + docker-compose.yml to manage apps on 1-3 servers. You just install docker and that's it. Swarm configuration is 95% similar to compose file you use locally if you're into developing with Docker. It has couple of edges if you try to use it for complex setups but for apps with < 5 devs and < 10 services it's really simple. You can still migrate to K8s when you hire dedicated DevOps team because you're running containers and all your setup documented in docker-compose file.

user5994461 6y ago

This page is raising a ton of security alerts: "NoScript detected a potential Cross-Site Scripting attack from https://www.techprowd.com https://www.techprowd.com to https://carbon.now.sh" https://carbon.now.sh" Images are failing to load too. Not sure what's going on. edit: Probably some misusing of DNS rather than actual attack but who knows. Author should fix the site.

Xylakant 6y ago
I think that's NoScript being overprotective. carbon.now.sh is a site that renders nice terminal sessions as html that you can include as iframes. - better than screenshots because you can actually copy the code. And as part of that request, the shell code is passed in the query string. I haven't investigated, but NoScript may be triggering on that.
aurimasniekis 6y ago
It's hosted on Ghost Pro serviced blog platform. https://carbon.now.sh https://carbon.now.sh is those code blocks u see in the article it's code highlighter

aasasd 6y ago

TLDR: - have a proper worldwide domain - obtain a certificate for that domain - point the domain to local IPs in your network and use the certificate on the local server. Doesn't change that you'd need to self-sign certs for .local or other funky domains.

tialaramex 6y ago
Probably just give things which need names actual globally unique names from the Internet's DNS hierarchy. More controversially I think you should give things globally unique addresses from the Internet's global address system but it's more important to at least give them names from the globally unique system even if you insist on using RFC1918 numbers.

Schwan 6y ago

Its TLS and not SSL. Its TLS for a long time now... And yes be aware that through this, it works fine but you are also exposing your internal infrastructure details through dns. I'm not seeing a big issue, just be aware of it.

samgranieri 6y ago
Just setup a wildcard cert, apply that to wherever, and you limit how much you expose your internal infra.
teh_klev 6y ago
> Its TLS and not SSL. Its TLS for a long time now... Sure, that's technically correct but a wee bit overly pedantic. When technical people speak about SSL/TLS certificates it's common parlance to say "SSL" and everyone usually knows what you're talking about, which includes TLS, and whatever other new acronym might come down the pipe in the future.
- Schwan 6y ago
  You know, i do get this but we are in IT not in Marketing. My most used skill is to make sure i'm pedantic aka 'so we need to calculate this from that after this? and we need accuracy of 0.32? And what should that button do exactly?' It is not SSL its TLS and i don't expect everyone to get it but its still wrong. The weirdes thing in IT is, that i don't know any other word which is so missused then SSL.
berbec 6y ago
I know the difference, but common usage is to refer to the whole group of protocols as SSL. I have yet to apt-get install opentls on any webserver.
darkarmani 6y ago
Good point even if it is pedantic. I shift between terms depending on my audience, but try to use TLS in technical circles as an example. I worry that I'll forever have to use both terms, because while SSL is the term that communicates better to semi-literate audiences, I worry that some security expert will assume I don't know anything because I'm using the "wrong" term.

varbhat 6y ago

Instead of using hacky fragile methods , use https://github.com/FiloSottile/mkcert https://github.com/FiloSottile/mkcert to automate setting up local CA and making it trusted.

Xylakant 6y ago
mkcert may work fine if you're the only person using those network resources on a single machine, ever. Otherwise you've just traded yourself a trust management problem: Now you need to secure the key of that CA and distribute the CA certificate to all devices that should be trusting it. This may or may not be trivial. The fundamental problem is that this CA that you generated gets basically the same trust level as a public CA, but it's just sitting there on your machine. An attacker could use it to generate certificates for almost every site and your devices would trust them. That's probably ok if only your machine trusts that CA since if the attacker rooted your box to the point that they gained access to that CA key, all is lost anyways. In a network with other devices - maybe even not under your direct control - that tradeoff looks substantially different.
samgranieri 6y ago
I love that program and use that only on my laptop when I'm doing web development paired with nginx and dnsmasq. I do have to set an environment variable so nodejs trusts it.

jimueller 6y ago

split dns is typically the solution for this, is it not?

deleted 6y ago

[deleted]

Hitton 6y ago

It's not a certificate for internal ip address, it's a certificate for host name. Ip address is irrelevant here.

ttouch 6y ago

I did that, the very hard way (I didn’t know better at the time): https://whynot.fail/homelab/lets-encrypt-the-house/ https://whynot.fail/homelab/lets-encrypt-the-house/

phrygian 6y ago

I use step-ca [0] for these sort of things and it works brilliantly. I barely see the point of having external DNS servers resolving your internal infrastructure. [0] https://smallstep.com/certificates/ https://smallstep.com/certificates/

samgranieri 6y ago
I thought about that but passed because I didn't feel like telling all my browsers to trust that new CA. Yes, that's incredibly lazy. I bought a real domain name, told my UBNT USG that was the domain for my network, set up the dns servers to use digital ocean, used jetstack's cert-manager [0] to acquire the a wildcart cert using DNS01 instead of HTTP01, and use kubed [1] to synchronize the TLS cert across namespaces. One key thing to consider is that you really should ensure that you use the staging let's encrypt server to test out issuance and see your browser complain about warnings before you switch to production let's encrypt. Honestly, I don't mind that the cert requests for my domain show up in a CT log. [0] https://cert-manager.io/ https://cert-manager.io/ [1] https://cert-manager.io/docs/faq/kubed/ https://cert-manager.io/docs/faq/kubed/

aamsuzon 6y ago

Cloudfair is anothers solution i think.

danShumway 6y ago

I wrote a similar post about a year ago[0], but even at the time I wasn't the first to come up with this idea. As someone who doesn't have a lot of experiences with DNS security, seeing other people floating similar setups without significant pushback gives me more confidence that the core idea isn't horribly unsafe. I'm pretty happy/relieved to see other people playing around in the same space. My perspective was (and is) that for portable devices (phones/laptops) that are interfacing with locally hosted services, having SSL for those services is really important because your device probably isn't configured to check what network it's on before automatically pinging 192.168.1.x. This is doubly important if you have other people occasionally hopping onto your network and connecting to those same services. It's imo bad practice to ask everyone connecting to your network to install certificates or set up a certificate manager. I wouldn't do that for any of my personal devices if someone asked me to. To push this a step farther, I imagined a world where my services could handle not just renewing their own certificates, but also updating their addresses if they were moved to a different network/address. If I build a physical device to give to someone, I'd like them to be able to plug it into their network, go to a web URL, and have everything just work -- no messing around with their internal DNS settings or worrying about whether they're using DNS over HTTPS in Firefox. [0]: https://danshumway.com/blog/encrypting-internal-networks/ https://danshumway.com/blog/encrypting-internal-networks/

alexellisuk 6y ago

inlets with the inlets-operator [0] does this by using the HTTP01 challenge, and gives you a LoadBalancer just like you'd have on AWS. The benefit is that you get a real IP and routable traffic, there's no tricks required. It would also work with DNS01 if that's of interest. [0] https://github.com/inlets/inlets-operator https://github.com/inlets/inlets-operator Feel free to check it out in this tutorial: https://docs.inlets.dev/#/get-started/quickstart-ingresscontroller-cert-manager https://docs.inlets.dev/#/get-started/quickstart-ingresscont...

digitalsanctum 6y ago

Another alternative is inlets which automates all of the steps necessary and offers Layer 4 as well as Layer 7: https://docs.inlets.dev/#/ https://docs.inlets.dev/#/

berbec 6y ago

Why not just get a wildcard LE cert and not worry about it?

deleted 6y ago

[deleted]

viro 6y ago

Honestly this feels overly complex when you can just create a CA and add the CA to ur devices. Still cool tho.

kakwa_ 6y ago
At home, with just a few devices, it's doable, but adding a ca on all the devices of a corporate network is a huge pain. On one hand, you have varying levels of control (from none to total) on the devices. On the other hand adding a ca is a bit of a pain, with the ca needed to be added in various ca collection s for different softwares (ex in linux, ca-certificates, java certificates, and to add them system wide for browsers, you need to recompile libnss).
throw0101a 6y ago
We looked into this at $WORK, but it can be slightly annoying as you have to create a workflow for each operating system's trust store, but you also have to deal with many browsers independently as well, since many of them don't use the OS' trust store.
darkarmani 6y ago
You don't want to be running a CA. Because devices don't implement "name constraints", once you import your CA Root cert into a device, all HTTPS communications on that device can be subverted if someone gets ahold of your Root CA's cert/private key. I find it incredibly annoying that i can't tell chrome to use THIS CA Root Cert only for *.mydomain.com and not my banking domains, email, etc.

Naac 6y ago

I just created a wildcard with letsencrypt in the format of .internal.mydomain.com My public services all run out of .mydomain.com and all my internal services run out of .internal.mydomain.com I have my internal dns set to resolve any .internal calls to an internal load balancer which hosts the ca certs. The downside is that all internal services are ssl terminated at the load balancer, but this makes handling internal certs easy as they're rotated in a single location. This is Good Enough for my homelab.

windexh8er 6y ago

I do this with Traefik [0] internally in almost the same way. I use DNS-01 to get a Let's Encrypt wildcard cert and all my internal A records point to the ingress IP and Traefik happily proxies the communications to the appropriate service - container based and non-container based - which is the real win I was looking to solve for in my home environment. The thing I like about just using Traefik is it doesn't rely on a lot of extraneous tooling (can just use Docker without Swarm/K8s) and will automatically consume orchestration services if I'd like it to. But the reality is the majority of things I want valid certs for are static mappings. One config file update of a few new lines of boilerplate is all it takes to get a valid cert fronting any service. And then to get a dashboard of all my internal services I use Heimdall [1]. [0] https://docs.traefik.io/ https://docs.traefik.io/ [1] https://heimdall.site/ https://heimdall.site/

j45 6y ago
Appreciate this breakdown, I've been using an nginx proxy to hold a few things over and wanted to move in the direction of a managed service, which Traefik looks perfect for. As time goes on the value of having services running as appliances is becoming more and more valuable.

rackforms 6y ago

No association what so ever to 'em but I so dearly love what they do, I'd encourage users to donate to keep them going healthy and strong! https://letsencrypt.org/donate/ https://letsencrypt.org/donate/