27 ms·

The relative proximity of the words "done right" and "split-horizon DNS" makes my insides hurt a little bit. Use DNS validation to allow these internal service

The relative proximity of the words "done right" and "split-horizon DNS" makes my insides hurt a little bit. Use DNS validation to allow these internal services to pull ACME certs. There's so much less headache, long-term. Split-horizon DNS (and the tedious make-work it can create when you start needing to mirror public-accessibly records in the private DNS) has always been something to aspire to move away from in my experience.

bombcar 3mo ago
Just LE a wild card cert and slap it everywhere.
- EvanAnderson 3mo ago
  I get antsy about a private key being in lots of places. You've also got to worry about renewal so you might as well just have "consumer" of the wildcard just provision its own non-wildcard certificate.
  - sandermvanvliet 3mo ago
    I run a reverse proxy that handles the TLS termination for that reason. Services themselves are constrained to the server, bound to listen on 127.0.0.1 only. Key is only available to the reverse proxy.
stock_toaster 3mo ago
Once dns-persist-01 becomes available/usable[1], it should make dns validation even easier. [1]: https://letsencrypt.org/2026/02/18/dns-persist-01 https://letsencrypt.org/2026/02/18/dns-persist-01
- isomorphic 3mo ago
  Thank you, I was unaware of that. It looks like it's already support in the acme.sh client, but there is a Let's Encrypt discussion saying it's still pending at LE: https://community.letsencrypt.org/t/dns-persist-01-deployment-status-and-timeline/246468/9 https://community.letsencrypt.org/t/dns-persist-01-deploymen... I wonder if the interim version has been rolled out to some CAs.
- mrl5 3mo ago
  OP here. Actually I tried to use it but apparently prematurely -> https://github.com/acmesh-official/acme.sh/issues/7085#issuecomment-4887449610 https://github.com/acmesh-official/acme.sh/issues/7085#issue... Once it's supported I think my next iteration will be DNS persist + internal ip addresses on the public zone. Thank you all for comments and feedback! It's cool to see real interest in this blog post
- theK 3mo ago
  I never understood the issue with DNS-01. if you have a process that you trust to maintain a zone's TLS identity what is the big deal about letting it control a record in that zone?
  - sigio 3mo ago
    You can even put it in a seperate zone (which I do) by using a CNAME for _acme-challenge.domain.tld. I have it to a seperate subdomain, which is served by a seperate desec.io account, which is only used for this specific subdomain.
    - blueflame7 3mo ago
      [dead]
    - mrl5 3mo ago
      TIL, thanks! This makes dns challange automation more approachable for me. I always had an issue with API keys which scope can't be limited to TXT records. This seems to be a nice workaround ... and it is even already documented at https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mode https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...
  - nijave 3mo ago
    We have a few subdomains for white labeling 3rd party SaaS where we do what is basically the AWS ACM equivalent and add a persistent record from a vendor. With this setup, I don't have to grant 3rd parties DNS access. I actually made a webhook that allows per hostname API keys to wrap dnsimple because they only had per zone keys and I didn't want each VM to have access to the entire zone. These challenges would have solved that by allowing the DNS record automation to pull record values from the VM instead of having the VMs push values. I think someone told me dnsimple might have more granular keys now but I haven't checked. Iirc we have the same concern with external-dns at work (some things need subdomains on the TLD but we don't want to give external-dns access to the whole zone so we usually cname the TLD subdomain to a per environment zone external-dns is allowed to update). With this, we could have the same pull based setup that applies arbitrary rules to decide if a requested record should be created. I think the main takeaway is allowing pull instead of push model Could be achieved with cnames but it's an extra layer of indirection to deal with and doesn't fully solve the "semi trusted 3rd party" case
- xg15 3mo ago
  At this point, can't we make a standard that skips the middlemen and just embeds the certificate directly inside the DNS entry? It seems the challenge system is converging towards that anyway.
  - zeeZ 3mo ago
    That would be DANE with TLSA (RFC 6698, not the stock ticker symbol). You've still got just another chain of trust with the DNSSEC requirement, and the recent DENIC outage breaking that for the entirety of .de isn't the greatest advertisement :D
    - tptacek 3mo ago
      It's also a dead letter: browsers won't implement it (they did at one point, and then withdrew it).
      - TAlborough 3mo ago
        [flagged]
    - xg15 3mo ago
      Yes, which goes back to the old question: why the heck are we using a bunch of insecure systems with awkward bolted-on workarounds instead of just using DNSSEC?
    - deleted 3mo ago
      [deleted]
    - xg15 3mo ago
      (To explain that some more: The current "way to go" with fully automated cert issuance is delegating its trust to DNS anyway - so we might as well get rid of CAs and use the DNSSEC chain of trust directly, with TLS certs linked to it.)
- ivlad 3mo ago
  Or, just use IPv6 and host Internet services in a routable address. Then, use ACLs at web server / proxy / L7 lb level to allow acme-challenge unauthenticated but everything else authenticated. Lets encrypt supports IPv6 for validation.
bruce511 3mo ago
Came here to say the same. I use DNS-Challenge rather than HTTP-challenge, and that makes internal servers trivial. You need a DNS provider which supports API calls (I use DNSimple) but the core is all very straightforward. To prevent having to include DNSimple authentication on the client's internal server I have a small API server on the web which does the Acme work.
- isomorphic 3mo ago
  I do this exactly, using ACME DNS: https://github.com/acme-dns/acme-dns https://github.com/acme-dns/acme-dns CAUTION, though, the last time I downloaded a binary release, ClamAV triggered on it, so I kept my old version which worked. I was using the 1.0 series (without any problems!), and now it seems the project has picked up development again with a 2.0 series.
- throw0101d 3mo ago
  > You need a DNS provider which supports API calls (I use DNSimple) but the core is all very straightforward. Library/CLI that speaks the API of several dozen DNS providers so you don't have to re-invent the wheel: * https://github.com/dns-lexicon/dns-lexicon https://github.com/dns-lexicon/dns-lexicon
  - aeden 3mo ago
    Interesting, this is the first time I've seen this project. One thing I saw that concerns me a bit is that they provide command-line options for passing your user credentials and multi-factor code. I'm not a fan of giving anything that level of access when a resource-specific access token will do, but that's me.
selfmodruntime 3mo ago
Also be careful when using split DNS and Tailscale, which increasingly won't work without MagicDNS enabled.
- JoeBOFH 3mo ago
  And you still have issues sometimes around dns not able to figure out which interface to use. We had issues around that before.
  - selfmodruntime 3mo ago
    Yes! That is our exact issue. Do you have any idea on how to circumvent this problem?
    - JoeBOFH 3mo ago
      Our issue was on multiple fronts. On windows it was GlobalProtect overriding dns requests. On Linux it was an issue with the OS fighting which nameservers to use. The fix was some experimental flag, tho I can’t remember it currently.
thayne 3mo ago
Another big problem with split-horizon is you can get clients that cache the dns result before connecting to the vpn, then after connecting, can't actually use the service, because it is using the public ip instead of the private one.
- throw0101d 3mo ago
  Just use IPv6: no "internal" (10/8) or "external" address, just an address. ;)
- waynesonfire 3mo ago
  > because it is using the public ip instead of the private one. For this edge case, the issue is not with DNS; it's that the router is not configured to allow internal clients to access the network through the public IP, vis-à-vis hairpin routing.
techcode 3mo ago
Yes - literally just pick the right Caddy (or similar) image with Let's Encrypt client included - and you can simply add a tag/label to your docker compose files for each of self hosted services to get a real SSL, that auto renewed ...etc. With one of self hosted services being Adguard-Home can do both ADs blocking and internal DNS... The public DNS records for your "internal use only" domain remain empty.
- s8kur 3mo ago
  [dead]
jve 3mo ago
... Could you please provide a solution? What should I do in my homelab? Saying something isn't bad without pointing to right direction makes my insides hurt a little bit.
- EvanAnderson 3mo ago
  I feel like I did. >smile< > Use DNS validation to allow these internal services to pull ACME certs... The major ACME clients support DNS validation. If your DNS host doesn't have an API that your ACME client supports either get a new ACME client or a new DNS host.
  - jve 3mo ago
    That's for the cert part, yeah, got it. I thought you have a solution to overcome not having to do split DNS where I define public IP in public DNS and internal IP within some local hosted DNS. Like how to make so that when connecting from inside local network the router recognize that by connecting to public IP, he has to route it back onto some local IP address?
    - EvanAnderson 3mo ago
      Hairpin NAT (or "NAT hairpinning") is the term for "...connecting from inside local network the router recognize that by connecting to public IP, he has to route it back onto some local IP address", and the Linux NAT implementation supports it. For a homelab situation split-horizon DNS is just fine. You're going to have minimal duplication of records from the public DNS into the private DNS. The canonical frustrating "bad split-horizon DNS" world I've seen, time and again, is a corporate network with a MSFT Active Directory named the same as the company's public-facing web property (e.g. "example.com" rather than "ad.example.com"). This creates the need to duplicate all the public-hosted resources into the internal "example.com" zone (and keep them updated when records inevitably change on the public Internet). It's make-work for no practical upside in that example.
      - jve 3mo ago

bombcar 3mo ago

Just LE a wild card cert and slap it everywhere.

EvanAnderson 3mo ago
I get antsy about a private key being in lots of places. You've also got to worry about renewal so you might as well just have "consumer" of the wildcard just provision its own non-wildcard certificate.
- sandermvanvliet 3mo ago
  I run a reverse proxy that handles the TLS termination for that reason. Services themselves are constrained to the server, bound to listen on 127.0.0.1 only. Key is only available to the reverse proxy.

stock_toaster 3mo ago

Once dns-persist-01 becomes available/usable[1], it should make dns validation even easier. [1]: https://letsencrypt.org/2026/02/18/dns-persist-01 https://letsencrypt.org/2026/02/18/dns-persist-01

isomorphic 3mo ago
Thank you, I was unaware of that. It looks like it's already support in the acme.sh client, but there is a Let's Encrypt discussion saying it's still pending at LE: https://community.letsencrypt.org/t/dns-persist-01-deployment-status-and-timeline/246468/9 https://community.letsencrypt.org/t/dns-persist-01-deploymen... I wonder if the interim version has been rolled out to some CAs.
mrl5 3mo ago
OP here. Actually I tried to use it but apparently prematurely -> https://github.com/acmesh-official/acme.sh/issues/7085#issuecomment-4887449610 https://github.com/acmesh-official/acme.sh/issues/7085#issue... Once it's supported I think my next iteration will be DNS persist + internal ip addresses on the public zone. Thank you all for comments and feedback! It's cool to see real interest in this blog post
theK 3mo ago
I never understood the issue with DNS-01. if you have a process that you trust to maintain a zone's TLS identity what is the big deal about letting it control a record in that zone?
- sigio 3mo ago
  You can even put it in a seperate zone (which I do) by using a CNAME for _acme-challenge.domain.tld. I have it to a seperate subdomain, which is served by a seperate desec.io account, which is only used for this specific subdomain.
  - blueflame7 3mo ago
    [dead]
  - mrl5 3mo ago
    TIL, thanks! This makes dns challange automation more approachable for me. I always had an issue with API keys which scope can't be limited to TXT records. This seems to be a nice workaround ... and it is even already documented at https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mode https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...
- nijave 3mo ago
  We have a few subdomains for white labeling 3rd party SaaS where we do what is basically the AWS ACM equivalent and add a persistent record from a vendor. With this setup, I don't have to grant 3rd parties DNS access. I actually made a webhook that allows per hostname API keys to wrap dnsimple because they only had per zone keys and I didn't want each VM to have access to the entire zone. These challenges would have solved that by allowing the DNS record automation to pull record values from the VM instead of having the VMs push values. I think someone told me dnsimple might have more granular keys now but I haven't checked. Iirc we have the same concern with external-dns at work (some things need subdomains on the TLD but we don't want to give external-dns access to the whole zone so we usually cname the TLD subdomain to a per environment zone external-dns is allowed to update). With this, we could have the same pull based setup that applies arbitrary rules to decide if a requested record should be created. I think the main takeaway is allowing pull instead of push model Could be achieved with cnames but it's an extra layer of indirection to deal with and doesn't fully solve the "semi trusted 3rd party" case
xg15 3mo ago
At this point, can't we make a standard that skips the middlemen and just embeds the certificate directly inside the DNS entry? It seems the challenge system is converging towards that anyway.
- zeeZ 3mo ago
  That would be DANE with TLSA (RFC 6698, not the stock ticker symbol). You've still got just another chain of trust with the DNSSEC requirement, and the recent DENIC outage breaking that for the entirety of .de isn't the greatest advertisement :D
  - tptacek 3mo ago
    It's also a dead letter: browsers won't implement it (they did at one point, and then withdrew it).
    - TAlborough 3mo ago
      [flagged]
  - xg15 3mo ago
    Yes, which goes back to the old question: why the heck are we using a bunch of insecure systems with awkward bolted-on workarounds instead of just using DNSSEC?
  - deleted 3mo ago
    [deleted]
  - xg15 3mo ago
    (To explain that some more: The current "way to go" with fully automated cert issuance is delegating its trust to DNS anyway - so we might as well get rid of CAs and use the DNSSEC chain of trust directly, with TLS certs linked to it.)
ivlad 3mo ago
Or, just use IPv6 and host Internet services in a routable address. Then, use ACLs at web server / proxy / L7 lb level to allow acme-challenge unauthenticated but everything else authenticated. Lets encrypt supports IPv6 for validation.

bruce511 3mo ago

Came here to say the same. I use DNS-Challenge rather than HTTP-challenge, and that makes internal servers trivial. You need a DNS provider which supports API calls (I use DNSimple) but the core is all very straightforward. To prevent having to include DNSimple authentication on the client's internal server I have a small API server on the web which does the Acme work.

isomorphic 3mo ago
I do this exactly, using ACME DNS: https://github.com/acme-dns/acme-dns https://github.com/acme-dns/acme-dns CAUTION, though, the last time I downloaded a binary release, ClamAV triggered on it, so I kept my old version which worked. I was using the 1.0 series (without any problems!), and now it seems the project has picked up development again with a 2.0 series.
throw0101d 3mo ago
> You need a DNS provider which supports API calls (I use DNSimple) but the core is all very straightforward. Library/CLI that speaks the API of several dozen DNS providers so you don't have to re-invent the wheel: * https://github.com/dns-lexicon/dns-lexicon https://github.com/dns-lexicon/dns-lexicon
- aeden 3mo ago
  Interesting, this is the first time I've seen this project. One thing I saw that concerns me a bit is that they provide command-line options for passing your user credentials and multi-factor code. I'm not a fan of giving anything that level of access when a resource-specific access token will do, but that's me.

selfmodruntime 3mo ago

Also be careful when using split DNS and Tailscale, which increasingly won't work without MagicDNS enabled.

JoeBOFH 3mo ago
And you still have issues sometimes around dns not able to figure out which interface to use. We had issues around that before.
- selfmodruntime 3mo ago
  Yes! That is our exact issue. Do you have any idea on how to circumvent this problem?
  - JoeBOFH 3mo ago
    Our issue was on multiple fronts. On windows it was GlobalProtect overriding dns requests. On Linux it was an issue with the OS fighting which nameservers to use. The fix was some experimental flag, tho I can’t remember it currently.

thayne 3mo ago

Another big problem with split-horizon is you can get clients that cache the dns result before connecting to the vpn, then after connecting, can't actually use the service, because it is using the public ip instead of the private one.

throw0101d 3mo ago
Just use IPv6: no "internal" (10/8) or "external" address, just an address. ;)
waynesonfire 3mo ago
> because it is using the public ip instead of the private one. For this edge case, the issue is not with DNS; it's that the router is not configured to allow internal clients to access the network through the public IP, vis-à-vis hairpin routing.

techcode 3mo ago

Yes - literally just pick the right Caddy (or similar) image with Let's Encrypt client included - and you can simply add a tag/label to your docker compose files for each of self hosted services to get a real SSL, that auto renewed ...etc. With one of self hosted services being Adguard-Home can do both ADs blocking and internal DNS... The public DNS records for your "internal use only" domain remain empty.

s8kur 3mo ago
[dead]

jve 3mo ago

... Could you please provide a solution? What should I do in my homelab? Saying something isn't bad without pointing to right direction makes my insides hurt a little bit.

EvanAnderson 3mo ago
I feel like I did. >smile< > Use DNS validation to allow these internal services to pull ACME certs... The major ACME clients support DNS validation. If your DNS host doesn't have an API that your ACME client supports either get a new ACME client or a new DNS host.
- jve 3mo ago
  That's for the cert part, yeah, got it. I thought you have a solution to overcome not having to do split DNS where I define public IP in public DNS and internal IP within some local hosted DNS. Like how to make so that when connecting from inside local network the router recognize that by connecting to public IP, he has to route it back onto some local IP address?
  - EvanAnderson 3mo ago
    Hairpin NAT (or "NAT hairpinning") is the term for "...connecting from inside local network the router recognize that by connecting to public IP, he has to route it back onto some local IP address", and the Linux NAT implementation supports it. For a homelab situation split-horizon DNS is just fine. You're going to have minimal duplication of records from the public DNS into the private DNS. The canonical frustrating "bad split-horizon DNS" world I've seen, time and again, is a corporate network with a MSFT Active Directory named the same as the company's public-facing web property (e.g. "example.com" rather than "ad.example.com"). This creates the need to duplicate all the public-hosted resources into the internal "example.com" zone (and keep them updated when records inevitably change on the public Internet). It's make-work for no practical upside in that example.
    - jve 3mo ago