11 ms·
Using Let's Encrypt for Internal Servers (2018)
- cm2187 8y agoAlso useful for smtp/imap servers.
- evenh 8y agoI'm in the process of writing a tool for automating this: https://github.com/evenh/intercert https://github.com/evenh/intercert. It’s still in the early stages though. EDIT: Based on the fantastic CertMagic library by mholt
- mholt 8y agoNeat, I hadn't known about this!
- mjlee 8y agoYou should be aware of Certificate Transparency if you go down this route. While I agree that it probably shouldn't matter it could help an attacker gather knowledge about your internal network. Perhaps it's of interest that you use a specific monitoring tool or mail server internally - that could make phishing emails easier to craft.
- mholt 8y agoEven without CT, you're still setting public DNS records -- so better yet, don't rely on "internal" hostnames staying private if you want them publicly trusted. Full stop.
- organsnyder 8y agoIt's very common for internal hostnames to be resolvable only internally (such as with split-zone or dedicated internal DNS servers). Edit: Or do you mean as part of the challenge-response mechanism? Yeah, that would have to be public.
- zeeZ 8y agoOnly needs to be public for the challenge though. We use split DNS with an external server that has an empty zone. Internal clients are limited to setting their respective _acme-challenge.subdomain TXT record and delete it afterwards.
- someone13 8y agoOut of curiosity, how is this implemented?
- w7 8y agoNot the person you asked, but: Besides possibly being a function of a provider's API; DNS server security policies can be used to limit updates to certain domains and/or record types based on preshared key. Since the DNS-01 challenge only needs to make a TXT record with a predetermined name you can configure a zone like so (using BIND syntax as an example): key "example-key" { algorithm hmac-sha512; secret <KEY_HERE>; }; zone { ... update-policy { grant "example-key" name _acme-challenge.example.com TXT; }; }; another option is to have a CNAME from _acme-challenge.example.com to a dedicated challenge zone like challenges.example.com that has similar restrictions. This coupled with something like acme.sh makes it easy and relatively secure for machines to generate their own certificates.
- isostatic 8y agoIt is common, but as dns over https becomes standard it won’t be able to be.
- organsnyder 8y agoWhy would that preclude the ability to host certain zones only on internal DNS servers?
- yjftsjthsd-h 8y agoIt appears that some clients just universally query third-party servers, thereby giving out a list of every host name that they're looking for to a third party.
- tracker1 8y agoConsidering how many office networks use 192.168.. or 10..., I'm not sure that in practice you're REALLY* leaking all that much... I mean it's a little bit of a hint, but if you can control a bridged device with internal network access, you'll probably get the same info (roughly) from a port scan. The "breach of info" is overblown imho.
- pawal 8y agoIn practice internal only DNS names will always leak to external resolvers in one way or another. Pretending otherwise is very naive.
- ce4 8y agoThis is not exclusive to LetsEncypt - you'll get this with any official CA that participates in CT. You could use a private CA or official wildcard certs.
- peterwwillis 8y agoYep. Until Google abandons CT or the industry grows a pair (and a brain) we're stuck with not being able to use stock web PKI for confidential internal hosts.
- devrand 8y agoI disagree. CT has exposed a ton of issues in PKI and held CAs accountable. If you're hostnames are sensitive then Web PKI is not the correct solution, like you have identified. At that point you should be running an internal PKI and be responsible for running/maintaining to whatever standards you deem fit. Making Web PKI less transparent to fit your use case is not the answer.
- peterwwillis 8y agoNo, CT is stupid. Transparency is not necessary for security, but preventing invalid certificate issuance is. CT doesn't even do that. It just alerts you to it after the fact - but you have to monitor it to be useful anyway, so it's actually opt-in security. And it creates annoying side-effects like this public/private thing, or actually preventing CAs from issuing certs when their CT endpoints go down. It's ridiculous. CT is a bank alarm that goes off after the bank's been robbed, that you won't hear unless your radio is tuned to pick it up.
- JoshTriplett 8y ago> Transparency is not necessary for security, but preventing invalid certificate issuance is. > CT is a bank alarm that goes off after the bank's been robbed > There are better solutions How do you propose to prevent invalid certificate issuance? In particular, how do you propose to observe whether any CA accepted by browsers has issued a certificate to someone other than you for a domain you control, if you presume that there are malicious actors willing and able to compromise certificate authorities? Your threat model is nation-states. Assume someone else is willing to put up the cash, what's your solution that works strictly better than certificate transparency, and in particular (since you called attention to it) prevents invalid certificate issuance rather than just logging it for subsequent audit (and CA trust revocation)?
- zackboe 8y agoChrome and Safari (the only browsers enforcing CT logs at the moment) can both be configured to disable enforcement for certain URLs. https://twitter.com/sleevi_/status/1102306640072716288 https://twitter.com/sleevi_/status/1102306640072716288
- JonathonW 8y agoBut you can't stop Let's Encrypt from publishing CT records for your internal-only certificates.
- johntash 8y agoI'd rather use an internal pki for internal-only stuff at work, but for my own homelab I do use a wildcard cert that is something like * .homelab.mydomain.com. Sure, someone can see you have internal servers under * .homelab.mydomain.com, but they aren't going to know what specific hostnames exist there.
- deleted 8y ago[deleted]
- mholt 8y agoIf you're writing Go, using the DNS challenge is really easy with CertMagic (disclosure: my library). And if you're running a web server, believe me when I say that the best solution is one that is integrated directly: use Caddy. It will spare you some grief and frustration. Both Caddy and CertMagic support nearly 3 dozen DNS providers, and adding your own is relatively easy as well. CertMagic: https://github.com/mholt/certmagic https://github.com/mholt/certmagic Caddy: https://caddyserver.com https://caddyserver.com
- archgoon 8y agoThe article doesn't discuss this; what are the advantages to using Let's Encrypt for internal services over deploying internally signed certificates? Are there any disadvantages to using Let's Encrypt? The rate limits alone seem to be a potential danger if they need to reissue new certificates for their 65,000 servers.
- mholt 8y agoThe convenience of not having to manage your own CA (doing so is complex and nuanced).
- wereHamster 8y agoIf you use your own CA you have to make sure all your apps trust it. With the public CAs that's not a problem because all the standard libraries usually trust them, and often 'apt-get install ca-certificates' is all you need to do.
- mjlee 8y agoIf you are going to manage your CA and don't have an expert available, you could do much worse than https://www.vaultproject.io/docs/secrets/pki/index.html https://www.vaultproject.io/docs/secrets/pki/index.html Even if you do have an expert available, it's a sound choice.
- Ajedi32 8y agoThe obvious advantage is that you don't need to deploy your root cert on client devices, or manage your own PKI. Reissuing certificates shouldn't be a problem since Let's Encrypt has a rate limit exception for renewals, though if I were managing 65,000 servers I'd be a little hesitant to put that kind of burden on Let's Encrypt's infrastructure without contacting them first, just as a matter of courtesy.
- binwiederhier 8y agoWe didn't just contact them before regarding a rate limit increase, we are also a silver sponsor and pay/donate yearly. See https://letsencrypt.org/sponsors/ https://letsencrypt.org/sponsors/
- Animats 8y ago(Deleted. Wrong.)
- ibotty 8y agoYou would not, but you don't give your private keys to them. Why do you think so?
- Ajedi32 8y agoACME doesn't give the issuing CA access to your private keys. That'd be a terrible idea.
- detaro 8y agoSince when do you give a CA your private keys to get a certificate signed?
- Animats 8y agoRight, sorry, forgot how the key generation works.
- organsnyder 8y agoWhy would you be sending your private keys to your certificate authority?
- Ajedi32 8y agoThis is a good conceptual overview of how you'd accomplish this with ACME. From a more concrete perspective, I'd suggest using acme-dns[1]. It's fairly easy to set up and is supported by a number of existing ACME clients like ACME.sh and Certbot (via an authentication hook). [1]: https://github.com/joohoi/acme-dns https://github.com/joohoi/acme-dns
- blhack 8y agoFor those curious why this is so important: A service I sell is a callcenter SAAS. It uses twilio to make outbound calls, but this won't work on internal servers. (I prefer to deploy the service as appliances that get shipped to the customers). So this is a great thing for me, since it means I can start offering my customers the ability to make outbound calls from their appliances.
- Naac 8y agoTo address the security issue of Certificate Transparency mentioned by another comment, you can combine Lets Encrypt wildcard certs, and SSL termination at the LB. Set up a Lets Encrypt wildcard cert, and use it on your internal LB. Set up your internal DNS to autoresolve custom internal hostnames. For example, if you own example.com, set up internal hostnames such as test1.example.com, test2.example.com These hostnames are internal, so they won't be resolvable ( or known about ) outside your private network. The wildcard cert combined with SSL termination at the LB means all your hosts now have ssl certs.
- kissgyorgy 8y agoIn this case, all your internal servers would have the same cert with the same private key. If one appliance is compromised, every other would be compromised. Not a good idea.
- Naac 8y agoCorrect. The solution I suggested above is for home/personal use. But if you're doing anything for production, you really shouldn't be using LB ssl termination anyway, and might even benefit from running your own internal CA service.
- shittyadmin 8y agoI wish there was a way to just get a wildcard and use it to issue "sub certificates" of some sort. ie: *.internal.mycompany.com could be used to issue valid certs for git.internal.mycompany.com...
- mholt 8y agoAs opposed to just using the wildcard itself?
- ozim 8y agoYou know wildcard is not working two levels down, you are getting .yourdomain.com but not .*.yourdomain.com. But yea maybe parent poster could consider scheme: git-internal.yourdomain.com.
- michaelt 8y agoIf one of Google's servers gets hacked, I'd rather the hacker get the mail-sor-f41.google.com certificate than the *.google.com certificate :)
- mholt 8y agoBut certificates are PUBLIC keys...
- majewsky 8y agoIt's evident that GP is referring to the private keys.
- mholt 8y agoNo, it's not.
- 420codebro 8y agoLet's say a single VM or docker container has a break-out exploit. They get the private key for (whatever).domain.tld. It would be nice to be able to issue whatever yourself, rather than having to A) do wildcards or B) expose every single record to CT. But I personally don't tie privacy to DNS. Best systems I've seen which handle sensitive stuff have very ambiguous DNS names which you would never guess what they really are.
- misiti3780 8y agoAnyone have a decent process / library for renewals ? Let Encrypt is great but the certs need to be renewed every 3 months which is annoying.
- kissgyorgy 8y agoYou should automate the process and not manually request certificates. They are saying this for a long time. Every client can auto-renew.
- Hackbraten 8y agoI’ve been trying to automate this for a while but don’t know how to do it. Am using split DNS with BIND. The device in question accepts a private key and certificate chain but won’t allow arbitrary software to run so I need to do a manual TXT challenge every 3 months, which is super annoying. Any pointers?
- kissgyorgy 8y agoLego [1] can do DNS (dns-01) challenge automatically, you can run it on a different device and transfer the certs to the device which needs it. This way you expose the secret key to some danger (secret keys should never leave the requester machine), but might be acceptable risk, it depends. [1]: https://github.com/xenolf/lego https://github.com/xenolf/lego
- Hackbraten 8y agoThanks! Yes, the risk is acceptable because the appliance can’t generate a key pair by itself. All machines involved (including the DNS server) are on the same LAN and physically under my control. Took me a while to figure out that Lego indeed supports BIND (via the RFC2136 DNS provider).
- tialaramex 8y agoLego can use a CSR, so you don't need to send private keys anywhere. (Secret keys and private keys aren't the same kind of thing. lyrics to U2's "The Fly" helps me remember, "A Secret is something you tell one other person, so I'm telling you" - somebody else needs to know the _secret_ key for it to work, but nobody at all knows the private key) A CSR is a signed document which says "I want a certificate for this identity with this public key". The signature proves you know the corresponding private key, but that key isn't transmitted anywhere. If you're willing to have relatively long lived keys (2-3 years isn't too scary for 2048-bit RSA) you can generate a private key and a CSR once, and have Lego or similar CSR-capable software obtain certificates from Let's Encrypt every couple of months with that CSR.
- bogomipz 8y agoThe article states: >"To increase this number, you have to either request a higher rate limit or get your domain added to the public suffix list (note: adding your domain here has other implications!)." Can someone say what those other implications are? I'm not sure why the author would mention this and then fail to state what those are.
- Boulth 8y agoYou can start by reading bullet points here: https://publicsuffix.org/ https://publicsuffix.org/ In a nutshell adding your domain there makes the subdomains completely isolated (they can't set cookies for higher level domain). This is a good idea if you're hosting user pages as subdomains. See also PRs https://github.com/publicsuffix/list/pull/722 https://github.com/publicsuffix/list/pull/722
- bogomipz 8y agoThanks your point about the inability to set cookies for a higher level domain(shorter)is seen as a net positive though correct?
- deleted 8y ago[deleted]
- xenophonf 8y agoI've set up my configuration management system to drive certbot and to handle certificate distribution: https://github.com/irtnog/certbot-formula https://github.com/irtnog/certbot-formula https://github.com/irtnog/salt-states/blob/production/certbot/files/etc/renewal-hooks/deploy/send-event https://github.com/irtnog/salt-states/blob/production/certbo... https://github.com/irtnog/salt-states/blob/production/salt/files/reactors/certbot-deploy.sls https://github.com/irtnog/salt-states/blob/production/salt/f... There are a few bits I haven't published mostly due to laziness on my part, like how I handle certificate assignments, but it's all pretty straightforward. The hardest part has been developing the necessary configuration scripts for Windows. It takes some effort to get keying material installed in the appropriate certificate store, which you can see part of here: https://github.com/irtnog/salt-states/blob/production/wincert/init.sls https://github.com/irtnog/salt-states/blob/production/wincer... And then there's how things that use CryptoAPI/CryptoNG reference the desired certificate by SHA-1 thumbprint, which you can see part of here: https://github.com/irtnog/salt-states/blob/production/iis/certificates.sls https://github.com/irtnog/salt-states/blob/production/iis/ce... https://github.com/irtnog/salt-states/blob/production/rd-gateway/certificates.sls https://github.com/irtnog/salt-states/blob/production/rd-gat... In some cases there isn't a clean API for certificate installation: https://github.com/irtnog/salt-states/blob/production/rdp/init.sls https://github.com/irtnog/salt-states/blob/production/rdp/in... In other cases one must run the necessary PowerShell cmdlets under user accounts with the correct privileges, which is something I haven't quite figured out (e.g., for Exchange 2007 and newer). Sometimes the services in question require the keying material be structured in very specific and slightly odd formats (e.g., Splunk). Sometimes, I just stick a load balancer in front of it, terminate the TLS connection there, and call it a day (e.g., Tableau), perfect being the enemy of good and all. I've given a lot of thought to the question "how would an attacker use CT logs against me", but I think the probable losses of something untoward happening due to certificate transparency is extremely small compared to attackers getting onto my networks and eavesdropping on sensitive, internal comms. I'm also not confident in my ability to run my own internal CA, both because it'd be a lot of work to secure and because the bus factor would be so damn high. At least this gives the rest of the team some incentive to tie new things into the configuration management system because they get certificates for "free". Maybe there's a better way. I don't know.
- therealmarv 8y agoUsing DNS verification also for a couple of non critical production websites which are hosted on GitLab Pages. It's really nice not to upload a challenge file for doing so. Tutorial here: https://about.gitlab.com/2016/04/11/tutorial-securing-your-gitlab-pages-with-tls-and-letsencrypt/#comment-3926513589 https://about.gitlab.com/2016/04/11/tutorial-securing-your-g...
- zyberzero 8y agoI wrote some shell scripts a while back [0] for almost this purpose. I have a backup MX that does not run any Web services, but I wanted TLS anyway for the SMTP. It uses the let's encrypt client (certbot) and nsupdate so I can use any RFC2136 compatible DNS server, in my case, Bind. This has been running on the backup MX since I wrote it, without any hickups so far. This can also be used for. Internal services, since you do not expose anything to the internal machine. [0] https://github.com/zyberzero/certbot-rfc2136 https://github.com/zyberzero/certbot-rfc2136
- binwiederhier 8y agoAuthor here. Just wanted to comment on the Certificate Transparency issues that some of you have raised: My company (Datto [1]; we're hiring, see [2]) sells backup appliances which our customers place in their company network (much like you would place a router in your own network). Since we do not control anything other than our appliance inside our customers' networks, we went this route to provide a secure web interface to our appliance. And since the 65k servers/appliances (now more like 80k) are located in tens of thousands of different networks, leaking the internal IP isn't bad at all -- especially since there is no way to correlate them with the customer. For normal "internal servers" within a company, I'd probably recommend using a wildcard cert and an internal DNS server. Also: Yeyy, I'm on HN! [1] https://www.datto.com/ https://www.datto.com/ [2] https://news.ycombinator.com/item?id=19071727 https://news.ycombinator.com/item?id=19071727 or https://www.datto.com/careers/ https://www.datto.com/careers/
- mmalone 8y agoIt sounds like using Let’s Encrypt makes sense for your use case since people are accessing these appliances via a web browser, but for “normal internal servers” the right answer is almost definitely to use your own internal PKI, not to use wildcard certs. See: https://github.com/cloudflare/cfssl https://github.com/cloudflare/cfssl https://github.com/hashicorp/vault https://github.com/hashicorp/vault https://github.com/smallstep/certificates https://github.com/smallstep/certificates
- forgotmypw 8y ago> leaking the internal IP isn't bad at all -- especially since there is no way to correlate them with the customer. You seem to be making many assumptions here, which some may not agree with.
- freedomben 8y agoAgreed. While the secrecy of IPs should never be relied on as a security mechanism, keeping them secret can force the attacker to use a scan, which can trigger IDS and tip their hand. If the attack is going through a proxy host (a vulnerable app server proxying an attack at the database for example), an unknown DB IP address could make the attack infeasible.
- jareware 8y agoWe had a similar need a while back, and open sourced our solution: [alley-oop](https://github.com/futurice/alley-oop https://github.com/futurice/alley-oop) is a Dynamic DNS server with an integrated Let's Encrypt proxy, enabling easy HTTPS and WSS for web servers on a local network (LAN). It effectively automates the process that's described in the article. Once you have it set up (you need a few DNS entries and a tiny Go server running somewhere), using it is as simple as issuing an HTTP API call to your alley-oop server with the LAN IP you have, and the DynDNS domain you want associated with it, and getting a valid cert in return. You're then ready to spin up your HTTPS server with it, and start serving LAN traffic with a valid cert.
- gerdesj 8y agoThe article starts: Let’s Encrypt is a revolutionary new certificate authority ... and in my opinion fails at that precise point. I think that SSL/TLS itself is the correct starting point and not a particular implementation. The article could have been titled "Using TLS for internal servers". The LE implementation could have been one of a few. Don't forget that you can manage your own Certificate Authority lists and really ought to do so if you actually give a shit about IT security. Abrogating your responsibility to MS, Apple, Mozilla, Google etc is way too easy, dangerous and probably immoral.
- amluto 8y agoThere is an obvious technical solution that is IMO far superior: get all the browsers to support CA certificates that are scoped to a particular domain and its subdomains. Then get your organization one of these CA certs (which is no more dangerous than a wildcard certificate) and issue your own subdomain certificates. Sadly, I don’t think any browsers support such a thing.
- redsavagefiero 8y agoSelf signed internal CA for private use all day + mandatory client certs. Internal approved CA for business. Never see a need to change this. This Lets encrypt stuff is faddish to me.
- Proven 8y agoI agree. I read the article three times trying to grasp the freaking point and I can't see any. Why on earth go through all that trouble and not get any additional security or convenience... in fact the security is lower and there are more attack vectors compared to own CA. I don't get it.
- cpitman 8y agoIs there any open source self-hosted CA that supports ACME? Boulder (the LetsEncrypt system) has big disclaimers in its docs about it not being a good fit for most companies. In most large companies, issuing certificates remains a manual process. Generate a CSR, create a ticket, wait for response, download certificate, install on host. ACME seems like it should be the standard API for fixing this.
- tatersolid 8y agoSince version 2003, Windows Certificate Services has automated issuance and renewal to domain joined clients managed via Group Policy. It just works and many orgs use this for managing VPN certificates to fleets of Windows clients, as well as internal TLS server certs. It uses Kerberos/AD for validating certificate names. No ACME of course, but considering the MSFT certificate management protocol pre-dates ACME by more than a decade that’s not surprising. Traditional manual issuance after CR submission workflow is also supported for any OS.
- damageboy 8y agoIsn't this a whole lot of words for installing traefik with DNS challenge and let it do basically all of the work?
- virgakwolfw 8y agoWhy not set the A record for the external DNS server to something like 1.1.1.1 and the internal DNS server has the correct IP address (or a view on the same DNS Server). This way you won’t leak internal IP addresses to the Internet.
- isostatic 8y agoMy understanding is that intermediate CAs can be limited to issuing certificates only for specific domains. Is it not possible to get a root-signed intermediate CA for someone who can prove control over a domain? This would allow you to issue certificates for "xxx.internal.mydomain.com", without the need for a wildcard certificate, and without the need for using a public CA for every individual certificate? CT would still be a problem unless the CA could be flagged to "allow non-CT certificates", and the browser ignore those requirements as they ignore CT requirements for manually installed root certs The benefits to this is 1) there's no need to manually install a root certificate on each client device 2) Your internal domains are not reliant on an external CA
- throw0101a 8y ago> My understanding is that intermediate CAs can be limited to issuing certificates only for specific domains. Technically, X.509 allows for constraining things, [1] but from a practical perspective [2] it's not really implemented. [1] https://tools.ietf.org/html/rfc5280#section-4.2.1.10 https://tools.ietf.org/html/rfc5280#section-4.2.1.10 [2] https://security.stackexchange.com/questions/31376/ https://security.stackexchange.com/questions/31376/
- isostatic 8y agoThat thread does indicate things are changing though and it's becoming more and more accepted.
- ohiovr 8y agoMy solution for my own private intranet at home is to use nginx's built in ip filtering and a vpn so that only browsers that are behind my wan ip can view my internal sites. Another interesting thing is that you can have the non approved ip addresses "see" a totally different site on a url than those 'in the know' do. I'm not aware of any downsides to this method. If there is one, would be good to know.