17 ms·
Should you use Let's Encrypt for internal hostnames?
- CountDrewku 5y agoLast time I considered using it, it was a pia for IIS servers because you have to manually renew every 30 days. Has this changed?
- meepmorp 5y agoI've used https://smallstep.com/docs/step-ca/ https://smallstep.com/docs/step-ca/ as a CA internally, works well.
- mojzu 5y agoI've been using it too and it works well, particularly with Caddy to do automatic certificates with ACME where possible Plus all my services go through Tailscale, so although I am leaking internal hostnames via DNS, all those records point to is 100.* addresses
- chrisweekly 5y agoI'm a fan of both Caddy and Tailscale; any chance you have any devnotes to share on your setup?
- mojzu 5y agoMy notes were pretty rough but I've tried putting them into a gist here: https://gist.github.com/mojzu/b093d79e73e7aa302dde8e335945b2cd https://gist.github.com/mojzu/b093d79e73e7aa302dde8e335945b2... Which covers using step-ca with Caddy to get TLS certs via ACME for subdomains, and protecting internal services using client certificates/mtls I then install Tailscale on the host which is running the docker containers, and configure the firewall so that only other 100.* IP addresses can connect to ports 80/443/444. The combination of VPN+MTLS mitigates most of my worries about exposing internal subdomains on public DNS
- chrisweekly 5y agoAwesome, thanks!
- Proven 5y ago
- dolmen 5y agoTailscale+TLS: isn't it two strong layers of encryption?
- mojzu 5y agoYeah, it's probably overkill but I think the multiple layers would help in cases I misconfigured something or if an account someone uses to log into Tailscale was compromised. For example when I ran the containers on a linux host I discovered later docker was bypassing the firewall rules and allowing all connections, but it probably wasn't a big deal because of the MTLS (and the server was behind a NAT router anyway so it was only addressable within the local network)
- mrweasel 5y agoWhat I'd want is an internal CA, like step-ca, but have the certificates signed by a "real" CA, so I don't have to distribute my own root CA certificate.
- filleokus 5y agoThe dream would truly be an internal CA backed by a publicly trusted subordinate cert (limited to the domain you control). But afaik that can’t happen until the Name Constraint Extension is enforced by “all” clients.
- deleted 5y ago[deleted]
- throw0101a 5y ago> But afaik that can’t happen until the Name Constraint Extension is enforced by “all” clients. For those curious about this extension, see RFC 5280 § 4.2.1.10: * https://www.rfc-editor.org/rfc/rfc5280#section-4.2.1.10 https://www.rfc-editor.org/rfc/rfc5280#section-4.2.1.10
- corobo 5y agoWouldn't that allow you to issue certificates for Google.com? Correct me if I've misunderstood but for the sake of discussion pretend cert pinning doesn't exist, use another example domain if it's easier
- mrweasel 5y agoI'm not a 100% sure how certificates work. What I imagined would be possible is having a certificate for mydomain.com, which can be used to sign certificates for subdomains.
- mmalone 5y agoYou can put "name constraints" on an intermediate that, in theory, can restrict the intermediate to only signing certs for a particular subdomain. In theory, name-constrained intermediate certificate for `.example.com` would have no more authority than a wildcard certificate for `.example.com`. But, name constraints are enforced by "relying parties" -- HTTPS/TLS clients & servers that are validating certificates and authenticating remote peers. In practice, there's a risk that a broken/misconfigured relying party would trust a cert for google.com signed by an intermediate that's name constrained / only trusted to issue for `*.example.com`.
- gclawes 5y agoI've been setting this up for my homelab under .home.arpa, seems to work pretty well so far.
- metafunctor 5y agoThis is the correct answer ;) If you're going to run a serious internal network, you'll need the basic things like NTP, DNS, a CA server, and, yes, some kind of MDM to distribute internal CA certificates to your people. The real PITA is when you don't have these in place.
- magicalhippo 5y agoI've done it with a few key services like Home Assistant, using split-horizon DNS, and considered it less than ideal. However the alternatives suck as far as I know. I don't want to install my own CA certificate on all the various devices in the home, for instance, and keeping that up to date. With browsers making self-signing a PITA, what choices do I have?
- LeSaucy 5y agoNot just browsers, but also iOS/android.
- deleted 5y ago[deleted]
- imadethis 5y agoThis seems like a perfect use case for wild card certs, especially if you have internal sites on a different (sub) domain from your prod servers. Yes, multiple servers have the same private key, but when the alternative is self-signed or no encryption, that is an easy trade off for me.
- justusthane 5y agoI don't know how LE does it, but at least with DigiCert (and I assume other commercial CAs), servers sharing the same wildcard cert don't have to share a private key. You generate a separate CSR from each server, and then request a duplicate copy of the wildcard cert using that CSR. That way they can have different SANs as well.
- zrail 5y agoWildcard certs are (only?) issued from DNS-01 challenges. As long as the requester can satisfy the DNS challenge ACME doesn't care about key uniqueness.
- Spooky23 5y agoWith Digicert, you do a different API call “duplicate certificate” to avoid buying another cert unnecessarily. I would consider it to be a best practice to keep unique keys as an SOP as it discourages bad behaviors, like keeping private keys accessible on file servers or even mail.
- dsr_ 5y agoRight. If you control the DNS, you can point names at any IP address and get appropriate certs for them. Therefore, you must protect your DNS infrastructure.
- Hamuko 5y agoIsn't the need to protect your DNS infrastructure pretty obvious anyways even when ignoring certificate validation?
- filleokus 5y agoSadly, the answer is probably no (for the information leakage mentioned in the article). But having an internal (even ACME API-supporting) CA is no walk in the park either. If you can swallow the trade off and design with publicly-known hostnames, I would highly recommend it. There’s always some annoying device/software/framework requiring their own little config dance to insert the root cert. Like outbound-proxy configuration, but almost worse. I don’t even want to imagine what would happen if/when the root key needs to be rotated due to some catastrophic HSM problem.
- michaelt 5y ago> Sadly, the answer is probably no (for the information leakage mentioned in the article). Eh, even in large organisations of expert IT users, the internal CA ends up training users to ignore certificate warnings. Sure, maybe the certificate is set up right on officially issued laptops - but the moment someone starts a container, or launches a virtual machine, or uses some weird tool with its own certificate store, or has a project that needs a raspberry pi, or the boss gets himself an ipad? They'll start seeing certificate errors. IMHO the risks created by users learning to ignore warnings are much greater than the risks from some outsider knowing that nexus.example.com exists.
- flatiron 5y agoIf you have a large organization your containers are based off the orgs containers which has the CA in it. Same with VMs, Java, .Net, etc.
- Mister_Snuggles 5y ago> The only real answer to this is to use Wildcard Certificates. You can get a TLS certificate for *.internal.example.com Does Let's Encrypt support Subject Alt Names on the wildcard certs? My experience suggests that wildcard certs work, but require a SAN entry for each "real" host because browsers don't trust the CN field anymore. e.g., my *.apps.blah cert doesn't work unless I include all of the things I use it on - homeassistant.apps.blah, nodered.apps.blah, etc. Do Let's Encrypt certificates have something special that negates this requirement? Or am I completely wrong about the SAN requirement?
- vladvasiliu 5y agoI use a few wildcard certs from Amazon, and they work well on Firefox, Safari and Chrome.
- Intermernet 5y agoWildcard certs from LE work fine for internal domains. I've been using one for a while now. I had to set up some cron jobs to copy them around and restart some services, but it seems to be working well.
- formerly_proven 5y agoThe whole point of a wildcard certificate is that you don't have to exhaustively list all covered hostnames.
- gregmac 5y agoThis sounds like something is broken in your client (or maybe server config)? I use Let's Encrypt wildcard certs quite extensively, both in production use at $dayjob and on my home network, and have never encountered anything like this. The only "trick" to wildcard certs is one for .apps.blah won't be valid for apps.blah. The normal way to handle this is request one with SANs .apps.blah and apps.blah. Similarly, it won't work for sub1.sub2.apps.blah. I don't run setups like this myself but if you need it I'd recommend using a separate *.sub2.apps.blah for that, mainly due to the potential for DNS issues when LE is validating. Same thing with multiple top-level domains. The reason is when renewing if one of N validations fail, your certificate gets re-issued without the failed domain which then means broken SSL. If you have completely separate certificates and validation of one fails the old (working) version stays in place. With normal renewals happening at 30 days before expiry, this means you have 29 days for this to resolve on its own, manually fix, etc, and LE even emails you a few days before expiry if a certificate hasn't been renewed.
- cure 5y ago> OK, so you decide to have an internal DNS - now the whole world knows you have doorbell-model-xyz.myhome.example.com! Uhm, or you use split horizon DNS? Who in their right mind would leak all their internal DNS names into a public DNS zone?
- edent 5y agoSorry for the poor wording on my part. I meant that if you issue a LE Cert for your doorbell, and give it a "sensible" name, the name will appear in the CT Log.
- cyberge99 5y agoNamed certs have the hostnames they’re valid for in the Certificate itself. “View Certificate” in a browser, or openssl sclient on cli will show you.
- icedchai 5y agoI don't bother with split horizon DNS for my home network.
- deleted 5y ago[deleted]
- teekert 5y agoThat's in the article, Let's encrypt leaks them for you, if you use them for your intranet.
- ChrisMarshallNY 5y agoThis is an interesting topic, for me. I write iOS apps, and iOS requires that all internet communications be done with HTTPS. It is possible to use self-signed certs, but you need to do a bit of work on the software, to validate and approve them. I don't like doing that, as I consider it a potential security vector (you are constantly reading about development code that is compiled into release product, and subsequently leveraged by crooks). I am working on a full-stack system. I can run the backend on my laptop, but the app won't connect to it, unless I do the self-signed workaround. It's easier for me to just leave the backend on the hosted server. I hardly ever need to work on that part.
- 0x0 5y agoIf you create a custom SSL CA, you can add that CA to your ios devices and simulators, and they will trust your backend served with an SSL certificate issued by your custom CA, no app modifications needed. (On modern Android, this does not work out of the box - it requires the custom SSL CA fingerprints to be added to a network configuration file embedded in the app - but you could always use gradle flavors and only add it to your debug/development builds)
- chedabob 5y agoFor the project I'm working on currently, I use Charles Proxy's "Map Remote" function to map our UAT server's HTTPS url to my local machines HTTP URL. Also ngrok.com works really well if you need to give other people access to your dev environment.
- ChrisMarshallNY 5y ago> I use Charles Proxy's "Map Remote" function to map our UAT server's HTTPS url to my local machines HTTP URL. This looks really interesting. Thanks! I'll see if I can get away with it.
- tzs 5y ago> I write iOS apps, and iOS requires that all internet communications be done with HTTPS What if the app is on the same network as the server? I've got a Denon A/V receiver that has an HTTP interface and the Denon iOS app is able to talk to it. I've watched this via a packet sniffer and it definitely is using plain HTTP.
- thrower123 5y agoIs it that hard to setup an internal CA? I have no idea what I'm doing, and I managed one for years until we moved offices and ditched our LAN.
- jeremyjh 5y agoThe hard part is getting the root certificate in the trust store on every device in your organization.
- tzs 5y agoWorse, it is often not the trust store on every device. It is often multiple trust stores on a device. The OS might have one. Each browser might have its own. For a developer, each language they use might need separate configuration to get its libraries to use the certificate.
- midasuni 5y agoActive Directory for windows, MDM for OSX and phones, custom package for linux.
- midasuni 5y agoThat should worry the hell out of you. If you could install CAs only for a certain domain (default to the name constraints but actually set in the browser/Os) that would be fine, but installing a CA gives anyone with access to that CA the ability to make pretty much any valid cert, and your potential lack of security raises flags
- mcgeez 5y agoI like the wildcard certificates option, however I have not been able to find an easy solution to distribute those certificates to every host I have internally. Is this usually done manually? is there some equivalent to acme.sh? The kind of hosts I have are OPNSense router, traefik servers, unifi controller etc.
- jasonjayr 5y agoFor any device that has a web interface, and no way of updating the cert automatically built in or via an API, you'd probably have to automate the process with something like puppeteer. https://www.npmjs.com/package/puppeteer https://www.npmjs.com/package/puppeteer
- willis936 5y agoIf you have root ssh on each machine you can make rsync cron jobs. Imo it's reasonably secure if you spend the time setting up ssh keys and disabling password auth after.
- dspillett 5y agoMy method is manual-ish¹. One VM is in charge of getting the wildcard certificates. Other than answering DNS requests for validation and SSH it has no public face. Each other machine regularly picks up the current outputs from there via SFTP weekly and restarts what-ever services. I'm not running anything that I need near-perfect availability on ATM, so it is no more complex than that. If wanting to avoid unnecessary service restarts check the for changes and only do that part if needed, and/or use services that can be told top reload certs without a restart. This does mean I'm using the same key on every host. If you want to be (or are required to be) more paranoid than that then this method won't work for you unmodified and perhaps you want per-name keys and certs instead of a wildcard anyway. For extra carefulness you might even separate the DNS service and certificate store onto different hosts. Not sure how you'd do it with unifi kit, my hosts are all things I can run shell scripts from cron on running services like nginx, Apache, Zimbra, … that I can configure and restart via script. [1] “manual” because each host has its own script doing the job, “ish” because once configured I don't need to do anything further myself
- 5y ago
- Hamuko 5y agoI'm using a wildcard certificate and CNAME records to interal hostnames, it works pretty nicely for my use case. I don't need to leak out a map of my hostnames and I don't need to do a full split-horizon DNS. So if I want to encrypt traffic to "service1.example.com", "service2.example.com" and "service3.example.com" that all run on server A, I'll make three CNAME records that all point to "server-a.internal", and I'll just resolve "server-a.internal" in my local network. Obviously, anyone can query what "service1.example.com" points to, but they won't figure out anything beyond "server A".
- nimbius 5y agoyou should not use wildcards or letsencrypt for internal authentication as its insecure for a few reasons. 0. implicit reliance on a network internet connection means any loss of ACME to the letsencrypt CA makes renewal of the cert or OCSP problematic. if the internet goes down, so does much of the intranet nonreliant upon it. 1. wildcard certs make setting up an attack on the network easier. you no longer need an issued cert for your malicious service, you just need to find a way to get/use the wildcard. you should know your services and SANs for the certs. these should be periodically audited.
- Macha 5y ago1. Renewal is scripted to try every day for 30 days in advance with most common utilities. If lets encrypt and all other acme hosts are down for 30 days, I think you have bigger issues. 2. If you can't secure a wildcard cert, how does the same problem not apply to a root CA cert, which could also then do things like sign google.com certs that your internal users trust, which feels strictly worse. (I know there are cert extensions that allow restricting certs to a subdomain, but they're not universally supported and still scoped as wide as a wildcard cert).
- nimbius 5y agoOCSP is still a problem, as youll need to either proxy a local ocsp response during outages or disable validation entirely. microservices in an aws partial outage, for example, would suffer here. a root CA cert is stored in a gemalto or other boutique special HSM. it has an overwhelming security framework to protect it (if its ever online.) security officers to reset pins with separate pins, and an attestation framework to access its functions through 2 or more known agents with privileges separated. even the keyboard connected to the device is cryptographically authenticated against the hardware to which it connects. commonly your root is even offline, unavailable (locked in a vault) and only comes out for new issuing CA's.
- Macha 5y ago> a root CA cert is stored in a gemalto or other boutique special HSM. it has an overwhelming security framework to protect it (if its ever online.) security officers to reset pins with separate pins, and an attestation framework to access its functions through 2 or more known agents with privileges separated. even the keyboard connected to the device is cryptographically authenticated against the hardware to which it connects. There are many organisations not large enough to justify this setup, for which Lets Encrypt is clearly safer than a custom root CA.
- snthd 5y agoCan lets encrypt issue multiple wildcard certs for different subdomains like *.banana.example.com and *.grapefruit.example.com Then you could give each server a different wildcard cert without exposing the full name to the certificate log: exchange.banana.example.com log4j.grapefruit.com Ugly, but functional. Alternatively should the certificate transparency log rules be changed to not include the subdomain? Maybe what matters is that you know that a certificate has been issued for a domain, when, and that you have a fingerprint to blacklist or revoke. Knowing which actual subdomain a certificate is for is very convenient, but is it proportionate?
- schoen 5y ago> Alternatively should the certificate transparency log rules be changed to not include the subdomain? Maybe what matters is that you know that a certificate has been issued for a domain, when, and that you have a fingerprint to blacklist or revoke. Knowing which actual subdomain a certificate is for is very convenient, but is it proportionate? That was a big debate in the CA/B Forum when CT was created; the current behavior is a deliberate choice on the part of the browser developers, which they will probably not want to revisit.
- _8j50 5y agoInternally even small companies should be doing some PKI for device enrollment and email security. If you manage your own dns you need a pki infra as well imo
- tomc1985 5y agoWhy not just be your own signing authority for internal domains? You can propagate your toplevel public cert with most enterprise network provisioning tools.
- YPPH 5y agoRunning your own PKI is fairly straightforward, particularly with tools like cfssl at your disposal. But running your own PKI properly is quite hard. Let's Encrypt gives you top tier PKI management for $0.
- justin_oaks 5y agoHow do you define "properly"? What are some of the things someone can do wrong that Let's Encrypt does correctly?
- YPPH 5y agoRoot certificate stored on offline HSM and intermediates on secure infrastructure. FIPS compliance. (Relatively) reliable revocation services. [See note 0] The result is security of issuance, that is near complete confidence that certificates will only be used for controlled domains (not necessary if you want to MITM of course). Also, ACME is generally easier and more reliable than other certificate rollover processes I've seen. I'm not sure if there's in-house PKI tools supporting it? Depends on your organisation size though. Maybe your in-house PKI is fine, but it's not for everyone! [Note 0] Revocation is of course a mess. Let's Encrypt isn't without fault either, particularly when used internally, since OCSP responders will need to be accessible from client devices.
- Spivak 5y agoI mean sure but an org doesn’t really need that much security. If you’re not taking that much care with your API keys and db passwords then you probably don’t need it for certs either. Keep your root CA offline and in an air gapped backup, issue team specific intermediates with med length and keep your endpoint certs short. You need as much security on your CA as the accounts in your org with the authority to replace them with your provisioning tools.
- cpach 5y agoInteresting question. Quite complex. IMHO there is no clear right or wrong here.
- StreamBright 5y agoI think using DNS for the intranet nodes is generally a good idea. It allows HTTPS to be seamless.
- justin_oaks 5y agoWe have an internal certificate authority for internal domains at my job. We add the root CA certificate to each desktop or server through an endpoint agent that runs on every machine. That agent is used for monitoring, provisioning users, and even running arbitrary commands. The article mentions BYOD (bring your own device) but we don't allow personal devices to connect to internal services, so this isn't an issue for us. You can use something like EasyRSA to set up an internal certificate authority and generate server certificates signed by that certificate authority. I started using plain old OpenSSL for generating certificates, which EasyRSA uses under the hood, but I would have liked to start by using EasyRSA in the first place. By the way, EasyRSA still isn't that easy, but it's better than using OpenSSL directly.
- throw0101a 5y ago> We have an internal certificate authority for internal domains at my job. We add the root CA certificate to each desktop or server through an endpoint agent that runs on every machine. One challenge to this is some software doesn't use the operating system's CA chain by default. A lot of browsers use their own internal one and ignore what the OS does (by default).
- YPPH 5y agoChrome, Edge, Safari and (god forbid) IE will use system certificate stores. Firefox was a challenge. But my understanding is that now, on Windows, it will now import enterprise root certificates from the system store automatically. https://bugzilla.mozilla.org/show_bug.cgi?id=1265113 https://bugzilla.mozilla.org/show_bug.cgi?id=1265113 https://support.mozilla.org/en-US/kb/how-disable-enterprise-roots-preference https://support.mozilla.org/en-US/kb/how-disable-enterprise-...
- RealStickman_ 5y agoOn Linux Firefox imports system certificates automatically, but shows a warning that the certificate is not trusted by Mozilla.
- 5y ago
- binwiederhier 5y agoMy company uses Let's Encrypt extensively for many thousands of customers edge devices which live in their own LAN. As long as the hostnames are random or at least not too telling there's pretty much nothing that you're leaking. Except for the internal IP address (10.x, 192.x,) and how many servers you have. If you can live with that then it's perfectly fine. I wrote about it a few years ago: https://blog.heckel.io/2018/08/05/issuing-lets-encrypt-certificates-for-65000-internal-servers/ https://blog.heckel.io/2018/08/05/issuing-lets-encrypt-certi...
- reincarnate0x14 5y agoIf you have split DNS you're not even leaking internal addresses, the public name record just has to exist.
- throw0101a 5y ago> […] the public name record just has to exist. Specifically a TXT record for _acme-challenge has to exist for the requested hostname. Or a CNAME of the requested hostname pointing somewhere else that you control: * https://dan.langille.org/2019/02/01/acme-domain-alias-mode/ https://dan.langille.org/2019/02/01/acme-domain-alias-mode/ * https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mode https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo... * https://www.eff.org/deeplinks/2018/02/technical-deep-dive-securing-automation-acme-dns-challenge-validation https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se... No A (or AAAA) records needed.
- Karunamon 5y agoVault makes a pretty decent internal PKI tool, even if the UI is limited. (https://www.vaultproject.io/docs/secrets/pki https://www.vaultproject.io/docs/secrets/pki)
- tedivm 5y agoThis is one area where I think AWS does a huge disservice by making their Private CA so expensive ($400 a month + cost of certificates). This ends up pushing people to use public domains instead of private ones, or relying on other solutions outside of AWS. If the cloud companies would make it as easy to get private certificates as public ones you wouldn't see as many issues like this.
- infogulch 5y agoSeveral comments here mention running your own CA. Maybe that could be a signed intermediate CA with the Name Constraint extension [0] (and critical bit?), but one roadblock on this path is that allegedly Apple devices do not support that extension (edit: actually this was fixed! see reply). You there, @ LetsEncrypt? To address the article a recent related discussion, "Analyzing the public hostnames of Tailscale users" [1], indicates in the title one reason you might not want to use LE for internal hostnames. There was a discussion about intermediate CAs there as well [2] with some more details. [0]: http://pkiglobe.org/name_constraints.html http://pkiglobe.org/name_constraints.html [1]: https://news.ycombinator.com/item?id=29579806 https://news.ycombinator.com/item?id=29579806 [2]: https://news.ycombinator.com/item?id=29614971 https://news.ycombinator.com/item?id=29614971
- ArchOversight 5y agoApple devices support the Name Constraint extension just fine. I've deployed a bunch of internal CA's with Name Constraint and Apple's macOS/iOS/iPadOS block certs that are signed for anything outside of the constraints. As is intended. AFAIK the Apple bug was fixed in macOS 10.13.3 from what I can find online. [1] [1]: https://security.stackexchange.com/questions/95600/are-x-509-nameconstraints-on-certificates-supported-on-os-x https://security.stackexchange.com/questions/95600/are-x-509...
- infogulch 5y agoThat's great to hear! I'd only heard secondhand, so I updated my comment to reflect this detail. Also I found https://bettertls.com https://bettertls.com publishes details about which TLS features are supported on different platforms over time, and it appears that the latest test in Dec 2021 shows most platforms support name constraints. With that roadblock evaporated, I think this would be the perfect solution to a lot of organization- and homelab-level certificate woes. I'd really like to hear from a domain expert on how feasible it would be to automate for free public certs, ACME-style.
- wolrah 5y ago
- pgroves 5y agoAnother nuisance is that unencrypted port 80 must be open to the outside world to do the acme negotiation (LE servers must be able to talk to your acme client running at the subdomain that wants a cert). They also intentionally don't publish a list of IPs that LetsEncrypt might be coming from [1]. So opening firewall ports on machines that are specifically internal hosts has to be a part of any renewal scripts that run every X days. Kinda sucks IMO. [1]https://letsencrypt.org/docs/faq/#what-ip-addresses-does-let-s-encrypt-use-to-validate-my-web-server https://letsencrypt.org/docs/faq/#what-ip-addresses-does-let... UPDATE: Apparently there is a DNS based solution that I wasn't aware of.
- duskwuff 5y agoOnly true if you're using HTTP validation. Use DNS validation instead and this isn't an issue.
- pgroves 5y agoFair enough. Although that seems rather complicated for those of us just trying to get a quick cert for an internal host. The LetsEncrypt forums are full of this discussion: [1] https://community.letsencrypt.org/t/whitelisting-le-ip-addresses-ranges-in-firewall/45190/6 https://community.letsencrypt.org/t/whitelisting-le-ip-addre... [2] https://community.letsencrypt.org/t/whitelist-hostnames-for-certbot-validation/115842/2 https://community.letsencrypt.org/t/whitelist-hostnames-for-... [3]https://community.letsencrypt.org/t/letsencrypt-ip-addresses-its-actually-pretty-important/129760/2 https://community.letsencrypt.org/t/letsencrypt-ip-addresses...
- wielebny 5y agoThat's not true. You can validate domains using dns-01, without exposing hosts.
- detaro 5y agoand even with HTTP challenge you don't have to expose the host directly, but e.g. can copy the challenge response to a public webserver from the internal host or from a coordinator server.
- 0xbadcafebee 5y agoRunning your own private CA is a great way to cause problems for yourself down the road (just ask anyone with a 5 year and 1 day old Kubernetes cluster). But I also don't want to be dependent on a 3rd party for my internal services. I want a better solution: not as annoying as a private CA, and not dependent on 3rd parties. I want to deploy apps that use certs that don't expire. When they should be rotated, I want to do them on my own time. And I want a standard method to automatically replace them when needed, that is not dependent on some cron job firing at the correct time or everything breaks. Cert expiration is a ticking time bomb blowing up my services just because "security best practice" says an arbitrary, hard expiration time is the best thing. Security is not more important than reliability. For a single external load balancer for a website, we deal with it. But when you have thousands of the little bastards in your backend, it's just ridiculous.
- tatersolid 5y ago> Security is not more important than reliability. Yes, it is. In most cases Confidentially > Integrity > Availability. Systems should fail-safe. There are some scenarios such as medical devices where integrity or availability trump confidentiality. But most information systems should favor going offline to prevent a breach of confidentiality or data integrity.
- marcosdumay 5y agoI will never understand the obsession people have with hiding their private server names. If somebody gets any access to your local network, there are plenty of ways to enumerate them, and if they can't get access, what's the big deal? I get that you may want to obfuscate your infrastructure details, but leaking infrastructure details on your server names is quite a red flag. It should really not happen. (Instead, you should care about the many, many ways people can enumerate your infrastructure details without looking at server names.)
- jon-wood 5y agoOne of the examples given wasn't a server name, it was leaking potentially confidential information via the domain olympics-campaign.staging.example.org - in many environments its fine if people know project names, but NDAs are a thing, and you could end up in hot water if you accidentally leak a partnership between two companies before it's been announced.
- marcosdumay 5y agoWell, if instead of making a lot of effort in hiding your names you just didn't, you wouldn't use a name like that. Every single person that connects to any of your networks (very likely the sandboxed mobile one too) can find that name. Basically no place hides it internally. There is very little difference between disclosing it to thousands of the people that care the most about you and disclosing it to everybody on the world.
- ttyprintk 5y agoThe other examples are better. Say a never-before-seen name appears, cisco520.internal.foo.bar. Suddenly, a well-formed email appears, “Re: Cisco Support Ticket #7779311” about some additional steps to provision your new appliance. It is trivial to automate that phish by crawling the CT log.
- marcosdumay 5y ago
- AtNightWeCode 5y agoA public CA is for having a third-party entity so two different parties do not need to trust each other. So, the answer is no. Why would you even consider this for internal communication?
- jopsen 5y agoInstalling a root CA on devices is risky. From the article: > It means your employees aren't constantly fighting browser warnings when trying to submit stuff internally. If your employees gets a habit of ignoring certificate warnings then you have much bigger problems than leaking internal domain names.
- AtNightWeCode 5y agoClients should not ignore the certificate warnings. You install the certificates on the client machines.
- fulafel 5y agoDNS names are public by nature. Split horizon, private roots, private CAs etc are a sign you are trying to bend things backwards. Just don't use sensitive DNS names.
- walrus01 5y agodisagree on that - it's entirely possible to have an openssl private root CA and private DNS that doesn't talk to the internet at all and exists in RFC1918 IP space with no gateway or route to the outside world. not just a matter of ACLs on things like DNS servers but those same servers/VMs not even having interfaces that have any way to get traffic to a global routing table. split horizon I agree is risky.
- normaler 5y agoI bought an extra domain for our internal network. $COMPANY_SHORT_FORM.network Works really well and wie no longer have the issue of deploying root certs in devices.
- walrus01 5y agoI run my own internal CA. Would not recommend to anyone that they use publicly-valid letsencrypt certs for internal hostnames, since certificate issuance transparency logs are public and will expose all of the hostnames of your internal infrastructure.
- ClumsyPilot 5y agoMy life experience has taught me that it's better to have an imperfect, but simple solution with known limitations (in this case LetsEncrypt), than an ideal solution that you can't configure correctly and do not fully understand (internal CA for a small team). The former give you known limitations, the latter work fine for a while and you get a great feeling, and then disaster strikes out of the blue. The same problem plagues IoT solutions and home networking - there are no industry-accepted frameworks to enable encryption on Lan like we do on the real internet. Thrre is no way to know that I connect to my home router or NAS when i type in it's address. This is an area where we have kind of failed as an industry
- rusk 5y agoI read somewhere a while ago that LE are working on what’s called “intermediate CA” [0] which would solve the problem. Apparently from a regulatory standpoint there are some questions around abuse that need to be answered before they can go ahead. The basic idea is that you can issue your own certificates based on the LE CA that is already recognised by the browsers. EDIT [0] https://community.letsencrypt.org/t/does-lets-encrypt-offer-intermediate-certificates/71957/3 https://community.letsencrypt.org/t/does-lets-encrypt-offer-...
- Ajedi32 5y agoWe're a long ways away from name-constrained intermediaries being viable from a regulatory and technical perspective. I'd explain, but commenter in a thread linked to the one you posted has a pretty detailed explanation already: https://community.letsencrypt.org/t/sign-me-as-an-intermediate-ca-for-my-domain-with-name-constraint/17140/5 https://community.letsencrypt.org/t/sign-me-as-an-intermedia...
- infogulch 5y agoFrom that it looks like the main issue is regulatory requirements that force CAs to log all issued certificates via CT (certificate transparency) logs. Given that this is the very thing we're trying to avoid with a private CA ("CT" and "leaking internal hostnames" are functionally synonymous) we seem to be at an impasse at the level of base requirements. Maybe an IP constraint that restricts certs to only be valid in private IP spaces (10.*, 192.168.1.*, etc)?
- zozbot234 5y agoYou can still use wildcard certificates to avoid leaking the entirety of your private hostnames, while providing transparency around the "authority" portion of your domains.
- infogulch 5y agoFirst, that has it's own security drawbacks because now every service has access to a wildcard cert that is valid for any conceivable subdomain. Second, how is that better than an intermediate CA with a short life where the CA cert is CT logged? The cert path would still include that logged CA cert...
- stncls 5y ago> But there is a downside. The CT logs are public and can be searched. Firstly, [...] This bit me recently. I have a certificate for homelab.myname.com, and as any public-facing IP address, I get the expected brute force ssh login attempts for users 'root', 'git', 'admin', etc... But I was terrified (until I remembered about the public cert) to find attempts for users 'homelab' and 'myname' -- which, being my actual name, actually corresponds to a user. It's obviously my fault for not thinking this through, and it's not a terrible issue, but thinking I was under a targeted attack was quite the scare!
- aussieguy1234 5y agoIt's possible to configure DNS to make a public domain point to an internal IP and register a certificate for that domain. For example, you can register a certificate for local.yourcompany.com and point local.yourcompany.com to 127.0.0.1 to get HTTPs locally. The same could be done for internal network IPs. It wouldn't work well with Let's Encrypt because their bot would just end up talking to itself in this scenario. Of course you could also use my side project (expose.sh) to get a https url in one command.
- toddnni 5y agoMaybe a dump question, but was it so, that it is not only Let's Encrypt that uses Certificate Transparency Log, but all the other providers too? If so, then the decision is more like, whether to use a public or private certificate for an internal service.
- mac-chaffee 5y agoYep, we recently moved from DigiCert to LE and someone was alarmed at the certificate transparency logs, until we scrolled down the page to reveal the same logs from DigiCert. Wildcards hide it somewhat, but DigiCert charges per subdomain now, and every user thinks they need their own subdomain for some reason. So LE it is.
- spondyl 5y agoDon't say that! I managed to sign up to Starbucks Rewards before it launched here in New Zealand by looking at the staging certificates that were issues ;) Lots of fun stuff is possible but yeah, it's definitely something you should consider. Let's Encrypt allows wildcart certs from memory so you should probably use one of those per subdomain.
- rswail 5y agoSo people seem to be conflating the requirements of a CA (the thing that signs certificates and is considered the authority) and an RA (the registration authority). Running a CA that issues certificates isn't that hard. There are off-the-shelf solutions and wraparounds for openssl as well. Running an RA is hard. That's the part that has to check who is asking for a certificate and whether they're authorized to get one and what the certificate restrictions etc are. Then there's the infrastructure issue on the TLS users (clients & servers) that need to have the internally trusted root of the CA installed and need the RA client software to automagically request and install the necessary leaf and chain certificates. AWS has private CAs for $400/month, but if you want a root and then some signing intermediates, that's $400 for each (effectively the PCA is just a key stored in an AWS HSM and an API for issuing certificates). A real HSM will cost roughly a year of that service, but the management of that hardware and protecting it and all the rigmarole around it is very expensive. Every mobile phone and most desktops have a TPM that could be used for this, but having an API to access it in a standard way isn't that available.
- eternityforest 5y agoI just finished writing a long proposal: https://github.com/WICG/proposals/issues/43 https://github.com/WICG/proposals/issues/43 PKI is fairly awful and bad for internal anything, unless you have a full IT team and infrastructure. A much simpler solution would be URLs with embedded public keys, with an optional discover and pair mechanism. Browsers already have managed profiles. Just set them up with a trusted set of "paired" servers and labels, push the configs with ansible(It's just like an old school hosts file!), and don't let them pair with anything new. If you have a small company of people you trust(probably a bad plan!), or are a home user, just use discovery. No more downloading an app to set up a smart device. The protocol as I sketched it out(and actually prototyped a version of) provides some extra layers of security, you can't connect unless you already know the URL, or discovery is on and you see it on the same LAN. We could finally stop talking to our routers and printers via plaintext on the LAN and encrypt everywhere. We already use exactly this kind of scheme to secure our keyboards and mice, with discovery in open air not even requiring being on the same LAN. We type our sensitive info into Google docs shared with an "anyone with this URL" feature. It seems we already trust opaque random URLs and pairing quite a bit. So why not trust them more than the terrible plaintext LAN services we use now?
- southerntofu 5y agoSo the problem is our naming scheme is insecure so we ask untrustworthy 3rd-party entities to vet our certificates. The CA mafia isn't gonna give up their hard-earned monopoly easily (remember CACert?), and most client companies are happy to have for-profit CAs for insurance/policy compliance. Something like DNSSEC+DANE[0] is more reasonable but unfortunately unsupported by most programs. [0] https://datatracker.ietf.org/doc/html/rfc6698 https://datatracker.ietf.org/doc/html/rfc6698
- tptacek 5y agoNo, the reasoning here is broken. Even with a secure naming scheme, you'd still need certificates, because you have to verify the authenticity of the secure channel you bring up (usually TLS), not just the security of the name. Any way you slice it, you end up with a third party vouching for your TLS certificate.
- southerntofu 5y agoIn the case of DNSSEC keys are distributed with the zone, so the trust anchor is the DNS root. Of course, your parent zone could lie about your keys (just like it could lie about your other records), but don't you think since DNS is already an attractive attack vector (as it can vouch for CAs to publish a trusted certificate), relying on it exclusively for certificate distribution would reduce the overall attack surface? I'm not saying no trusted parties is the end goal (though Tor's onion or the GNU Name System work in this area), but maybe giving dozens of corporations/institutes the power to impersonate your server (from a client UX perspective) isn't the best we can do.
- tptacek 5y agoDNSSEC does the same thing (tradeoff: fewer corporations, but most of the important ones de jure controlled by governments, and no transparency logs†) --- you're just handing the CA role off to TLD operators. To the extent DNS is an attractive attack vector, DNSSEC doesn't actually do much to mitigate those attacks. Most DNS corruption doesn't come from the on-the-wire cache corruption attacks DNSSEC was designed to address, but from attacks directly on registrars. There's nothing DNSSEC can do to mitigate those attacks, but not having CAs tied directly to DNS does mitigate them: it means the resultant misissued certificates are CT-logged. If there was a huge difference in security from switching to DANE, this would be a different story. But in practice, the differences are marginal, and sometimes they're in the wrong direction. Two really big things happened in the last decade that influenced the calculus here: 1. WebPKI certs are now reliably available free of charge, because of LetsEncrypt and the market pressures it created. 2. Chrome and Mozilla were unexpectedly successful at cleaning up the WebPKI, to the extent that some of the largest CAs were summarily executed for misissuance. That's not something people would have predicted in 2008! But WebPKI governance is now on its toes, in a way that DNS governance is unlikely ever to be. (Cards on the table: I'd be a vocal opponent of DANE even if 1 & 2 weren't the case.) † Not only is there no CT for DANE, but there's unlikely ever to be any --- CT was rolled out in the WebPKI under threat of delisting from Mozilla and Chrome's root cert programs, and that's not a threat you can make of DNS TLD operators.