9 ms·

I really wish Let's Encrypt had a solution for those of us with lots of "internal" devices that neither need public access nor are capable of running an automat

I really wish Let's Encrypt had a solution for those of us with lots of "internal" devices that neither need public access nor are capable of running an automated renewal job (routers, switches, IoT gizmos, etc). All of these things come with half-baked self-signed certs, and web browsers and mobile devices these days are making it increasingly difficult to actually stomach such certs for anything at all. I wish the "SSL All The Things!" crusaders would consider this use case once in a while, and actually offer a usable solution. (Some of us could go through the trouble of setting up a private CA to deal with this issue, but getting everything to trust your own personal CA root seems like almost as much of a hassle at times.)

ricktdotorg 6y ago
look into Cloudflare’s cfssl. it’s really great if you just want a simple internal CA.
arkadiyt 6y ago
I use mkcert [1] with great success - it's easy to setup locally trusted certs which I load onto my router and other devices. [1]: https://github.com/FiloSottile/mkcert https://github.com/FiloSottile/mkcert
- wtallis 6y ago
  That seems like a nifty tool, but it doesn't seem to do anything to address the difficulty of getting your private CA trusted on more than one of your devices. I'd like to be able to access my router and switch configuration interfaces from more than one client PC, and perhaps even from a smartphone.
  - nemosaltat 6y ago
    Same here. I run OpenWRT on my home routers, and your comment prompted me to do a quick web search. uacme looks promising for luci. I’d love to hear HN experiences with it.
  - cpach 6y ago
    You can copy the (public part) of the generated root certificate over to your other computers and smartphones.
    - otabdeveloper4 6y ago
      Yes, that is the hard part.
      - exikyut 6y ago
        FWIW, tapping on .crt files in Chrome offers to open them with the certificate manager on Android, where it's then straightforward to install them. Unfortunately this also produces some very irritating (non-removable) notifications... >.< ...And I can't deny I was fetching the certificate from my PC across the room over HTTP, not HTTPS. :/
  - octorian 6y ago
    Getting your private CA trusted on a smartphone seems to keep getting more and more difficult and/or annoying with every iteration of the various mobile OSes. If it was possible to make a private CA cert that was only valid for hosts under a single domain, and then for mobile devices to STFU about you wanting to install it (as it wouldn't be as "risky" from their POV), life would be a little easier.
    - namibj 6y ago
      It's actually possible to do so, but many non-browser/mainstream cert validation libraries have a habit of just ignoring that restriction.
sleevi 6y ago
One of the challenges in this space is similar to captive portal detection. It’s not enough to have a solution, you need the vendors of those devices to adopt the solution. Unfortunately, just like captive portals want to avoid detection by your OS [*], the vendors of these devices want to do the least possible work, for the lowest possible cost, never updating or supporting the device once sold. Any new solution is more work for them, especially early on when things are still in development/in flux and may require changes over time, since the problems in this space are actually rather difficult. It’s not that this problem isn’t important, but rather that there’s no clear good path here. However, even still, the alternative (of HTTP) is much worse for the Web, so TLS is still promoted aggressively. [1] The reason captive portals want to avoid OS detection is because the OS-spawned UI generally doesn’t have access to your browser cookies. The ecosystem has mostly ossified around a few captive portal vendors, who want your cookies either for advertising/demographics/traffic, or want to offer “Sign in with Social Network” (to both track and... advertising). So they want you to use your main browser, not the OS UI, and thus want to actively avoid OS detection. It’s sad.
- tialaramex 6y ago
  The very long term good news on captive portals is that users hate them, so they're doomed on that basis.
  - thayne 6y ago
    Doesn't matter. Users aren't the ones the ones deciding to use them or not.
- octorian 6y ago
  > One of the challenges in this space is similar to captive portal detection I really wish someone had thought of that problem when defining the 802.11 specs. Though I suspect the idea hadn't even occurred to them at the time. This was much worse back in the mid-00's when WiFi was a newer use case. Seemed like flipping open my laptop in a public place back then would suddenly cause all the network-connected widgets on my desktop to go absolutely berserk as a result of portal nonsense.
at_a_remove 6y ago
I could not agree more. Information Technology, for reasons unknown to me, seems to ferment a kind of extremism with a large helping of "if you are NOT doing it this way you are what is killing the world," and the "SSL All the Things!" people are a great example of it. I've heard people suggest that you should just dump your web-serving software in favor of a different webserver that works better with Let's Encrypt, nevermind that a lot of people in the real world are running niche proprietary software that has very specific webservers they are tightly-integrated with. And -- it has been argued to me -- the solution was to not just get in that service sector in the first place.
- DenseComet 6y ago
  Most of the time, you can stick Nginx or HAProxy in front of it all, and terminate TLS there, with the last part on your interal network being HTTP.
  - alex_young 6y ago
    Sounds great until you consider threats inside your network
    - foepys 6y ago
      I think many here don't get that not working in a tech startup is very different. The IT budget is shrinking every year instead of increasing with company profits and sometimes a 3 person team needs to handle 300 users plus those machines that nowadays are tightly interconnected. These 300 users are essentially a threat to the company because they often click on anything they get per mail. So you need to encrypt everything as much as possible to decrease the attack surface inside the network.
      - zepearl 6y ago
        > These 300 users are essentially a threat to the company because they often click on anything they get per mail. I agree, but I have to mention as well that in general all teams (not only IT) got reductions of budgets and amount of employees and in the end everybody is very stressed most of the time, which in turn makes everybody pay less attention to what they're doing. In my case (IT) last year I received an email with the title "ACTION REQUIRED" and I just blindly opened it and clicked on the link of the document which I was supposed to fill out, and (luckily) it was just a well-crafted internal test related to phishing (which then of course I failed => I then had to take an online course) - in my opinion the problem is that nowadays we're permanently under pressure to deliver/react/do stuff with very short lead times and it's therefore hard after a while not to start to unwillingly forget best practices related to security. This year the company I'm working for has started as well to use quite often external companies (websites) to provide some services => doing that makes identifying phishing emails even more difficult for me.
baybal2 6y ago
Reminds me how either Xiaomi, or some other fairly big smart things co. bricked few millions of their smart lightbulbs when their root cert expired, and they had no valid cert left to sign a new firmware.
galaxyLogic 6y ago
> Some of us could go through the trouble of setting up a private CA to deal with this issue I've read about doing that for development purposes. But it sounds complicated. Therefore I wonder couldn't there be an open source solution that produces an npm module which when I run it creates a private CA for me? I mean if it is complicated to create a private CA, why not automate that task?
- martijnvds 6y ago
  "mkcert" is a tool to automate this: https://github.com/FiloSottile/mkcert https://github.com/FiloSottile/mkcert
- omBratteng 6y ago
  I've deployed step-ca internally and have my root certificate added through cloud-init for new servers. Also supports ACME https://smallstep.com/docs/getting-started/ https://smallstep.com/docs/getting-started/
silon42 6y ago
> (Some of us could go through the trouble of setting up a private CA to deal with this issue, but getting everything to trust your own personal CA root seems like almost as much of a hassle at times.) This is the actual problem... setting up your own CA for internal networks could be automated relatively easily.
- bsdubernerd 6y ago
  The problem is not the self-signed cert or CA. The problem is managing trust on the devices themselves. Imagine you want to trust only _your_ self-signed cert or CA root for a specific service. Good luck making that work. This issue should be so common that there should baked-in functionality in every piece of software to allow for this. It's often totally missing or implemented incorrectly (self-signed certs or custom CAs are often trusted in _addition_ to the system CA roots!).
hvidgaard 6y ago
Let's Encrypt revolves around short lived certificates and automated solutions for updating. I think your best bet is having an internal "service" (could be in a docker container), that do the updating. If you can SSH into the device, you can write a script that does it.
deleted 6y ago
[deleted]
mrzool 6y ago
I point a bunch of subdomains to my local IPs using https://nip.io https://nip.io and authenticate them all using a wildcard Let’s Encrypt certificate. Works great.
- g8oz 6y ago
  A very detailed tutorial with screenshots for those of us with no clue would be much appreciated.
vmladenov 6y ago
https://github.com/acmesh-official/acme.sh https://github.com/acmesh-official/acme.sh this works for me for most of my devices at home. pfSense and Proxmox both have plugins for it. Each device requests its own certificate for a host on a private subdomain of a domain I own e.g. hostname.private.domain.com
- mrweasel 6y ago
  That still only works if your device is able to reach the internet. We have a bunch of servers which aren't directly on the internet, because they don't need to.
  - carlhjerpe 6y ago
    You can run acme.sh on another device using dns-01 authentication. Then you can move the certificates to your servers in some way. This would mean one machine having access to the internet and the private servers though.
    - dastx 6y ago
      But also, LE certs expire after 90 days, meaning this manual process needs to be repeated every <90 days. Though, I'd be curious to know what the threat model for such a service is. Why is it not allowed to connect to the Internet? Can a different internal machine connect to it? And if so, is that machine allowed to connect to the Internet. What happens if someone connected to that machine and opens a tunnel through their machine to the Internet (if at all possible). Understanding the threat model will give you a solution on how best to use dns-01 challenge to generate certs and keep them updated.
      - castillar76 6y ago
        Alternately, you could use a fast CA solution like Netflix' lemur or step-ca or even just openssl, make your own CA, and distribute it to everything. If your threat model is such that that stuff should never connect to the outside world, why risk it by repeatedly moving things in from outside? Just generate a closed CA for your closed environment, trust it once, and move on.
filleokus 6y ago
Even when you have a private CA, you still have the issue of getting everything in your environment to trust it. Even in an enterprise setting, from my experience. Most workstations and servers are managed and have the root cert in the trust store, but then you have a long tail of appliances and containerised applications, or OSS software that doesn't use the OS trust store or BYO devices. With publicly trusted certs you instead trade the trusting-CA-issue for the issuing-issue. I'm starting to think that it's easier to use publicly trusted certs with FQDN for internal use. Some kind of ACME-proxy, and portal, which is publicly accessible and can respond to ACME inquires. Maybe in combination with some non-LE CA that can issue long lived certs for certain uses where updates can't be automated (e.g network appliances). But then there's the Certificate Transparency Issue, leaking internal names etc. The dream would be a LE issued sub-CA limited to *.internal.example.com, so you can have an "internal" CA that is publicly trusted.
- szszrk 6y ago
  That would mean I can issue a valid, trusted worldwide, certificate what will work on your internal site. Using public domains for internal sites are the way to go in my opinion. As long as you pay the bills you have guarantee that you control your domain. You can issue any kind of certificates and so on. If you have a nice naming scheme you won't leak that much info in then Internet. A lot of companies I know does that, by the recommendation of security teams (due to issues with using .local or .example.com domains).
  - michaelt 6y ago
    > If you have a nice naming scheme you won't leak that much info in then Internet. IME even if you leak the name of every internal service, you're still more secure than training all users to ignore certificate errors, which is what internal CAs do 85% of the time.
  - filleokus 6y ago
    Maybe I was unclear, but I was wishing for a situation where the Name Constraint extension was wildly enough used that LE would issue scoped sub-CA certs for *.whatever.yourdomain.com, which I hold the private keys to. But yes, I agree with your larger point. However, "nice" naming scheme might conflict with "secure" naming scheme. E.g, perhaps admin-{not-yet-released-product-line}.acme.com is the most resonable domain name, but it would be unsuitable if it was publicly announced via CT logs. Any way, having the domains in the CT logs, and naming them considering that is probably a tradeoff worth making.
cm2187 6y ago
Well Let's Encrypt has a solution, it is DNS authentication, which doesn't require Let's Encrypt to establish a direct connection to the device. But you still need a way to automatically renew the certificate (your DNS must have a supported API) and deploy it automatically. But that's not a Let's Encrypt problem to solve.
- jackweirdy 6y ago
  I use this solution myself. I have a domain name I own and only have A records configured for use on internal DNS. Externally it is managed by cloudflare, and certbot uses the cloudflare plugin to renew certificates before deploying certs to my hardware over telnet
  - exikyut 6y ago
    How I read this comment, presented as an audio-playback-style progress bar: [--------------------------------------------??] ^ Interesting! ^ Huh, that's a good i-- ^ Wait, what?
    - iso947 6y ago
      Deploying via telnet is reality for a lot of kit
- dingaling 6y ago
  > But that's not a Let's Encrypt problem to solve. But it's a problem they permitted to arise by enforcing 90-day expiry An internal web config interface could happily use a 10-year cert
  - cm2187 6y ago
    Browsers are also enforcing this, so again not really a let’s encrypt problem. But create your own root certs. That’s a much better route if you want to never update your certs on the device.
alanfranz 6y ago
> lots of "internal" devices that neither need public access nor are capable of running an automated renewal job (routers, switches, IoT gizmos, etc). How is that a CA's reponsibility? I wish vendors would setup an appropriate API for dealing with that on their devices. Quite often you need to enter strange web interfaces in order to setup a certificate. Let's Encrypt needs to verify you're in control of the hostname. This can happen with either http or dns verification. DNS verification with certbot requires you to setup almost nothing (i.e. you don't need the machine running certbot to be exposed to the public internet) if you're using a supported DNS provider (there're many) for the zone. So, the missing part is NOT in let's encrypt infrastructure. It's in the device API. You can launch the certificate renewal script just about anywhere (e.g. a small server or vm within your network that can access the internet); but then, how would you automatically deliver the certificate to the device? That's the missing link.
- deleted 6y ago
  [deleted]
- johannes1234321 6y ago
  > How is that a CA's reponsibility? The request wasn't (only) asking for a solution by the CAs but by the '"SSL All The Things!" crusaders' which includes creators of browsers and mobile devices enforcing stricter rules and a request to take these things into more consideration. In my words: We need solutions for non-technical individuals to run a somewhat secure environment locally without being confronted with security warnings everywhere. A CA can play a role in that, but probably not fully solve it.
- imtringued 6y ago
  Are you joking? The primary reason why one would want one certificate per device is because you are directly connecting to the device instead of letting the device connect to the vendor API which is already behind HTTPS. Your idea will just lead to a lot of bricked IoT devices once the vendor takes his certificate renewal API down.
  - ptman 6y ago
friendzis 6y ago
The solution is somewhat simple - have Let's Encrypt (or any root CA) issue intermediary CA limited to your domain. You "only" need devices to trust the root CA and could issue certs at will. The problem is that this use case was/is a threat to CA business model and is not really supported in the current cert infrastructure.
- lxgr 6y ago
  Isn‘t that mostly because the necessary X.509 extension isn‘t widely supported? Of course, this seems like a chicken and egg problem, but my point is that adoption would take a couple of years in practice.
  - friendzis 6y ago
    Yep, as long as relevant specs are not implemented by the majority of things touching trust (including appliances with very long update cycles), this is, sadly, "no bueno".
ikornaselur 6y ago
I use DNS auth for internal certificates. I have multiple domains that only have entries on 10.0.0.0/8, mostly as *.identifier.domain.tld. With DNS auth they all have valid certificates from LetsEncrypt.
- gfxgirl 6y ago
  Do you actually have that or are you suggesting it a solution? Unless something changed you can't get certs for subdomains unless they they are the public suffix list and getting on that list is non-trivial.
  - ikornaselur 6y ago
    Maybe I'm misunderstanding something, but I've been doing what I suggested for years. I just verified, the domain that I use doesn't have a single A entry that isn't on 10.0.0.0/8. I actually only have 6 A entries, that are the in the form of a.domain.tld *.a.domain.tld b.domain.tld *.b.domain.tld c.domain.tld *.c.domain.tld Then I use Caddy server as a reverse proxy on the internal network, which is configured to do DNS challenges to get certificates. Here's the plugin for AWS Route53 for example: https://github.com/caddy-dns/route53 https://github.com/caddy-dns/route53 - The challenge just verifies that I have control over the domain through DNS and provides a certificate to me no problem. It's been working perfectly for few years for us on our internal networks. Was the OP asking for something different? I'm not entirely sure what the 'public suffix list' is for subdomains, but I definitely have a valid certificate right now for *.a.domain.tld, served internally and provided by LetsEncrypt.
    - gfxgirl 6y ago
      https://letsencrypt.org/docs/rate-limits/ https://letsencrypt.org/docs/rate-limits/ The public suffix list is (or was) used to tell if foo.domain.com is considered different user than bar.domain.com If domain.com is on the list then foo.domain.com is different than bar.domain.com. If not then it's considered a single user and the rate limits apply.
tracker1 6y ago
You can use dns for certs instead of web/http verification.
qwertox 6y ago
Maybe some kind of wildcard approach would be helpful, where you generate your Let's Encrypt certificate with a wildcard, and are able to distribute it automatically to every device via a standardized API. That way only one device (possibly a Raspberry Pi) would actually be confronted with the task of generating the certificate and validating the verification requests from LE's servers. This is what I'm doing at home: on a public VPS I have a custom made DNS/HTTP-server, which is pointed to by the NS-Record for the subdomain int.example.com by my DNS-Provider for my domain example.com. So everything but int.example.com is under control of the DNS-Provider, int.example.com is under the control of my custom DNS/HTTP-Server. The local Raspberry Pi invokes certbot, gets the authentication tokens from LE's servers, POSTs them to the DNS/HTTP-server via HTTP, LE's servers query the DNS-server for the TXT-Record and certbot then continues with its work of storing the new certificates. Finally a script then distributes these certificates to the local HTTP-servers. The thing about these certificates is that they contain a wildcard certificate for * .int.example.com, and this way I get a wildcard certificate covering server-1.int.example.com, server-2.int.example.com, ... If the IoT-devices would have a standardized API to deploy the certificates to them, then I could access the ip camera via https://ipcam-1.int.example.com https://ipcam-1.int.example.com. The public DNS/HTTP-server does not respond to requests of any of the * .int.example.com subdomains. The local Pi-Hole has all the DNS-records for those subdomains.

ricktdotorg 6y ago

look into Cloudflare’s cfssl. it’s really great if you just want a simple internal CA.

arkadiyt 6y ago

I use mkcert [1] with great success - it's easy to setup locally trusted certs which I load onto my router and other devices. [1]: https://github.com/FiloSottile/mkcert https://github.com/FiloSottile/mkcert

wtallis 6y ago
That seems like a nifty tool, but it doesn't seem to do anything to address the difficulty of getting your private CA trusted on more than one of your devices. I'd like to be able to access my router and switch configuration interfaces from more than one client PC, and perhaps even from a smartphone.
- nemosaltat 6y ago
  Same here. I run OpenWRT on my home routers, and your comment prompted me to do a quick web search. uacme looks promising for luci. I’d love to hear HN experiences with it.
- cpach 6y ago
  You can copy the (public part) of the generated root certificate over to your other computers and smartphones.
  - otabdeveloper4 6y ago
    Yes, that is the hard part.
    - exikyut 6y ago
      FWIW, tapping on .crt files in Chrome offers to open them with the certificate manager on Android, where it's then straightforward to install them. Unfortunately this also produces some very irritating (non-removable) notifications... >.< ...And I can't deny I was fetching the certificate from my PC across the room over HTTP, not HTTPS. :/
- octorian 6y ago
  Getting your private CA trusted on a smartphone seems to keep getting more and more difficult and/or annoying with every iteration of the various mobile OSes. If it was possible to make a private CA cert that was only valid for hosts under a single domain, and then for mobile devices to STFU about you wanting to install it (as it wouldn't be as "risky" from their POV), life would be a little easier.
  - namibj 6y ago
    It's actually possible to do so, but many non-browser/mainstream cert validation libraries have a habit of just ignoring that restriction.

sleevi 6y ago

One of the challenges in this space is similar to captive portal detection. It’s not enough to have a solution, you need the vendors of those devices to adopt the solution. Unfortunately, just like captive portals want to avoid detection by your OS [*], the vendors of these devices want to do the least possible work, for the lowest possible cost, never updating or supporting the device once sold. Any new solution is more work for them, especially early on when things are still in development/in flux and may require changes over time, since the problems in this space are actually rather difficult. It’s not that this problem isn’t important, but rather that there’s no clear good path here. However, even still, the alternative (of HTTP) is much worse for the Web, so TLS is still promoted aggressively. [1] The reason captive portals want to avoid OS detection is because the OS-spawned UI generally doesn’t have access to your browser cookies. The ecosystem has mostly ossified around a few captive portal vendors, who want your cookies either for advertising/demographics/traffic, or want to offer “Sign in with Social Network” (to both track and... advertising). So they want you to use your main browser, not the OS UI, and thus want to actively avoid OS detection. It’s sad.

tialaramex 6y ago
The very long term good news on captive portals is that users hate them, so they're doomed on that basis.
- thayne 6y ago
  Doesn't matter. Users aren't the ones the ones deciding to use them or not.
octorian 6y ago
> One of the challenges in this space is similar to captive portal detection I really wish someone had thought of that problem when defining the 802.11 specs. Though I suspect the idea hadn't even occurred to them at the time. This was much worse back in the mid-00's when WiFi was a newer use case. Seemed like flipping open my laptop in a public place back then would suddenly cause all the network-connected widgets on my desktop to go absolutely berserk as a result of portal nonsense.

at_a_remove 6y ago

I could not agree more. Information Technology, for reasons unknown to me, seems to ferment a kind of extremism with a large helping of "if you are NOT doing it this way you are what is killing the world," and the "SSL All the Things!" people are a great example of it. I've heard people suggest that you should just dump your web-serving software in favor of a different webserver that works better with Let's Encrypt, nevermind that a lot of people in the real world are running niche proprietary software that has very specific webservers they are tightly-integrated with. And -- it has been argued to me -- the solution was to not just get in that service sector in the first place.

DenseComet 6y ago
Most of the time, you can stick Nginx or HAProxy in front of it all, and terminate TLS there, with the last part on your interal network being HTTP.
- alex_young 6y ago
  Sounds great until you consider threats inside your network
  - foepys 6y ago
    I think many here don't get that not working in a tech startup is very different. The IT budget is shrinking every year instead of increasing with company profits and sometimes a 3 person team needs to handle 300 users plus those machines that nowadays are tightly interconnected. These 300 users are essentially a threat to the company because they often click on anything they get per mail. So you need to encrypt everything as much as possible to decrease the attack surface inside the network.
    - zepearl 6y ago
      > These 300 users are essentially a threat to the company because they often click on anything they get per mail. I agree, but I have to mention as well that in general all teams (not only IT) got reductions of budgets and amount of employees and in the end everybody is very stressed most of the time, which in turn makes everybody pay less attention to what they're doing. In my case (IT) last year I received an email with the title "ACTION REQUIRED" and I just blindly opened it and clicked on the link of the document which I was supposed to fill out, and (luckily) it was just a well-crafted internal test related to phishing (which then of course I failed => I then had to take an online course) - in my opinion the problem is that nowadays we're permanently under pressure to deliver/react/do stuff with very short lead times and it's therefore hard after a while not to start to unwillingly forget best practices related to security. This year the company I'm working for has started as well to use quite often external companies (websites) to provide some services => doing that makes identifying phishing emails even more difficult for me.

baybal2 6y ago

Reminds me how either Xiaomi, or some other fairly big smart things co. bricked few millions of their smart lightbulbs when their root cert expired, and they had no valid cert left to sign a new firmware.

galaxyLogic 6y ago

> Some of us could go through the trouble of setting up a private CA to deal with this issue I've read about doing that for development purposes. But it sounds complicated. Therefore I wonder couldn't there be an open source solution that produces an npm module which when I run it creates a private CA for me? I mean if it is complicated to create a private CA, why not automate that task?

martijnvds 6y ago
"mkcert" is a tool to automate this: https://github.com/FiloSottile/mkcert https://github.com/FiloSottile/mkcert
omBratteng 6y ago
I've deployed step-ca internally and have my root certificate added through cloud-init for new servers. Also supports ACME https://smallstep.com/docs/getting-started/ https://smallstep.com/docs/getting-started/

silon42 6y ago

> (Some of us could go through the trouble of setting up a private CA to deal with this issue, but getting everything to trust your own personal CA root seems like almost as much of a hassle at times.) This is the actual problem... setting up your own CA for internal networks could be automated relatively easily.

bsdubernerd 6y ago
The problem is not the self-signed cert or CA. The problem is managing trust on the devices themselves. Imagine you want to trust only _your_ self-signed cert or CA root for a specific service. Good luck making that work. This issue should be so common that there should baked-in functionality in every piece of software to allow for this. It's often totally missing or implemented incorrectly (self-signed certs or custom CAs are often trusted in _addition_ to the system CA roots!).

hvidgaard 6y ago

Let's Encrypt revolves around short lived certificates and automated solutions for updating. I think your best bet is having an internal "service" (could be in a docker container), that do the updating. If you can SSH into the device, you can write a script that does it.

deleted 6y ago

[deleted]

mrzool 6y ago

I point a bunch of subdomains to my local IPs using https://nip.io https://nip.io and authenticate them all using a wildcard Let’s Encrypt certificate. Works great.

g8oz 6y ago
A very detailed tutorial with screenshots for those of us with no clue would be much appreciated.

vmladenov 6y ago

https://github.com/acmesh-official/acme.sh https://github.com/acmesh-official/acme.sh this works for me for most of my devices at home. pfSense and Proxmox both have plugins for it. Each device requests its own certificate for a host on a private subdomain of a domain I own e.g. hostname.private.domain.com

mrweasel 6y ago
That still only works if your device is able to reach the internet. We have a bunch of servers which aren't directly on the internet, because they don't need to.
- carlhjerpe 6y ago
  You can run acme.sh on another device using dns-01 authentication. Then you can move the certificates to your servers in some way. This would mean one machine having access to the internet and the private servers though.
  - dastx 6y ago
    But also, LE certs expire after 90 days, meaning this manual process needs to be repeated every <90 days. Though, I'd be curious to know what the threat model for such a service is. Why is it not allowed to connect to the Internet? Can a different internal machine connect to it? And if so, is that machine allowed to connect to the Internet. What happens if someone connected to that machine and opens a tunnel through their machine to the Internet (if at all possible). Understanding the threat model will give you a solution on how best to use dns-01 challenge to generate certs and keep them updated.
    - castillar76 6y ago
      Alternately, you could use a fast CA solution like Netflix' lemur or step-ca or even just openssl, make your own CA, and distribute it to everything. If your threat model is such that that stuff should never connect to the outside world, why risk it by repeatedly moving things in from outside? Just generate a closed CA for your closed environment, trust it once, and move on.

filleokus 6y ago

Even when you have a private CA, you still have the issue of getting everything in your environment to trust it. Even in an enterprise setting, from my experience. Most workstations and servers are managed and have the root cert in the trust store, but then you have a long tail of appliances and containerised applications, or OSS software that doesn't use the OS trust store or BYO devices. With publicly trusted certs you instead trade the trusting-CA-issue for the issuing-issue. I'm starting to think that it's easier to use publicly trusted certs with FQDN for internal use. Some kind of ACME-proxy, and portal, which is publicly accessible and can respond to ACME inquires. Maybe in combination with some non-LE CA that can issue long lived certs for certain uses where updates can't be automated (e.g network appliances). But then there's the Certificate Transparency Issue, leaking internal names etc. The dream would be a LE issued sub-CA limited to *.internal.example.com, so you can have an "internal" CA that is publicly trusted.

szszrk 6y ago
That would mean I can issue a valid, trusted worldwide, certificate what will work on your internal site. Using public domains for internal sites are the way to go in my opinion. As long as you pay the bills you have guarantee that you control your domain. You can issue any kind of certificates and so on. If you have a nice naming scheme you won't leak that much info in then Internet. A lot of companies I know does that, by the recommendation of security teams (due to issues with using .local or .example.com domains).
- michaelt 6y ago
  > If you have a nice naming scheme you won't leak that much info in then Internet. IME even if you leak the name of every internal service, you're still more secure than training all users to ignore certificate errors, which is what internal CAs do 85% of the time.
- filleokus 6y ago
  Maybe I was unclear, but I was wishing for a situation where the Name Constraint extension was wildly enough used that LE would issue scoped sub-CA certs for *.whatever.yourdomain.com, which I hold the private keys to. But yes, I agree with your larger point. However, "nice" naming scheme might conflict with "secure" naming scheme. E.g, perhaps admin-{not-yet-released-product-line}.acme.com is the most resonable domain name, but it would be unsuitable if it was publicly announced via CT logs. Any way, having the domains in the CT logs, and naming them considering that is probably a tradeoff worth making.

cm2187 6y ago

Well Let's Encrypt has a solution, it is DNS authentication, which doesn't require Let's Encrypt to establish a direct connection to the device. But you still need a way to automatically renew the certificate (your DNS must have a supported API) and deploy it automatically. But that's not a Let's Encrypt problem to solve.

jackweirdy 6y ago
I use this solution myself. I have a domain name I own and only have A records configured for use on internal DNS. Externally it is managed by cloudflare, and certbot uses the cloudflare plugin to renew certificates before deploying certs to my hardware over telnet
- exikyut 6y ago
  How I read this comment, presented as an audio-playback-style progress bar: [--------------------------------------------??] ^ Interesting! ^ Huh, that's a good i-- ^ Wait, what?
  - iso947 6y ago
    Deploying via telnet is reality for a lot of kit
dingaling 6y ago
> But that's not a Let's Encrypt problem to solve. But it's a problem they permitted to arise by enforcing 90-day expiry An internal web config interface could happily use a 10-year cert
- cm2187 6y ago
  Browsers are also enforcing this, so again not really a let’s encrypt problem. But create your own root certs. That’s a much better route if you want to never update your certs on the device.

alanfranz 6y ago

> lots of "internal" devices that neither need public access nor are capable of running an automated renewal job (routers, switches, IoT gizmos, etc). How is that a CA's reponsibility? I wish vendors would setup an appropriate API for dealing with that on their devices. Quite often you need to enter strange web interfaces in order to setup a certificate. Let's Encrypt needs to verify you're in control of the hostname. This can happen with either http or dns verification. DNS verification with certbot requires you to setup almost nothing (i.e. you don't need the machine running certbot to be exposed to the public internet) if you're using a supported DNS provider (there're many) for the zone. So, the missing part is NOT in let's encrypt infrastructure. It's in the device API. You can launch the certificate renewal script just about anywhere (e.g. a small server or vm within your network that can access the internet); but then, how would you automatically deliver the certificate to the device? That's the missing link.

deleted 6y ago
[deleted]
johannes1234321 6y ago
> How is that a CA's reponsibility? The request wasn't (only) asking for a solution by the CAs but by the '"SSL All The Things!" crusaders' which includes creators of browsers and mobile devices enforcing stricter rules and a request to take these things into more consideration. In my words: We need solutions for non-technical individuals to run a somewhat secure environment locally without being confronted with security warnings everywhere. A CA can play a role in that, but probably not fully solve it.
imtringued 6y ago
Are you joking? The primary reason why one would want one certificate per device is because you are directly connecting to the device instead of letting the device connect to the vendor API which is already behind HTTPS. Your idea will just lead to a lot of bricked IoT devices once the vendor takes his certificate renewal API down.
- ptman 6y ago

friendzis 6y ago

The solution is somewhat simple - have Let's Encrypt (or any root CA) issue intermediary CA limited to your domain. You "only" need devices to trust the root CA and could issue certs at will. The problem is that this use case was/is a threat to CA business model and is not really supported in the current cert infrastructure.

lxgr 6y ago
Isn‘t that mostly because the necessary X.509 extension isn‘t widely supported? Of course, this seems like a chicken and egg problem, but my point is that adoption would take a couple of years in practice.
- friendzis 6y ago
  Yep, as long as relevant specs are not implemented by the majority of things touching trust (including appliances with very long update cycles), this is, sadly, "no bueno".

ikornaselur 6y ago

I use DNS auth for internal certificates. I have multiple domains that only have entries on 10.0.0.0/8, mostly as *.identifier.domain.tld. With DNS auth they all have valid certificates from LetsEncrypt.

gfxgirl 6y ago
Do you actually have that or are you suggesting it a solution? Unless something changed you can't get certs for subdomains unless they they are the public suffix list and getting on that list is non-trivial.
- ikornaselur 6y ago
  Maybe I'm misunderstanding something, but I've been doing what I suggested for years. I just verified, the domain that I use doesn't have a single A entry that isn't on 10.0.0.0/8. I actually only have 6 A entries, that are the in the form of a.domain.tld *.a.domain.tld b.domain.tld *.b.domain.tld c.domain.tld *.c.domain.tld Then I use Caddy server as a reverse proxy on the internal network, which is configured to do DNS challenges to get certificates. Here's the plugin for AWS Route53 for example: https://github.com/caddy-dns/route53 https://github.com/caddy-dns/route53 - The challenge just verifies that I have control over the domain through DNS and provides a certificate to me no problem. It's been working perfectly for few years for us on our internal networks. Was the OP asking for something different? I'm not entirely sure what the 'public suffix list' is for subdomains, but I definitely have a valid certificate right now for *.a.domain.tld, served internally and provided by LetsEncrypt.
  - gfxgirl 6y ago
    https://letsencrypt.org/docs/rate-limits/ https://letsencrypt.org/docs/rate-limits/ The public suffix list is (or was) used to tell if foo.domain.com is considered different user than bar.domain.com If domain.com is on the list then foo.domain.com is different than bar.domain.com. If not then it's considered a single user and the rate limits apply.

tracker1 6y ago

You can use dns for certs instead of web/http verification.

qwertox 6y ago

Maybe some kind of wildcard approach would be helpful, where you generate your Let's Encrypt certificate with a wildcard, and are able to distribute it automatically to every device via a standardized API. That way only one device (possibly a Raspberry Pi) would actually be confronted with the task of generating the certificate and validating the verification requests from LE's servers. This is what I'm doing at home: on a public VPS I have a custom made DNS/HTTP-server, which is pointed to by the NS-Record for the subdomain int.example.com by my DNS-Provider for my domain example.com. So everything but int.example.com is under control of the DNS-Provider, int.example.com is under the control of my custom DNS/HTTP-Server. The local Raspberry Pi invokes certbot, gets the authentication tokens from LE's servers, POSTs them to the DNS/HTTP-server via HTTP, LE's servers query the DNS-server for the TXT-Record and certbot then continues with its work of storing the new certificates. Finally a script then distributes these certificates to the local HTTP-servers. The thing about these certificates is that they contain a wildcard certificate for * .int.example.com, and this way I get a wildcard certificate covering server-1.int.example.com, server-2.int.example.com, ... If the IoT-devices would have a standardized API to deploy the certificates to them, then I could access the ip camera via https://ipcam-1.int.example.com https://ipcam-1.int.example.com. The public DNS/HTTP-server does not respond to requests of any of the * .int.example.com subdomains. The local Pi-Hole has all the DNS-records for those subdomains.