8 ms·

Heh, working with a number of large companies I've seen most of them moving to internally signed certs on everything because of ever shortening expiration time

by pixl97 2y ago

Heh, working with a number of large companies I've seen most of them moving to internally signed certs on everything because of ever shortening expiration times. They'll have public certs on edge devices/load balancers but internal services with have internal CA signed certs with long expire times because of the number of crappy apps that make using certs a pain in the ass.

xienze 2y ago
> but internal services with have internal CA signed certs with long expire times because of the number of crappy apps that make using certs a pain in the ass. Introduce them to the idea of having something like Caddy sit in front of apps solely for the purpose of doing TLS termination... Caddy et al can update the certs automatically.
- pixl97 2y ago
  Unless they are web/tech companies they aren't doing that. Banks, finance, large manufacturing are all terminating at F5's and AVI's. I'm pretty sure those update certs just fine, but it's not really what I do these days so I don't have a direct answer.
  - xienze 2y ago
    Sure. The point is, don't bother letting the apps themselves do TLS termination. Too much work that's better handled by something else.
    - hedora 2y ago
      Also, moving termination off the endpoint server makes it much easier for three letter agencies to intercept + log.
      - qmarchi 2y ago
        Most responsible orgs do TLS termination on the public side of a connection, but will still make a backend connection protected by TLS, just with a internal CA.
  - tikkabhuna 2y ago
    F5s don't support ACME, which has been a pain for us.
    - EvanAnderson 2y ago
      Exactly. According to posters here you should just throw them away and buy hardware from a vendor who does. >sigh< Don't expect firmware / software updates to enable ACME-type functionality for tons of gear. At best it'll be treated as an excuse by vendors to make Customers forklift and replace otherwise working gear. Corporate hardware lifecycles are longer than the proposed timeline for these changes. This feels like an ill thought-out initiative by bureaucrats working in companies who build their own infrastructure (in their white towers). Meanwhile, we plebs who work in less-than-Fortune 500 companies stuck with off-the-shelf solutions will be forced to suffer.
    - cpach 2y ago
      It might be possible to run an ACME client on another host in your environment. (IMHO, the DNS-01 challenge is very useful for this.) Then you can (probably) transfer the cert+key to BIG IP, and activate it, via the REST API. I haven’t used BIG IP in a long while, so take this with a grain of salt, but it seems to me that it might not be impossible to get something going – despite the fact that BIG IP itself doesn’t have native support for ACME. Two pointers that might be of interest: https://community.f5.com/discussions/technicalforum/upload-ltm-ssl-certificate-and-key-using-rest-api/324404 https://community.f5.com/discussions/technicalforum/upload-l... https://clouddocs.f5.com/api/icontrol-rest/APIRef_tm_sys_crypto_cert.html https://clouddocs.f5.com/api/icontrol-rest/APIRef_tm_sys_cry...
      - dijit 2y ago
        Sounds suspiciously similar to a rube goldberg machine. Those tend to be quite brittle in reality. What’s the old adage about engineering vs architecture again? Something like this I think: https://www.reddit.com/r/PeterExplainsTheJoke/comments/16141tk/all_comments_were_in_foreign_language/ https://www.reddit.com/r/PeterExplainsTheJoke/comments/16141...
- cryptonym 2y ago
  You now have to build and self-shot a complete CA/PKI. Or request a certificate over the public internet, for an internal service. Your hostname must be exposed to the web and will be publicly visible in transparency reports.
  - stackskipton 2y ago
    You could always ask for wildcard for internal subdomain and use that instead so you will leak your internal FQDN but not individual hosts.
    - pixl97 2y ago
      I'm pretty sure every bank will auto fail wildcard certs these days, at least the ones I've worked with. Key loss on one of those is like a takeover of an entire chunk of hostnames. Really opens you up.
  - mox1 2y ago
    Companies have software to manage this for you. We utilize https://www.cyberark.com/products/machine-identity-security/ https://www.cyberark.com/products/machine-identity-security/
  - JoshTriplett 2y ago
    > Or request a certificate over the public internet, for an internal service. Your hostname must be exposed to the web and will be publicly visible in transparency reports. That doesn't seem like the end of the world. It means you shouldn't have `secret-plans-for-world-takeover.example.com`, but it's already the case that secret projects should use opaque codenames. Most internal domain names would not actually leak any information of value.
rsstack 2y ago
> I've seen most of them moving to internally signed certs Isn't this a good default? No network access, no need for a public certificate, no need for a certificate that might be mistakenly trusted by a public (non-malicious) device, no need for a public log for the issued certificate.
- pavon 2y ago
  Yes, but it is a lot more work to run an internal CA and distribute that CA cert to all the corporate clients. In the past getting a public wildcard cert was the path of least resistance for internal sites - no network access needed, and you aren't leaking much info into the public log. That is changing now, and like you said it is probably a change for the better.
  - pkaye 2y ago
    What about something like step-ca? I got the free version working easily on my home network. https://smallstep.com/docs/step-ca/ https://smallstep.com/docs/step-ca/
    - simiones 2y ago
      Not everything that's easy to do on a home network is easy to do on a corporate network. The biggest problem with corporate CAs is how to emit new certificates for a new device in a secure way, a problem which simply doesn't exist on a home network where you have one or at most a handful of people needing new certs to be emitted.
  - bravetraveler 2y ago
    > A lot more work 'ipa-client-install' for those so motivated. Certificates are literally one among many things part of your domain services. If you're at the scale past what IPA/your domain can manage, well, c'est la vie.
    - Spivak 2y ago
      I think you're being generous if you think the average "cloud native" company is joining their servers to a domain at all. They've certainly fallen out of fashion in favor of the servers being dumb and user access being mediated by an outside system.
plorkyeran 2y ago
This is a desired outcome. The WebPKI ecosystem would really like it if everyone stopped depending on them for internal things because it's actually a pretty different set of requirements. Long-lived certs with an internal CA makes a lot of sense and is often more secure than using a public CA.
- ozim 2y ago
  Problem is browsers will most likely follow the enforcement of short certificates so internal sites will be affected as well. Non browser things usually don’t care even if cert is expired or trusted. So I expect people still to use WebPKI for internal sites.
  - ryao 2y ago
    Why would they? The old certificates will expire and the new ones will have short lifespans. Web browsers do not need to do anything. That said, it would be really nice if they supported DANE so that websites do not need CAs.
  - nickf 2y ago
    'Most likely' - with the exception of Apple enforcing 825-day maximum for private/internal CAs, this change isn't going to affect those internal certificates.
  - akerl_ 2y ago
    The browser policies are set by the same entities doing the CAB voting, and basically every prior change around WebPKI has only been enforced by browsers for CAs in the browser root trust stores. Which is exactly what's defined in this CAB vote as well. Why would browsers "most likely" enforce this change for internal CAs as well?
- rlpb 2y ago
  Browsers don't design for internal use though. They insist on HTTPS for various things that are intranet only, such as some browser APIs, PWAs, etc
shlant 2y ago
this is exactly what I do because mongo and TLS is enough of a headache. I am not dealing with rotating certificates regularly on top of that for endpoints not exposed to the internet.
- SoftTalker 2y ago
  Yep letsencrypt is great for public-facing web servers but for stuff that isn't a web server or doesn't allow outside queries none of that "easy" automation works.
  - procaryote 2y ago
    Acme dns challenge works for things that aren't webservers. For the other case perhaps renew the cert at a host allowed to do outside queries for the dns challenge and find some acceptable automated way to propagate an updated cert to the host that isn't allowed outside queries.
    - SoftTalker 2y ago
      I don't have an API or any permission to add TXT records to my DNS. That's a support ticket and has about a 24-hour turnaround best case.
      - procaryote 2y ago
        That's not great, sorry to hear
      - immibis 2y ago
        Is this just because your DNS is with some provider, or is it something that leads from your organizational structure? If it's just because your DNS is at a provider, you should be aware that it's possible to self-host DNS.
        SoftTalker 2y ago
jiggawatts 2y ago
I just got a flashback to trying to automate the certificate issuance process for some ESRI ArcGIS product that used an RPC configuration API over HTTPS to change the certificate. So yes, you had to first ignore the invalid self-signed certificate while using HTTPS with a client tool that really, really didn't want to ignore the validity issue, then upload a valid certificate, restart the service... which would terminate the HTTPS connection with an error breaking your script in a different not-fun way, and then reconnect... at some unspecified time later to continue the configuration. Fun times...
lokar 2y ago
I’ve always felt a major benefit of an internal CA is making it easy to have very sort TTLs
- SoftTalker 2y ago
  Or very long ones. I often generate 10 year certs because then I don't have to worry about renewing them for the lifetime of the hardware.
  - lokar 2y ago
    In a production environment with customer data?
    - SoftTalker 2y ago
      No for internal stuff.
- formerly_proven 2y ago
  I'm surprised there is no authorization-certificate-based challenge type for ACME yet. That would make ACME practical to use in microsegmented networks. The closest thing is maybe described (but not shown) in these posts: https://blog.daknob.net/workload-mtls-with-acme/ https://blog.daknob.net/workload-mtls-with-acme/ https://blog.daknob.net/acme-end-user-client-certificates/ https://blog.daknob.net/acme-end-user-client-certificates/
  - bigp3t3 2y ago
    I'd set that up the second it becomes available if it were a standard protocol. Just went through setting up internal certs on my switches -- it was a chore to say the least! With a Cert Template on our internal CA (windows), at least we can automate things well enough!
    - formerly_proven 2y ago
      Yeah it's almost weird it doesn't seem to exist, at least publicly. My megacorp created their own protocol for this purpose (though it might actually predate ACME, I'm not sure), and a bunch of in-house people and suppliers created the necessary middlewares to integrate it into stuff like cert-manager and such (basically everything that needs a TLS certificate and is deployed more than thrice). I imagine many larger companies have very similar things, with the only material difference being different organizational OIDs for the proprietary extension fields (I found it quite cute when I learned that the corp created a very neat subtree beneath its organization OID).
maccard 2y ago
I’ve unfortunately seen the opposite - internal apps are now back to being deployed over VPN and HTTP
Pxtl 2y ago
At this point I wish we could just get all our clients to say "self-signed is fine if you're connecting to a .LOCAL domain name". https is intrinsically useful over raw http, but the overhead of setting up centralized certs for non-public domains is just dumb. Give us a big global *.local cert we can all cheat with, so I don't have to blast my credentials in the clear when I log into my router's admin page.
JackSlateur 2y ago
Haa, yes ! We have that, too ! Accepted warning in browsers ! curl -k ! verify=False ! Glorious future to the hacking industry !
tomjen3 2y ago
I would love to do that for my homelab, but not all docker containers trust root certs from the system so getting it right would have been a bigger challenge than dns hacking to get a valid certificate for something that can’t be accessed from outside the network. I am not willing to give credentials to alter my dns to a program. A security issue there would be too much risk.

by pixl97 2y ago

xienze 2y ago

> but internal services with have internal CA signed certs with long expire times because of the number of crappy apps that make using certs a pain in the ass. Introduce them to the idea of having something like Caddy sit in front of apps solely for the purpose of doing TLS termination... Caddy et al can update the certs automatically.

pixl97 2y ago
Unless they are web/tech companies they aren't doing that. Banks, finance, large manufacturing are all terminating at F5's and AVI's. I'm pretty sure those update certs just fine, but it's not really what I do these days so I don't have a direct answer.
- xienze 2y ago
  Sure. The point is, don't bother letting the apps themselves do TLS termination. Too much work that's better handled by something else.
  - hedora 2y ago
    Also, moving termination off the endpoint server makes it much easier for three letter agencies to intercept + log.
    - qmarchi 2y ago
      Most responsible orgs do TLS termination on the public side of a connection, but will still make a backend connection protected by TLS, just with a internal CA.
- tikkabhuna 2y ago
  F5s don't support ACME, which has been a pain for us.
  - EvanAnderson 2y ago
    Exactly. According to posters here you should just throw them away and buy hardware from a vendor who does. >sigh< Don't expect firmware / software updates to enable ACME-type functionality for tons of gear. At best it'll be treated as an excuse by vendors to make Customers forklift and replace otherwise working gear. Corporate hardware lifecycles are longer than the proposed timeline for these changes. This feels like an ill thought-out initiative by bureaucrats working in companies who build their own infrastructure (in their white towers). Meanwhile, we plebs who work in less-than-Fortune 500 companies stuck with off-the-shelf solutions will be forced to suffer.
  - cpach 2y ago
    It might be possible to run an ACME client on another host in your environment. (IMHO, the DNS-01 challenge is very useful for this.) Then you can (probably) transfer the cert+key to BIG IP, and activate it, via the REST API. I haven’t used BIG IP in a long while, so take this with a grain of salt, but it seems to me that it might not be impossible to get something going – despite the fact that BIG IP itself doesn’t have native support for ACME. Two pointers that might be of interest: https://community.f5.com/discussions/technicalforum/upload-ltm-ssl-certificate-and-key-using-rest-api/324404 https://community.f5.com/discussions/technicalforum/upload-l... https://clouddocs.f5.com/api/icontrol-rest/APIRef_tm_sys_crypto_cert.html https://clouddocs.f5.com/api/icontrol-rest/APIRef_tm_sys_cry...
    - dijit 2y ago
      Sounds suspiciously similar to a rube goldberg machine. Those tend to be quite brittle in reality. What’s the old adage about engineering vs architecture again? Something like this I think: https://www.reddit.com/r/PeterExplainsTheJoke/comments/16141tk/all_comments_were_in_foreign_language/ https://www.reddit.com/r/PeterExplainsTheJoke/comments/16141...
cryptonym 2y ago
You now have to build and self-shot a complete CA/PKI. Or request a certificate over the public internet, for an internal service. Your hostname must be exposed to the web and will be publicly visible in transparency reports.
- stackskipton 2y ago
  You could always ask for wildcard for internal subdomain and use that instead so you will leak your internal FQDN but not individual hosts.
  - pixl97 2y ago
    I'm pretty sure every bank will auto fail wildcard certs these days, at least the ones I've worked with. Key loss on one of those is like a takeover of an entire chunk of hostnames. Really opens you up.
- mox1 2y ago
  Companies have software to manage this for you. We utilize https://www.cyberark.com/products/machine-identity-security/ https://www.cyberark.com/products/machine-identity-security/
- JoshTriplett 2y ago
  > Or request a certificate over the public internet, for an internal service. Your hostname must be exposed to the web and will be publicly visible in transparency reports. That doesn't seem like the end of the world. It means you shouldn't have `secret-plans-for-world-takeover.example.com`, but it's already the case that secret projects should use opaque codenames. Most internal domain names would not actually leak any information of value.

rsstack 2y ago

> I've seen most of them moving to internally signed certs Isn't this a good default? No network access, no need for a public certificate, no need for a certificate that might be mistakenly trusted by a public (non-malicious) device, no need for a public log for the issued certificate.

pavon 2y ago
Yes, but it is a lot more work to run an internal CA and distribute that CA cert to all the corporate clients. In the past getting a public wildcard cert was the path of least resistance for internal sites - no network access needed, and you aren't leaking much info into the public log. That is changing now, and like you said it is probably a change for the better.
- pkaye 2y ago
  What about something like step-ca? I got the free version working easily on my home network. https://smallstep.com/docs/step-ca/ https://smallstep.com/docs/step-ca/
  - simiones 2y ago
    Not everything that's easy to do on a home network is easy to do on a corporate network. The biggest problem with corporate CAs is how to emit new certificates for a new device in a secure way, a problem which simply doesn't exist on a home network where you have one or at most a handful of people needing new certs to be emitted.
- bravetraveler 2y ago
  > A lot more work 'ipa-client-install' for those so motivated. Certificates are literally one among many things part of your domain services. If you're at the scale past what IPA/your domain can manage, well, c'est la vie.
  - Spivak 2y ago
    I think you're being generous if you think the average "cloud native" company is joining their servers to a domain at all. They've certainly fallen out of fashion in favor of the servers being dumb and user access being mediated by an outside system.

plorkyeran 2y ago

This is a desired outcome. The WebPKI ecosystem would really like it if everyone stopped depending on them for internal things because it's actually a pretty different set of requirements. Long-lived certs with an internal CA makes a lot of sense and is often more secure than using a public CA.

ozim 2y ago
Problem is browsers will most likely follow the enforcement of short certificates so internal sites will be affected as well. Non browser things usually don’t care even if cert is expired or trusted. So I expect people still to use WebPKI for internal sites.
- ryao 2y ago
  Why would they? The old certificates will expire and the new ones will have short lifespans. Web browsers do not need to do anything. That said, it would be really nice if they supported DANE so that websites do not need CAs.
- nickf 2y ago
  'Most likely' - with the exception of Apple enforcing 825-day maximum for private/internal CAs, this change isn't going to affect those internal certificates.
- akerl_ 2y ago
  The browser policies are set by the same entities doing the CAB voting, and basically every prior change around WebPKI has only been enforced by browsers for CAs in the browser root trust stores. Which is exactly what's defined in this CAB vote as well. Why would browsers "most likely" enforce this change for internal CAs as well?
rlpb 2y ago
Browsers don't design for internal use though. They insist on HTTPS for various things that are intranet only, such as some browser APIs, PWAs, etc

shlant 2y ago

this is exactly what I do because mongo and TLS is enough of a headache. I am not dealing with rotating certificates regularly on top of that for endpoints not exposed to the internet.

SoftTalker 2y ago
Yep letsencrypt is great for public-facing web servers but for stuff that isn't a web server or doesn't allow outside queries none of that "easy" automation works.
- procaryote 2y ago
  Acme dns challenge works for things that aren't webservers. For the other case perhaps renew the cert at a host allowed to do outside queries for the dns challenge and find some acceptable automated way to propagate an updated cert to the host that isn't allowed outside queries.
  - SoftTalker 2y ago
    I don't have an API or any permission to add TXT records to my DNS. That's a support ticket and has about a 24-hour turnaround best case.
    - procaryote 2y ago
      That's not great, sorry to hear
    - immibis 2y ago
      Is this just because your DNS is with some provider, or is it something that leads from your organizational structure? If it's just because your DNS is at a provider, you should be aware that it's possible to self-host DNS.
      - SoftTalker 2y ago

jiggawatts 2y ago

I just got a flashback to trying to automate the certificate issuance process for some ESRI ArcGIS product that used an RPC configuration API over HTTPS to change the certificate. So yes, you had to first ignore the invalid self-signed certificate while using HTTPS with a client tool that really, really didn't want to ignore the validity issue, then upload a valid certificate, restart the service... which would terminate the HTTPS connection with an error breaking your script in a different not-fun way, and then reconnect... at some unspecified time later to continue the configuration. Fun times...

lokar 2y ago

I’ve always felt a major benefit of an internal CA is making it easy to have very sort TTLs

SoftTalker 2y ago
Or very long ones. I often generate 10 year certs because then I don't have to worry about renewing them for the lifetime of the hardware.
- lokar 2y ago
  In a production environment with customer data?
  - SoftTalker 2y ago
    No for internal stuff.
formerly_proven 2y ago
I'm surprised there is no authorization-certificate-based challenge type for ACME yet. That would make ACME practical to use in microsegmented networks. The closest thing is maybe described (but not shown) in these posts: https://blog.daknob.net/workload-mtls-with-acme/ https://blog.daknob.net/workload-mtls-with-acme/ https://blog.daknob.net/acme-end-user-client-certificates/ https://blog.daknob.net/acme-end-user-client-certificates/
- bigp3t3 2y ago
  I'd set that up the second it becomes available if it were a standard protocol. Just went through setting up internal certs on my switches -- it was a chore to say the least! With a Cert Template on our internal CA (windows), at least we can automate things well enough!
  - formerly_proven 2y ago
    Yeah it's almost weird it doesn't seem to exist, at least publicly. My megacorp created their own protocol for this purpose (though it might actually predate ACME, I'm not sure), and a bunch of in-house people and suppliers created the necessary middlewares to integrate it into stuff like cert-manager and such (basically everything that needs a TLS certificate and is deployed more than thrice). I imagine many larger companies have very similar things, with the only material difference being different organizational OIDs for the proprietary extension fields (I found it quite cute when I learned that the corp created a very neat subtree beneath its organization OID).

maccard 2y ago

I’ve unfortunately seen the opposite - internal apps are now back to being deployed over VPN and HTTP

Pxtl 2y ago

At this point I wish we could just get all our clients to say "self-signed is fine if you're connecting to a .LOCAL domain name". https is intrinsically useful over raw http, but the overhead of setting up centralized certs for non-public domains is just dumb. Give us a big global *.local cert we can all cheat with, so I don't have to blast my credentials in the clear when I log into my router's admin page.

JackSlateur 2y ago

Haa, yes ! We have that, too ! Accepted warning in browsers ! curl -k ! verify=False ! Glorious future to the hacking industry !

tomjen3 2y ago

I would love to do that for my homelab, but not all docker containers trust root certs from the system so getting it right would have been a bigger challenge than dns hacking to get a valid certificate for something that can’t be accessed from outside the network. I am not willing to give credentials to alter my dns to a program. A security issue there would be too much risk.