8 ms·

> The only things that continue to amaze me are the number of (mostly "enterprise") software products that simply won't get with the times Yeah, no one's rewri

> The only things that continue to amaze me are the number of (mostly "enterprise") software products that simply won't get with the times Yeah, no one's rewriting a bunch of software to support automating a specific, internet-facing, sometimes-reliable CA. Yes it's ACME, a standard you say. A standard protocol with nonstop changing profile requirements at LE's whim. Who's going to keep updating the software every 3 months to keep up? When the WebPKI sneeze in a different direction and change their minds yet again. Because 45 will become 30 will become 7 and they won't stop till the lifetime is 6 hours. "Enterprise" products are more often than not using internal PKI so it's a waste. I would like to see the metrics on how much time and resources are wasted babysitting all this automation vs. going in and updating a certificate manually once a year and not having to worry the automation will fail in a week.

ZeroConcerns 11mo ago
> I would like to see the metrics Well, I could regale you with my anecdotes on how all my 'grab a LetsEncrypt cert on Docker deploy and whenever remaining lifetime goes below 75%' services have not had a single TLS-related outage in at least a year, and how we just had a multi-day meltdown caused by a load-bearing cert that everyone forgot about expiring, but I doubt you'll be interested. I'm not here to take away your right to set up a few more 5-year+ internal CAs (or, hey, how about an entire self-signed hierarchy? can't be that hard...), and neither is LetsEncrypt. But on the Big Bad Internet (where Google is the leading cause of More Security, and LetsEncrypt is merely the leading vendor catering to that), things have moved on slightly.
happymellon 11mo ago
> I would like to see the metrics on how much time and resources are wasted babysitting all this automation vs. going in and updating a certificate manually once a year and not having to worry the automation will fail in a week. I have multiple systems. Legacy that I've inherited, and modern that are able to automatically update their certs. The automated updates require almost zero maintenance, there was a week a couple of years ago when I had to switch some configuration for a new root cert. The legacy system requires manual installation and multiple weeks of my time every year because of the number of environments, and since generation of the certs requires me to file a request for someone to manually create it, they invariably typo something and it has to be redone everywhere. So multiple engineers, over multiple weeks? Manual process at a guess is £50k pa, while the automated is close to an annual £0?
crote 11mo ago
The change in validity does not in any way alter the protocol itself. As mentioned in the linked blog post: if you've already got automated cert renewal, it'll almost certainly require zero change. After all, the logical approach is to schedule a daily cron job for your renewal process, which only contacts the server when the current cert has less than X days of validity remaining. Scheduling a one-off cron job every, say, 60 days would be extremely error-prone! With regards to future reductions: the point is to force companies to automate renewal, as they have proven to be unable to renew in time during incidents due to dozens of layers of bureaucracy. 45 days means one renewal a month, which is probably sufficient for that. Anything below 7 days isn't likely to happen, because at that point the industry considers certification revocation unnecessary. Internal PKI is not relevant here. Anyone with weird internal cert use shouldn't be using public certs anyways, so moving those people to certs backed by self-signed internal CAs is a win. They are free to use whatever validity they want with those.
- basscomm 11mo ago
  > the point is to force companies to automate renewal Cool. I'm a small-time webmaster with a couple of hobby sites with no more than a handful of visitors. Why do I need to set up automation to renew certs every 45 days, too?
  - nickf 11mo ago
    Is the certificate you use on your website any different to that on google.com? Does/could a browser know this and act differently?
  - Shish2k 11mo ago
    I'm a small time webmaster and I haven't "set up" any automation - for my shared-hosting sites, the host has it built in; and for my self-hosted sites, the web server has it built in
    - iggldiggl 11mo ago
      The problem is that this breaks down if you don't want to leak any obscure subdomains you might be using via CT-logs – shared hosting rarely supports DNS-based certificate renewals for wildcard certificates, and even less so for domains hosted by an external registrar. (Even for a fully self-hosted system you'd still have to figure out how to interface the certificate renewal mechanism with your DNS provider, so not as easy to set up as individual certificates for each subdomain.)
homebrewer 11mo ago
In circles I'm running in, automatic certificate renewal has not caused a single problem over 7 years of using it, and whatever time was spent on setting it up, has paid many times over, both in saving effort on renewal, and in putting out fires when (not if) someone forgets to renew a certificate. You just have to be careful picking your automation — I haven't been impressed with certbot, for example. Also, everything is using https now. Living in a low-income country, certificates were too expensive to use them where they weren't absolutely required, but not anymore. This is orthogonal to automation, I'm just pointing out that LE is not as demonic as you make it out to be. I'm afraid enterprise users are on their own, probably approximately no-one else is interested in going back to the old ways of doing it. (Maybe embedded.)
- imtringued 11mo ago
  Forcing automation would be fine if the default software package (certbot) was any good but from my experience certbot is simply not fit for purpose. Certbot doesn't support the industry standard PKCS#12 format, which makes it extremely brittle for anyone using a Java based webserver. Instead it uses the non-standard PEM format which requires conversion before usage. That conversion step breaks all the time and requires manual intervention. It's ridiculous.
  - arccy 11mo ago
    at this point PEM is more standard and prevalent than pkcs#12
  - nine_k 11mo ago
    PEM is very standard. Calling `openssl pkcs12` also should not be hard; IDK about certbot, but there is a hook for acmetool (which I use) that does just that for you: https://github.com/dlitz/acmetool-pkcs12-hooks https://github.com/dlitz/acmetool-pkcs12-hooks
  - cpach 11mo ago
    I hear ya. I’m also not fond of certbot and other existing clients. The best solution I’ve found so far was to implement a custom cert manager using the formidable acmez library.
arp242 11mo ago
> A standard protocol with nonstop changing profile requirements at LE's whim. Who's going to keep updating the software every 3 months to keep up? It really doesn't change that often. And whether this is a "breaking" change is something that's debatable (you shouldn't hard-code the cert lifetime, but I suspect many programs do, so de-facto it may or may not be a breaking change). If you look at the Go implementation for example (https://github.com/golang/crypto/commits/master/acme https://github.com/golang/crypto/commits/master/acme), then there haven't been any major changes since it was first written in 2018: just bugfixes and basic maintenance tasks. Maybe I missed a commit but it certainly doesn't require 3-month update cycles. You can probably take that 2018 code and it'll work fine today. And since it doesn't hard-code the 90 days it will continue to work in 2028 when this change is made. So it's more "you need to keep updating the software every decade to keep up".
ForHackernews 11mo ago
I'll happily take failed automation with errors I can fix over being totally screwed when we find out the guy who did it every year left the company this year and nobody else knows anything about it.
patmorgan23 11mo ago
All the enterprise software needs to do is create an API for configuring the certificate in the product. Then they can integrate Certbot or one of the many other ACME solutions, or let the customer do it (they are enterprises after all).
rainsford 11mo ago
> I would like to see the metrics on how much time and resources are wasted babysitting all this automation vs. going in and updating a certificate manually once a year and not having to worry the automation will fail in a week. I would also like to see those metrics, because I strongly suspect the cost is dramatically in favor of automation, especially when you consider the improved security posture you get from short lived certificates. My personal experience with Let's Encrypt has been that the only hassle is in systems that don't implement automation, so I get the worst of both worlds having to effectively manually renew short lived certificates every 90 days. CRL based revocation is largely a failure, both because implementation is a problem and because knowing when to revoke a certificate is hard. So the only way to achieve the goal of security in WebPKI that's resilient to private key compromise is short-lived certificates. And the only way to implement those is with automated renewal.

ZeroConcerns 11mo ago

> I would like to see the metrics Well, I could regale you with my anecdotes on how all my 'grab a LetsEncrypt cert on Docker deploy and whenever remaining lifetime goes below 75%' services have not had a single TLS-related outage in at least a year, and how we just had a multi-day meltdown caused by a load-bearing cert that everyone forgot about expiring, but I doubt you'll be interested. I'm not here to take away your right to set up a few more 5-year+ internal CAs (or, hey, how about an entire self-signed hierarchy? can't be that hard...), and neither is LetsEncrypt. But on the Big Bad Internet (where Google is the leading cause of More Security, and LetsEncrypt is merely the leading vendor catering to that), things have moved on slightly.

happymellon 11mo ago

> I would like to see the metrics on how much time and resources are wasted babysitting all this automation vs. going in and updating a certificate manually once a year and not having to worry the automation will fail in a week. I have multiple systems. Legacy that I've inherited, and modern that are able to automatically update their certs. The automated updates require almost zero maintenance, there was a week a couple of years ago when I had to switch some configuration for a new root cert. The legacy system requires manual installation and multiple weeks of my time every year because of the number of environments, and since generation of the certs requires me to file a request for someone to manually create it, they invariably typo something and it has to be redone everywhere. So multiple engineers, over multiple weeks? Manual process at a guess is £50k pa, while the automated is close to an annual £0?

crote 11mo ago

The change in validity does not in any way alter the protocol itself. As mentioned in the linked blog post: if you've already got automated cert renewal, it'll almost certainly require zero change. After all, the logical approach is to schedule a daily cron job for your renewal process, which only contacts the server when the current cert has less than X days of validity remaining. Scheduling a one-off cron job every, say, 60 days would be extremely error-prone! With regards to future reductions: the point is to force companies to automate renewal, as they have proven to be unable to renew in time during incidents due to dozens of layers of bureaucracy. 45 days means one renewal a month, which is probably sufficient for that. Anything below 7 days isn't likely to happen, because at that point the industry considers certification revocation unnecessary. Internal PKI is not relevant here. Anyone with weird internal cert use shouldn't be using public certs anyways, so moving those people to certs backed by self-signed internal CAs is a win. They are free to use whatever validity they want with those.

basscomm 11mo ago
> the point is to force companies to automate renewal Cool. I'm a small-time webmaster with a couple of hobby sites with no more than a handful of visitors. Why do I need to set up automation to renew certs every 45 days, too?
- nickf 11mo ago
  Is the certificate you use on your website any different to that on google.com? Does/could a browser know this and act differently?
- Shish2k 11mo ago
  I'm a small time webmaster and I haven't "set up" any automation - for my shared-hosting sites, the host has it built in; and for my self-hosted sites, the web server has it built in
  - iggldiggl 11mo ago
    The problem is that this breaks down if you don't want to leak any obscure subdomains you might be using via CT-logs – shared hosting rarely supports DNS-based certificate renewals for wildcard certificates, and even less so for domains hosted by an external registrar. (Even for a fully self-hosted system you'd still have to figure out how to interface the certificate renewal mechanism with your DNS provider, so not as easy to set up as individual certificates for each subdomain.)

homebrewer 11mo ago

In circles I'm running in, automatic certificate renewal has not caused a single problem over 7 years of using it, and whatever time was spent on setting it up, has paid many times over, both in saving effort on renewal, and in putting out fires when (not if) someone forgets to renew a certificate. You just have to be careful picking your automation — I haven't been impressed with certbot, for example. Also, everything is using https now. Living in a low-income country, certificates were too expensive to use them where they weren't absolutely required, but not anymore. This is orthogonal to automation, I'm just pointing out that LE is not as demonic as you make it out to be. I'm afraid enterprise users are on their own, probably approximately no-one else is interested in going back to the old ways of doing it. (Maybe embedded.)

imtringued 11mo ago
Forcing automation would be fine if the default software package (certbot) was any good but from my experience certbot is simply not fit for purpose. Certbot doesn't support the industry standard PKCS#12 format, which makes it extremely brittle for anyone using a Java based webserver. Instead it uses the non-standard PEM format which requires conversion before usage. That conversion step breaks all the time and requires manual intervention. It's ridiculous.
- arccy 11mo ago
  at this point PEM is more standard and prevalent than pkcs#12
- nine_k 11mo ago
  PEM is very standard. Calling `openssl pkcs12` also should not be hard; IDK about certbot, but there is a hook for acmetool (which I use) that does just that for you: https://github.com/dlitz/acmetool-pkcs12-hooks https://github.com/dlitz/acmetool-pkcs12-hooks
- cpach 11mo ago
  I hear ya. I’m also not fond of certbot and other existing clients. The best solution I’ve found so far was to implement a custom cert manager using the formidable acmez library.

arp242 11mo ago

> A standard protocol with nonstop changing profile requirements at LE's whim. Who's going to keep updating the software every 3 months to keep up? It really doesn't change that often. And whether this is a "breaking" change is something that's debatable (you shouldn't hard-code the cert lifetime, but I suspect many programs do, so de-facto it may or may not be a breaking change). If you look at the Go implementation for example (https://github.com/golang/crypto/commits/master/acme https://github.com/golang/crypto/commits/master/acme), then there haven't been any major changes since it was first written in 2018: just bugfixes and basic maintenance tasks. Maybe I missed a commit but it certainly doesn't require 3-month update cycles. You can probably take that 2018 code and it'll work fine today. And since it doesn't hard-code the 90 days it will continue to work in 2028 when this change is made. So it's more "you need to keep updating the software every decade to keep up".

ForHackernews 11mo ago

I'll happily take failed automation with errors I can fix over being totally screwed when we find out the guy who did it every year left the company this year and nobody else knows anything about it.

patmorgan23 11mo ago

All the enterprise software needs to do is create an API for configuring the certificate in the product. Then they can integrate Certbot or one of the many other ACME solutions, or let the customer do it (they are enterprises after all).

rainsford 11mo ago

> I would like to see the metrics on how much time and resources are wasted babysitting all this automation vs. going in and updating a certificate manually once a year and not having to worry the automation will fail in a week. I would also like to see those metrics, because I strongly suspect the cost is dramatically in favor of automation, especially when you consider the improved security posture you get from short lived certificates. My personal experience with Let's Encrypt has been that the only hassle is in systems that don't implement automation, so I get the worst of both worlds having to effectively manually renew short lived certificates every 90 days. CRL based revocation is largely a failure, both because implementation is a problem and because knowing when to revoke a certificate is hard. So the only way to achieve the goal of security in WebPKI that's resilient to private key compromise is short-lived certificates. And the only way to implement those is with automated renewal.