5 ms·

A similar problem with Azure happened way back in 2013: https://www.computerworld.com/article/2495453/microsoft-s-azure-service-hit-by-expired-ssl-certificate.h

A similar problem with Azure happened way back in 2013: https://www.computerworld.com/article/2495453/microsoft-s-azure-service-hit-by-expired-ssl-certificate.html https://www.computerworld.com/article/2495453/microsoft-s-az... More recently, it happened with Ericsson: https://www.theverge.com/2018/12/7/18130323/ericsson-software-certificate-o2-softbank-uk-japan-smartphone-4g-network-outage https://www.theverge.com/2018/12/7/18130323/ericsson-softwar... This article has some information about how Let's Encrypt enabled an "automated process that handles renewals": https://duo.com/decipher/proposal-to-make-https-certificate-expire-yearly-back-on-the-table https://duo.com/decipher/proposal-to-make-https-certificate-... I wonder if such a process should be made an industry standard? Does anyone know if there are any proposals for it?

riffic 7y ago
It's called ACME protocol and it's RFC 8555. https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment https://en.wikipedia.org/wiki/Automated_Certificate_Manageme...
cbhl 7y ago
Let's Encrypt literally is an implementation of the industry standard; the standard is called Automatic Certificate Management Environment. https://tools.ietf.org/html/rfc8555 https://tools.ietf.org/html/rfc8555
- deleted 7y ago
  [deleted]
- cpitman 7y ago
  But, somewhat annoyingly, it is only seen as applicable to the public internet. There's no effort to make ACME based CAs for non-internet usage.
  - pmlnr 7y ago
    For internal use, create and distribute you own root CA with self-signed certificates.
    - cpitman 7y ago
      Sure, distribute your own root CA, but then how do people get signed certificates? I tend to work with large companies, where getting a signed certificate involves opening a ticket and waiting for someone on the other side to respond. ACME would be ideal, but the official response of Let's Encrypt is that ACME is overkill for corporate environments and you should roll your own certificate automation.
      - Already__Taken 7y ago
        Delegating enroll permissions is a solved problem technically at least in a windows domain. At that point it's an org policy problem and ACME won't help.
      - Spooky23 7y ago
        In an enterprise, you use the API that a CA provides, and build it into the ticketing system. I helped build a system that took care of this with ServiceNow a few times now. At one place we aggressively policed external facing certificates. Don’t follow the process, your service gets whacked. It’s a process you should look into, because the compliance regimes will start paying attention to it someday soon.
    - eliaspro 7y ago
      And by letting smallstep/certificates [1] handle ACME, it's just as easy as using LetsEncrypt for public certificates. [1] https://github.com/smallstep/certificates https://github.com/smallstep/certificates
      - cpitman 7y ago
        This is great, thanks for sharing.
      - 7y ago
  - dpacmittal 7y ago
    I think you can create your own acme server for that
    - dilyevsky 7y ago
      No need to do acme when you can establish user identity and associated policies via internally deployed authn/z service
  - dilyevsky 7y ago
    Because everyone just does normal pki via api (like vault). No need for acme flow when you control entire system
  - tialaramex 7y ago
    In practice most non-Internet PKIs don't actually want to do proof of control which is ACME's secret sauce. As you'll see much of the document is about the proof of control, because that's both the hardest problem and the only reason for ACME to exist at all. In private PKIs either proof of control over the name is considered an out-of-band problem or it's elided altogether. I wrote about this when Peter Gutmann was being angry about ACME years ago, Gutmann saw ACME as redundant because of existing protocols like SCEP. If you want certificate issuance automation but you don't need proof of control over the names then you don't need ACME - SCEP (or half a dozen others) are fine for this purpose and you should use those. Now, of course if you don't want proof of control but you do want automation it's worth taking a moment to reconsider exactly what your goals really are: What is a certificate doing for me here? What exactly am I certifying, if anything? But if the technical requirement is there, regardless of whether it makes philosophical sense, SCEP delivers and ACME is overkill. As I wrote in my thread with Peter, it's usual with things like SCEP to provide a default implementation in which the part where you only give certificates to the Right People is left marked /* TODO */. Thus often the result is pure security theatre, certificates are issued equally to bad guys or good guys without distinction and nothing at all is really being certified. Manual processes may or may not be better in terms of actually certifying anything.
  - sebazzz 7y ago
    Unless your DNS public. That still allows Let's Encrypt to verify you and is actually used by RavenDB to provide SSL certificates to internal instances.
giovannibajo1 7y ago
Automated renewal doesn’t mean that you don’t need to have monitoring. I’ve seen many letsencrypt-baked servers generating certificate errors because the automated renewal system broke (misconfigurations, borked updates, corporate-level firewalls/proxies blocking requests, etc.). Even with letsencrypt you still MUST have a monitoring system, notifications going to the right person, and in general an organization that can act on this. The problem is often not technical: your organization must be structured in a way that those notifications are acted upon. If anything, letsencrypt lessens the frequency of those notifications so I posit that it’s even worse from the point of view of validating your organization: with regular certs, you get notified once every year or two. Anyway, letsencrypt is a good thing: it’s just not a solution to this problem.
- YawningAngel 7y ago
  Let's Encrypt does exactly the right thing, which is to email you if a certificate is expiring without being renewed. Pretty much the behaviour you'd want from any CA, no?
  - omh 7y ago
    That's a great start. But in a non-technical organisation, who should those messages go to? Often the initial LetsEncrypt setup will be handled, correctly, by some IT staff. Then it might break several months or years later for some odd reason. The organisational challenge is to get the message through to someone who understands it and will act on it.
    - outworlder 7y ago
      > The organisational challenge is to get the message through to someone who understands it and will act on it. Yes, and to keep a very infrequently used channel working and up-to-date.
      - brongondwana 7y ago
        This is why letsencrypt very sensibly creates short-lived certificates, meaning that the channel is not infrequently used.
alpb 7y ago
Pro-tip: Renew your certs automated every 30 days and set to expire 90 days and set alerts at 35 days.
andrewcamel 7y ago
AppViewX and Venafi are the two main competitors in this space of certificate lifecycle automation - many Fortune 1000s are customers of one or the other to prevent exactly this type of problem - Venafi is the larger incumbent but AppViewX has the newer platform and is replacing Venafi in some key accounts Challenge is discovering all the certs that exist across the org (10k+ often) and then having a fully automated renewal process requires a pretty advanced / complex platform - don’t know anyone who has built this tech successfully in house - curious if anyone has though in the HN community? Disclosure I’m an investor in AppViewX

riffic 7y ago

It's called ACME protocol and it's RFC 8555. https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment https://en.wikipedia.org/wiki/Automated_Certificate_Manageme...

cbhl 7y ago

Let's Encrypt literally is an implementation of the industry standard; the standard is called Automatic Certificate Management Environment. https://tools.ietf.org/html/rfc8555 https://tools.ietf.org/html/rfc8555

deleted 7y ago
[deleted]
cpitman 7y ago
But, somewhat annoyingly, it is only seen as applicable to the public internet. There's no effort to make ACME based CAs for non-internet usage.
- pmlnr 7y ago
  For internal use, create and distribute you own root CA with self-signed certificates.
  - cpitman 7y ago
    Sure, distribute your own root CA, but then how do people get signed certificates? I tend to work with large companies, where getting a signed certificate involves opening a ticket and waiting for someone on the other side to respond. ACME would be ideal, but the official response of Let's Encrypt is that ACME is overkill for corporate environments and you should roll your own certificate automation.
    - Already__Taken 7y ago
      Delegating enroll permissions is a solved problem technically at least in a windows domain. At that point it's an org policy problem and ACME won't help.
    - Spooky23 7y ago
      In an enterprise, you use the API that a CA provides, and build it into the ticketing system. I helped build a system that took care of this with ServiceNow a few times now. At one place we aggressively policed external facing certificates. Don’t follow the process, your service gets whacked. It’s a process you should look into, because the compliance regimes will start paying attention to it someday soon.
  - eliaspro 7y ago
    And by letting smallstep/certificates [1] handle ACME, it's just as easy as using LetsEncrypt for public certificates. [1] https://github.com/smallstep/certificates https://github.com/smallstep/certificates
    - cpitman 7y ago
      This is great, thanks for sharing.
    - 7y ago
- dpacmittal 7y ago
  I think you can create your own acme server for that
  - dilyevsky 7y ago
    No need to do acme when you can establish user identity and associated policies via internally deployed authn/z service
- dilyevsky 7y ago
  Because everyone just does normal pki via api (like vault). No need for acme flow when you control entire system
- tialaramex 7y ago
  In practice most non-Internet PKIs don't actually want to do proof of control which is ACME's secret sauce. As you'll see much of the document is about the proof of control, because that's both the hardest problem and the only reason for ACME to exist at all. In private PKIs either proof of control over the name is considered an out-of-band problem or it's elided altogether. I wrote about this when Peter Gutmann was being angry about ACME years ago, Gutmann saw ACME as redundant because of existing protocols like SCEP. If you want certificate issuance automation but you don't need proof of control over the names then you don't need ACME - SCEP (or half a dozen others) are fine for this purpose and you should use those. Now, of course if you don't want proof of control but you do want automation it's worth taking a moment to reconsider exactly what your goals really are: What is a certificate doing for me here? What exactly am I certifying, if anything? But if the technical requirement is there, regardless of whether it makes philosophical sense, SCEP delivers and ACME is overkill. As I wrote in my thread with Peter, it's usual with things like SCEP to provide a default implementation in which the part where you only give certificates to the Right People is left marked /* TODO */. Thus often the result is pure security theatre, certificates are issued equally to bad guys or good guys without distinction and nothing at all is really being certified. Manual processes may or may not be better in terms of actually certifying anything.
- sebazzz 7y ago
  Unless your DNS public. That still allows Let's Encrypt to verify you and is actually used by RavenDB to provide SSL certificates to internal instances.

giovannibajo1 7y ago

Automated renewal doesn’t mean that you don’t need to have monitoring. I’ve seen many letsencrypt-baked servers generating certificate errors because the automated renewal system broke (misconfigurations, borked updates, corporate-level firewalls/proxies blocking requests, etc.). Even with letsencrypt you still MUST have a monitoring system, notifications going to the right person, and in general an organization that can act on this. The problem is often not technical: your organization must be structured in a way that those notifications are acted upon. If anything, letsencrypt lessens the frequency of those notifications so I posit that it’s even worse from the point of view of validating your organization: with regular certs, you get notified once every year or two. Anyway, letsencrypt is a good thing: it’s just not a solution to this problem.

YawningAngel 7y ago
Let's Encrypt does exactly the right thing, which is to email you if a certificate is expiring without being renewed. Pretty much the behaviour you'd want from any CA, no?
- omh 7y ago
  That's a great start. But in a non-technical organisation, who should those messages go to? Often the initial LetsEncrypt setup will be handled, correctly, by some IT staff. Then it might break several months or years later for some odd reason. The organisational challenge is to get the message through to someone who understands it and will act on it.
  - outworlder 7y ago
    > The organisational challenge is to get the message through to someone who understands it and will act on it. Yes, and to keep a very infrequently used channel working and up-to-date.
    - brongondwana 7y ago
      This is why letsencrypt very sensibly creates short-lived certificates, meaning that the channel is not infrequently used.

alpb 7y ago

Pro-tip: Renew your certs automated every 30 days and set to expire 90 days and set alerts at 35 days.

andrewcamel 7y ago

AppViewX and Venafi are the two main competitors in this space of certificate lifecycle automation - many Fortune 1000s are customers of one or the other to prevent exactly this type of problem - Venafi is the larger incumbent but AppViewX has the newer platform and is replacing Venafi in some key accounts Challenge is discovering all the certs that exist across the org (10k+ often) and then having a fully automated renewal process requires a pretty advanced / complex platform - don’t know anyone who has built this tech successfully in house - curious if anyone has though in the HN community? Disclosure I’m an investor in AppViewX