4 ms·

Tedious to set up, tedious to maintain. January 31 of this year I got an email telling me that my LE client used the older ACMEv1 protocol, not the newer ACMEv

by user1980 6y ago

Tedious to set up, tedious to maintain. January 31 of this year I got an email telling me that my LE client used the older ACMEv1 protocol, not the newer ACMEv2 protocol. They gave me 4 months notice to update my LE client to something compliant. I burnt the time and did the work. On March 3 myself and many others[0] got an email demanding that we manually re-issue our certificates because of a vulnerability discovered in the LE service. They gave us one day to comply, after that they would revoke the certificates and our users would receive security errors. I begrudgingly went through all my servers and issued the command to forcibly renew certificates. Not a huge burden for me, but likely a bigger burden for larger operations. As the feature set grows (new challenge types, wildcard support, etc.) and the service gets even more popular, it's going to be an even bigger target and the effects of a monoculture will really be felt. I'm starting to see the value in paying for certificates, and more specifically, using providers that don't provide a public certificate issuance API (or at least stick it behind a paywall.) How many times would LE have to accidentally issue gstatic.com or fbcdn.net before they get the Symantec treatment[1]? Too big to fail: It's not just for investment banks. And that should give anyone seeking a decentralized internet pause. [0]: https://www.zdnet.com/article/lets-encrypt-to-revoke-3-million-certificates-on-march-4-due-to-bug/ https://www.zdnet.com/article/lets-encrypt-to-revoke-3-milli... [1]: https://www.zdnet.com/article/mozilla-warns-it-plans-to-distrust-all-symantec-chained-certs-in-october/ https://www.zdnet.com/article/mozilla-warns-it-plans-to-dist...

na85 6y ago
>Tedious to set up, tedious to maintain. It took me maybe 10 minutes to set up for my nginx setup. Debian and OpenSUSE both package letsencrypt's certbot. Tedious to set up and maintain is essentially the opposite of my experience.
inetknght 6y ago
> How many times would LE have to accidentally issue gstatic.com or fbcdn.net before they get the Symantec treatment[1]? Too big to fail: It's not just for investment banks. And that should give anyone seeking a decentralized internet pause. I agree that some problems are unfortunate. But let's contrast for a moment. LetsEncrypt has demonstrated track record of quickly fixing issues. Symantec has a demonstrated track record of hiding issues instead of fixing them. It's wise to consider options carefully. LetsEncrypt isn't the be-all end-all service for TLS and your needs might not be compatible. But I don't think it's fair to shove LetsEncrypt aside just because it's had its share of problems. For a free service it's pretty damn reputable.
- treis 6y ago
  It's a good service, but I think the GP's point is that it's not trivial to do. Lots of websites probably went down because they missed that e-mail and we've seen lots of major websites go down due to some issue related to their certificate. Let's Encrypt has lowered the bar, but it's still a bar that needs to be overcome.
  - tialaramex 6y ago
    As we'd expect ISRG not only fixed the immediate problem they also accelerated plans to ensure that any similar problem would have less serious ill effects. In particular a current Certbot (or similar software from other developers) will conclude that it should try to replace a certificate which has been revoked and not only certificates that will shortly expire. So if a similar event happened, and you missed the email, your Certbot will treat the certificates much as if they'd expired and replace them automatically. Also if you didn't replace a revoked certificate the thing is: Online revocation is broken. Most of your users will not have noticed your certificate was revoked. Popular browsers do have an out-of-band way to enforce revocation but they didn't use it on that Let's Encrypt incident because they felt it was low risk. So maybe some people are running Internet Explorer (really?) or have explicitly turned on revocation, everybody else doesn't even see a warning page.
tialaramex 6y ago
> How many times would LE have to accidentally issue gstatic.com or fbcdn.net before they get the Symantec treatment Our concern with Symantec was inadequate oversight. This is not some clumsy "Three strikes and you're out" rule. Symantec did not have the culture needed to do the job properly and we had no confidence that their management was capable of instilling such a culture. If you're American or just follow American events somewhat you may have seen the "One rotten apple" argument being pulled apart in respect of problems with their police. Symantec used this argument, asserting on two occasions that their policies were fine but an employee had fallen short and this employee was terminated so now everything is fine. I am not sure I believe them but it doesn't matter because: That is not good enough. We need public CAs to design procedures so that merely incompetent or lazy employees cannot sabotage things. Because individual humans are by their nature incompetent and lazy, such problems are to be expected and must be allowed for in your processes. The big incident that blew up for Symantec was Crosscert. Symantec had not explicitly disclosed that the Crosscert relationship existed. In fact even if you read their paperwork closely (as we did after the incident) they actually simply did not disclose key facts about the relationship to anyone, not to their users, not to relying parties (ie you and me), and not to their independent auditor. Perhaps not even to their own board of directors (of course maybe private documents available to the board had such a disclosure). It is likely that in practice Symantec as a corporation was unaware of what Crosscert were doing. Even if one or two Symantec employees had a good idea, the organisation as a whole was ignorant. As a result there was in practice no oversight over this entirely separate entity in a foreign country issuing certificates! We concluded that building confidence in a new management of new infrastructure would take several years and that was the minimum we could allow. At first Symantec decided to fight this at the executive level, which of course only made us more confident that we'd been correct not to have confidence in them. When that failed (my impression is that trying to bully Google senior management isn't a good strategy) they settled upon a plan of selling their CA business instead. So all that's a long way from Let's Encrypt accidentally mis-issuing a certificate from their own systems to bad guys.

na85 6y ago

>Tedious to set up, tedious to maintain. It took me maybe 10 minutes to set up for my nginx setup. Debian and OpenSUSE both package letsencrypt's certbot. Tedious to set up and maintain is essentially the opposite of my experience.

inetknght 6y ago

> How many times would LE have to accidentally issue gstatic.com or fbcdn.net before they get the Symantec treatment[1]? Too big to fail: It's not just for investment banks. And that should give anyone seeking a decentralized internet pause. I agree that some problems are unfortunate. But let's contrast for a moment. LetsEncrypt has demonstrated track record of quickly fixing issues. Symantec has a demonstrated track record of hiding issues instead of fixing them. It's wise to consider options carefully. LetsEncrypt isn't the be-all end-all service for TLS and your needs might not be compatible. But I don't think it's fair to shove LetsEncrypt aside just because it's had its share of problems. For a free service it's pretty damn reputable.

treis 6y ago
It's a good service, but I think the GP's point is that it's not trivial to do. Lots of websites probably went down because they missed that e-mail and we've seen lots of major websites go down due to some issue related to their certificate. Let's Encrypt has lowered the bar, but it's still a bar that needs to be overcome.
- tialaramex 6y ago
  As we'd expect ISRG not only fixed the immediate problem they also accelerated plans to ensure that any similar problem would have less serious ill effects. In particular a current Certbot (or similar software from other developers) will conclude that it should try to replace a certificate which has been revoked and not only certificates that will shortly expire. So if a similar event happened, and you missed the email, your Certbot will treat the certificates much as if they'd expired and replace them automatically. Also if you didn't replace a revoked certificate the thing is: Online revocation is broken. Most of your users will not have noticed your certificate was revoked. Popular browsers do have an out-of-band way to enforce revocation but they didn't use it on that Let's Encrypt incident because they felt it was low risk. So maybe some people are running Internet Explorer (really?) or have explicitly turned on revocation, everybody else doesn't even see a warning page.

tialaramex 6y ago

> How many times would LE have to accidentally issue gstatic.com or fbcdn.net before they get the Symantec treatment Our concern with Symantec was inadequate oversight. This is not some clumsy "Three strikes and you're out" rule. Symantec did not have the culture needed to do the job properly and we had no confidence that their management was capable of instilling such a culture. If you're American or just follow American events somewhat you may have seen the "One rotten apple" argument being pulled apart in respect of problems with their police. Symantec used this argument, asserting on two occasions that their policies were fine but an employee had fallen short and this employee was terminated so now everything is fine. I am not sure I believe them but it doesn't matter because: That is not good enough. We need public CAs to design procedures so that merely incompetent or lazy employees cannot sabotage things. Because individual humans are by their nature incompetent and lazy, such problems are to be expected and must be allowed for in your processes. The big incident that blew up for Symantec was Crosscert. Symantec had not explicitly disclosed that the Crosscert relationship existed. In fact even if you read their paperwork closely (as we did after the incident) they actually simply did not disclose key facts about the relationship to anyone, not to their users, not to relying parties (ie you and me), and not to their independent auditor. Perhaps not even to their own board of directors (of course maybe private documents available to the board had such a disclosure). It is likely that in practice Symantec as a corporation was unaware of what Crosscert were doing. Even if one or two Symantec employees had a good idea, the organisation as a whole was ignorant. As a result there was in practice no oversight over this entirely separate entity in a foreign country issuing certificates! We concluded that building confidence in a new management of new infrastructure would take several years and that was the minimum we could allow. At first Symantec decided to fight this at the executive level, which of course only made us more confident that we'd been correct not to have confidence in them. When that failed (my impression is that trying to bully Google senior management isn't a good strategy) they settled upon a plan of selling their CA business instead. So all that's a long way from Let's Encrypt accidentally mis-issuing a certificate from their own systems to bad guys.