8 ms·

This might just be me, but I expected this issue to be taken far less seriously than it has been in the given communication [0]. In the linked similar issue [1]

This might just be me, but I expected this issue to be taken far less seriously than it has been in the given communication [0]. In the linked similar issue [1], they even talk about revoking the certificates, which seems insane to me for being one second off. Is there any actual serious problem with this extra second I'm missing? [0] https://bugzilla.mozilla.org/show_bug.cgi?id=1715455 https://bugzilla.mozilla.org/show_bug.cgi?id=1715455 [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1708965 https://bugzilla.mozilla.org/show_bug.cgi?id=1708965

joshka 5y ago
I can't see an immediately obvious attack that would be available by having an extra second of certificate validity. I can think of a possible place it would be useful to know this value correctly - automated expiration monitoring and unit tests
pdpi 5y ago
Certificates have been mis-issued. No ifs, ands, or buts, that by itself means revocation has to be on the table. In practice, it's a small problem and revocation is an excessive response, but those are conclusions you reach after duly investigating the issue, not before.
jcranmer 5y ago
This is a "no brown M&Ms" kind of violation. The direct impact of this violation is basically nil. However, it's a violation that suggests that the requirements haven't been very carefully read and that conformance to them hasn't been very carefully audited. So how can you have be assured that there aren't violations of more important rules still lurking about?
- teraflop 5y ago
  There was a similar issue a while back, where an open-source system used by a number of CAs was by default configured to generate serial numbers with only 63 bits of entropy, when the CA/B forum rules require 64 bits: https://www.schneier.com/blog/archives/2019/03/cas_reissue_ove.html https://www.schneier.com/blog/archives/2019/03/cas_reissue_o... This incident was complicated by the fact that it was first noticed while investigating a particular CA (DarkMatter) that was already suspected of various shady activities. Most of the other affected organizations took the incident seriously, fixed the problem and revoked the affected certificates, as required by the letter of the rules, even though the actual risk was small. DarkMatter chose to spend a lot of time complaining and making weaselly arguments on the mailing list (like "a 64-bit number where the leading bit is always zero is still technically a 64-bit random number") which did not do much to convince anyone of their trustworthiness.
- andrewmunsell 5y ago
  In case anyone is confused, I believe the reference to "no brown M&Ms" is a reference to Van Halen's contracts: https://www.snopes.com/fact-check/brown-out/ https://www.snopes.com/fact-check/brown-out/ > So, when I would walk backstage, if I saw a brown M&M in that bowl … well, line-check the entire production. Guaranteed you’re going to arrive at a technical error. They didn’t read the contract. Guaranteed you’d run into a problem. Sometimes it would threaten to just destroy the whole show. Something like, literally, life-threatening.
  - cpeterso 5y ago
    Did Van Halen ever actually cancel any shows because of brown M&Ms? The Snopes article says David Lee Roth found a brown M&M, but the show proceeded (and technical problems caused damage to the venue floor). "I found some brown M&M’s, I went into full Shakespearean “What is this before me?” … you know, with the skull in one hand … and promptly trashed the dressing room. Dumped the buffet, kicked a hole in the door, twelve thousand dollars’ worth of fun. The staging sank through their floor. They didn’t bother to look at the weight requirements or anything, and this sank through their new flooring and did eighty thousand dollars’ worth of damage to the arena floor. The whole thing had to be replaced.
    - jjeaff 5y ago
      Vandalism and destruction of property is a crime, even if you say you will pay for it. I guess if you are rich or famous enough, everyone just let's everything slide.
      - ceejayoz 5y ago
        Unless, of course, said destruction of property is in said contract. Otherwise, "smash therapy" places couldn't exist. If I were writing contracts for rock stars, I'd probably look to account for tantrums and raging parties in the wording.
  - travisjungroth 5y ago
    There's something I don't get about that method, as clever as it sounds. The method is if there are brown M&Ms, then check the entire production for safety. What about the else? Don't check everything? So if the person in charge of the green room happens to be on top of things and picks out a few M&Ms, they don't line-check the entire production even though there could be something "literally life-threatening"? It's easy to say that the brown M&Ms acted as a signal, but I can't think of any way to depend on that signal that isn't reckless. Just check the actual production, since that seems to be an option.
- LeifCarrotson 5y ago
  On reading the explanation: > The Zlint project only attempts to check whether a certificate exceeds the maximum validity allowed by the baseline requirements [398 days], and is not configurable. To extend the 'no brown M&Ms' analogy, this would be like Van Halen learning the caterer was only checking for FDA safety requirements and not their specific needs. It's good that it's food safe, and a brown M&M (or one with a fleck of brown exposed) isn't going to hurt anyone, but it means that you've got a breakdown in your process. Specifically, Let's Encrypt needs a review of their linting process. The requirement was that the lifetime be set to 7775999 seconds, or 90 days, this particular tool may not have thrown an error if the time had been 180 days or 360 days, and instead only thrown an error at 399 days. One second is not a problem, but 300 days would be a problem for sure! How were the lints which they're running selected, in particular how was the rule 'e_tls_server_cert_valid_time_longer_than_398_days' accepted, knowing that their validity was shorter than that? Are there other rules where the linter is checking against the most liberal specs but Let's Encrypt is offering something more precise? Fuzzing against a validity clock of 7775998 seconds, 7775999 seconds, and 7776000 seconds would find this issue, are there any other parameters that can be fuzzed? Questions like this need to be asked.
- teachingassist 5y ago
  > This is a "no brown M&Ms" kind of violation. I see it more as a "no brwn M&Ms" violation. We would all assume the intent was to say no brown M&Ms, rather than imagine that no action was required.
progbits 5y ago
It makes for good practice. Ideally real and severe incidents are rare, so you should be doing testing exercises to ensure the processes and tools are in place. But when a real incident, even though seemingly minor one, comes along you might as well use it for practice and "do things right".
fullstop 5y ago
I'm all for revoking the certificates, but feel that ample time should be taken to study what sorts of issues revoking would cause. It would not surprise me if this study takes approximately 90 days to complete.
- apocalyptic0n3 5y ago
  They've done mass revocations in the past. There was one about a year ago from what I remember. So whatever study could be done, likely already has been. The question now is whether the issue is sever enough to warrant that action.
  - fullstop 5y ago
    I feel that you've missed the joke. Also, there is no way, at all, that they are going to revoke nearly all of their certs because they were valid for 1 second longer than they should have been. It would mark the end of Let's Encrypt if they did. We've lived with certs that were valid for a second longer than they should have been since the inception of Let's Encrypt, and three months won't kill anyone. edit follows: When they revoked 3% of their certificates, not all of them were able to renew in time due to physical server limitations. The renewals required server administrators to forcibly renew their certificates, and the email address associated with the certificate was contacted to let them know. It would be an unmitigated disaster if 185 million certificates were suddenly revoked.
    - tingletech 5y ago
      Let's Encrypt has capacity to reissue all certificates in 24 hours. I think this is why you are supposed to run certbot every day. https://letsencrypt.org/2021/02/10/200m-certs-24hrs.html https://letsencrypt.org/2021/02/10/200m-certs-24hrs.html
      - fullstop 5y ago
        They have the physical infrastructure to do so, but you've missed this part of article you linked: > In order to get all those certificates replaced, we need an efficient and automated way to notify ACME clients that they should perform early renewal. Normally ACME clients renew their certificates when one third of their lifetime is remaining, and don’t contact our servers otherwise. We published a draft extension to ACME last year that describes a way for clients to regularly poll ACME servers to find out about early-renewal events. We plan to polish up that draft, implement, and collaborate with clients and large integrators to get it implemented on the client side. Running certbot daily will currently do nothing. It won't think that the certificate needs to be replaced since it has not yet reached the limit required to renew.
Macha 5y ago
I think part of is that the CAB (the browser group setting standards for CAs) is much of the same people that initially supported Lets Encrypt, and also use Lets Encrypt as a justification for tightening requirements on other CAs. They really do _not_ want to be seen to be giving Lets Encrypt special treatment because of the connection between the CAB members and Lets Encrypt and the threat that Lets Encrypt poses to the business models of the traditional CAs. So a violation of the rules by Lets Encrypt is being dealt with by the letter, even when the issue clearly does not warrant that level of reaction. (See also the time Google Search banned Google Chrome from results for a month when one of their advertising campaigns breached the rules for paid SEO)
hda111 5y ago
IMHO they must revoke them because when they sign incorrect data it would affect their reputation.

joshka 5y ago

I can't see an immediately obvious attack that would be available by having an extra second of certificate validity. I can think of a possible place it would be useful to know this value correctly - automated expiration monitoring and unit tests

pdpi 5y ago

Certificates have been mis-issued. No ifs, ands, or buts, that by itself means revocation has to be on the table. In practice, it's a small problem and revocation is an excessive response, but those are conclusions you reach after duly investigating the issue, not before.

jcranmer 5y ago

This is a "no brown M&Ms" kind of violation. The direct impact of this violation is basically nil. However, it's a violation that suggests that the requirements haven't been very carefully read and that conformance to them hasn't been very carefully audited. So how can you have be assured that there aren't violations of more important rules still lurking about?

teraflop 5y ago
There was a similar issue a while back, where an open-source system used by a number of CAs was by default configured to generate serial numbers with only 63 bits of entropy, when the CA/B forum rules require 64 bits: https://www.schneier.com/blog/archives/2019/03/cas_reissue_ove.html https://www.schneier.com/blog/archives/2019/03/cas_reissue_o... This incident was complicated by the fact that it was first noticed while investigating a particular CA (DarkMatter) that was already suspected of various shady activities. Most of the other affected organizations took the incident seriously, fixed the problem and revoked the affected certificates, as required by the letter of the rules, even though the actual risk was small. DarkMatter chose to spend a lot of time complaining and making weaselly arguments on the mailing list (like "a 64-bit number where the leading bit is always zero is still technically a 64-bit random number") which did not do much to convince anyone of their trustworthiness.
andrewmunsell 5y ago
In case anyone is confused, I believe the reference to "no brown M&Ms" is a reference to Van Halen's contracts: https://www.snopes.com/fact-check/brown-out/ https://www.snopes.com/fact-check/brown-out/ > So, when I would walk backstage, if I saw a brown M&M in that bowl … well, line-check the entire production. Guaranteed you’re going to arrive at a technical error. They didn’t read the contract. Guaranteed you’d run into a problem. Sometimes it would threaten to just destroy the whole show. Something like, literally, life-threatening.
- cpeterso 5y ago
  Did Van Halen ever actually cancel any shows because of brown M&Ms? The Snopes article says David Lee Roth found a brown M&M, but the show proceeded (and technical problems caused damage to the venue floor). "I found some brown M&M’s, I went into full Shakespearean “What is this before me?” … you know, with the skull in one hand … and promptly trashed the dressing room. Dumped the buffet, kicked a hole in the door, twelve thousand dollars’ worth of fun. The staging sank through their floor. They didn’t bother to look at the weight requirements or anything, and this sank through their new flooring and did eighty thousand dollars’ worth of damage to the arena floor. The whole thing had to be replaced.
  - jjeaff 5y ago
    Vandalism and destruction of property is a crime, even if you say you will pay for it. I guess if you are rich or famous enough, everyone just let's everything slide.
    - ceejayoz 5y ago
      Unless, of course, said destruction of property is in said contract. Otherwise, "smash therapy" places couldn't exist. If I were writing contracts for rock stars, I'd probably look to account for tantrums and raging parties in the wording.
- travisjungroth 5y ago
  There's something I don't get about that method, as clever as it sounds. The method is if there are brown M&Ms, then check the entire production for safety. What about the else? Don't check everything? So if the person in charge of the green room happens to be on top of things and picks out a few M&Ms, they don't line-check the entire production even though there could be something "literally life-threatening"? It's easy to say that the brown M&Ms acted as a signal, but I can't think of any way to depend on that signal that isn't reckless. Just check the actual production, since that seems to be an option.
LeifCarrotson 5y ago
On reading the explanation: > The Zlint project only attempts to check whether a certificate exceeds the maximum validity allowed by the baseline requirements [398 days], and is not configurable. To extend the 'no brown M&Ms' analogy, this would be like Van Halen learning the caterer was only checking for FDA safety requirements and not their specific needs. It's good that it's food safe, and a brown M&M (or one with a fleck of brown exposed) isn't going to hurt anyone, but it means that you've got a breakdown in your process. Specifically, Let's Encrypt needs a review of their linting process. The requirement was that the lifetime be set to 7775999 seconds, or 90 days, this particular tool may not have thrown an error if the time had been 180 days or 360 days, and instead only thrown an error at 399 days. One second is not a problem, but 300 days would be a problem for sure! How were the lints which they're running selected, in particular how was the rule 'e_tls_server_cert_valid_time_longer_than_398_days' accepted, knowing that their validity was shorter than that? Are there other rules where the linter is checking against the most liberal specs but Let's Encrypt is offering something more precise? Fuzzing against a validity clock of 7775998 seconds, 7775999 seconds, and 7776000 seconds would find this issue, are there any other parameters that can be fuzzed? Questions like this need to be asked.
teachingassist 5y ago
> This is a "no brown M&Ms" kind of violation. I see it more as a "no brwn M&Ms" violation. We would all assume the intent was to say no brown M&Ms, rather than imagine that no action was required.

progbits 5y ago

It makes for good practice. Ideally real and severe incidents are rare, so you should be doing testing exercises to ensure the processes and tools are in place. But when a real incident, even though seemingly minor one, comes along you might as well use it for practice and "do things right".

fullstop 5y ago

I'm all for revoking the certificates, but feel that ample time should be taken to study what sorts of issues revoking would cause. It would not surprise me if this study takes approximately 90 days to complete.

apocalyptic0n3 5y ago
They've done mass revocations in the past. There was one about a year ago from what I remember. So whatever study could be done, likely already has been. The question now is whether the issue is sever enough to warrant that action.
- fullstop 5y ago
  I feel that you've missed the joke. Also, there is no way, at all, that they are going to revoke nearly all of their certs because they were valid for 1 second longer than they should have been. It would mark the end of Let's Encrypt if they did. We've lived with certs that were valid for a second longer than they should have been since the inception of Let's Encrypt, and three months won't kill anyone. edit follows: When they revoked 3% of their certificates, not all of them were able to renew in time due to physical server limitations. The renewals required server administrators to forcibly renew their certificates, and the email address associated with the certificate was contacted to let them know. It would be an unmitigated disaster if 185 million certificates were suddenly revoked.
  - tingletech 5y ago
    Let's Encrypt has capacity to reissue all certificates in 24 hours. I think this is why you are supposed to run certbot every day. https://letsencrypt.org/2021/02/10/200m-certs-24hrs.html https://letsencrypt.org/2021/02/10/200m-certs-24hrs.html
    - fullstop 5y ago
      They have the physical infrastructure to do so, but you've missed this part of article you linked: > In order to get all those certificates replaced, we need an efficient and automated way to notify ACME clients that they should perform early renewal. Normally ACME clients renew their certificates when one third of their lifetime is remaining, and don’t contact our servers otherwise. We published a draft extension to ACME last year that describes a way for clients to regularly poll ACME servers to find out about early-renewal events. We plan to polish up that draft, implement, and collaborate with clients and large integrators to get it implemented on the client side. Running certbot daily will currently do nothing. It won't think that the certificate needs to be replaced since it has not yet reached the limit required to renew.

Macha 5y ago

I think part of is that the CAB (the browser group setting standards for CAs) is much of the same people that initially supported Lets Encrypt, and also use Lets Encrypt as a justification for tightening requirements on other CAs. They really do _not_ want to be seen to be giving Lets Encrypt special treatment because of the connection between the CAB members and Lets Encrypt and the threat that Lets Encrypt poses to the business models of the traditional CAs. So a violation of the rules by Lets Encrypt is being dealt with by the letter, even when the issue clearly does not warrant that level of reaction. (See also the time Google Search banned Google Chrome from results for a month when one of their advertising campaigns breached the rules for paid SEO)

hda111 5y ago

IMHO they must revoke them because when they sign incorrect data it would affect their reputation.