13 ms·
2021.06.08 Certificate Lifetime Incident
- billpg 5y agoIs this a joke? Just leave the extra second on. Any release of software has a risk associated with it so you shouldn't risk it unless the update is worth the risk. The time they spent investigating and updating is time wasted. Anything else they could have done would have been a better use of that time. The time I spent writing this comment was a better use of time. And this comment is really not important.
- Sebguer 5y agoThis is not the approach you can take to writing software that the security of the entire internet relies upon.
- anyfoo 5y agoYou'd be surprised how often something that does not seem like a big deal is some kind of deal after all, sometimes even a big one. Software is complex. So far, all signs point to this not being a big deal (EDIT: from the software side), but that is contingent on the aforementioned investigation. Others have also mentioned how it is a matter of principle: Certificates were misissued, where do you draw the line if not at that general fact?
- NovemberWhiskey 5y agoThis isn't a software complexity issue; the only reason we're talking about this is the sensitivity of root trust store process. CAs are required to issue Certificate Practice Statements (CPS) which tell the world-at-large the parameters under which they issue certificates. Here's the CPS for Let's Encrypt that was in effect when this problem was detected: https://letsencrypt.org/documents/isrg-cps-v3.2/ https://letsencrypt.org/documents/isrg-cps-v3.2/ These CPSs are used by the various browser/operating system developers to make decisions about which CAs should be included in their root trust stores; i.e. basically whether you're a CA accepted by "the internet". Let's Encrypt's CPS said one thing (we issue certificates valid for 90 days), but their implementation did a different thing (issued certificates valid for 90 days + 1 second). An inability to adhere to your CPS is a big deal - it is effectively an internal controls failure; and self-reporting of failures to adhere to your CPS is essential. The owners of the root trust stores are going to want to see seriousness from the CAs when it comes to adherence to their CPS, even in apparently de minimis cases. e.g. refer to Mozilla's policy guide and its reliance on the CA CPS: https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/ https://www.mozilla.org/en-US/about/governance/policies/secu... No-one is suggesting that a one second difference in validity makes any difference in the real world, and Let's Encrypt has already uprev'd their CPS to make it clear that actually all they promise is that certificates will be valid "less than 100 days": https://letsencrypt.org/documents/isrg-cps-v3.3/ https://letsencrypt.org/documents/isrg-cps-v3.3/
- anyfoo 5y agoThanks, I understand, but I wanted to address a broader point about software, which the comment I replied to talked about. To repeat: Sometimes software, or things that interact with software (which is minimally what this is), has issues that may seem small, but end up big because of complexity. This turned out not to be one of them. But it would be a consideration in an investigation, even though (or so I guess) it was likely straightforward to dismiss it quickly in this particular case. Nevertheless, the time to investigate potential repercussions of something so critical is not "wasted". I updated my comment above to clarify that this does not seem to be a big deal so far from the software side.
- kps 5y agoIt does literally say ‘90 days’. Pedantically, if that's read as a physical measurement, it's precise to plus or minus half a day, which more than covers the actual state. Perhaps Mozilla also erred in accepting a statement that said ‘90 days’ rather than ‘7776000 seconds’.
- zaarn 5y agoThe X.509 RFC and it's relevant substandards define a day to be 86400 seconds long. Not longer, not shorter. There is no half a day tolerance in there, only tolerance of less than a second (which LE exceeded by hitting an entire second).
- corroclaro 5y agoThis is yet another nail in why we need to get rid of the whole accursed trusted CA+Browser disaster.
- suprfsat 5y agoPreviously: https://news.ycombinator.com/item?id=20573429 https://news.ycombinator.com/item?id=20573429 Serial numbers that should have 64 bits of randomness only had 63.
- billpg 5y agoEach bit doubles the range of values. If the error was that certs were issued for 180 days instead of 90, I'd agree that's a problem.
- Arnavion 5y agoI clicked on the CAB bugzilla link thinking it was not going to become a big deal, but surprisingly it has. There's also a link to a bug from a month ago for another CA whose certs lasted one second longer, that also became a big deal.
- Quarrel 5y agoFirst fastly blows up, now my certs are lasting an extra second?! OpEng is off for the week or something? (/s)
- nodesocket 5y agoNot sure what impact this would have though.
- cpach 5y agoI also wonder this.
- elliot07 5y agoCan someone ELI5 what the actual impact of this is? Is there security concerns I'm not seeing here?
- SteveNuts 5y agoAs far as I can tell there's zero real world impact here, I think they just want to maintain a stellar track record for any reported bug that would affect the certificate issuance in any way. Basically, had it been a second, a day, a month, doesn't matter - they still treated it seriously. That sort of thing goes a long way towards building trust.
- blfr 5y agoTrust with whom? To me acting like an automaton lowers trust. The suggestion to invalidate millions of certs over a second longer validity sounds like terrible judgement.
- deknos 5y agoacting like an automaton would mean that they just recall all certificates. which they won't do. but thinking and perhaps adapting for the future, they do. and that's the right thing computers are about exactness. if the do not value that and inspect that thoroughly, i would not trust them for security.
- tingletech 5y agoThey say they can issue 200 million certificates in 24 Hours https://letsencrypt.org/2021/02/10/200m-certs-24hrs.html https://letsencrypt.org/2021/02/10/200m-certs-24hrs.html
- jaywalk 5y agoThat's great, but it doesn't mean 200 million certificates will actually be renewed in 24 hours.
- gnabgib 5y agoWouldn't the bugzilla entry [0] a better choice since the discourse community entry just points there? The preamble by Josh (ISRG Executive Director) is literally just a copy of the first paragraph from the bug, no additional data. [0]: https://bugzilla.mozilla.org/show_bug.cgi?id=1715455 https://bugzilla.mozilla.org/show_bug.cgi?id=1715455
- mjw1007 5y agoI think Bugzilla is more likely to become unhappy with large numbers of visitors than Discourse would be. This way all the readers who think "one extra second, I don't care" and close the tab won't contribute to the problem.
- Sebb767 5y agoThis might just be me, but I expected this issue to be taken far less seriously than it has been in the given communication [0]. In the linked similar issue [1], they even talk about revoking the certificates, which seems insane to me for being one second off. Is there any actual serious problem with this extra second I'm missing? [0] https://bugzilla.mozilla.org/show_bug.cgi?id=1715455 https://bugzilla.mozilla.org/show_bug.cgi?id=1715455 [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1708965 https://bugzilla.mozilla.org/show_bug.cgi?id=1708965
- joshka 5y agoI can't see an immediately obvious attack that would be available by having an extra second of certificate validity. I can think of a possible place it would be useful to know this value correctly - automated expiration monitoring and unit tests
- pdpi 5y agoCertificates have been mis-issued. No ifs, ands, or buts, that by itself means revocation has to be on the table. In practice, it's a small problem and revocation is an excessive response, but those are conclusions you reach after duly investigating the issue, not before.
- jcranmer 5y agoThis is a "no brown M&Ms" kind of violation. The direct impact of this violation is basically nil. However, it's a violation that suggests that the requirements haven't been very carefully read and that conformance to them hasn't been very carefully audited. So how can you have be assured that there aren't violations of more important rules still lurking about?
- teraflop 5y agoThere was a similar issue a while back, where an open-source system used by a number of CAs was by default configured to generate serial numbers with only 63 bits of entropy, when the CA/B forum rules require 64 bits: https://www.schneier.com/blog/archives/2019/03/cas_reissue_ove.html https://www.schneier.com/blog/archives/2019/03/cas_reissue_o... This incident was complicated by the fact that it was first noticed while investigating a particular CA (DarkMatter) that was already suspected of various shady activities. Most of the other affected organizations took the incident seriously, fixed the problem and revoked the affected certificates, as required by the letter of the rules, even though the actual risk was small. DarkMatter chose to spend a lot of time complaining and making weaselly arguments on the mailing list (like "a 64-bit number where the leading bit is always zero is still technically a 64-bit random number") which did not do much to convince anyone of their trustworthiness.
- ArchOversight 5y agoIs Ryan Sleevi in https://bugzilla.mozilla.org/show_bug.cgi?id=1715455#c1 https://bugzilla.mozilla.org/show_bug.cgi?id=1715455#c1 really suggesting that Let's Encrypt revoke 185 million certificates forcing people to get new ones because the certificates have a lifetime that is 1 second too long? 1 measly second is going to break WebPKI?!
- pdpi 5y agoCertificates were mis-issued. Even if the answer is "no, definitely not", the question of whether to revoke them has to be asked.
- tyingq 5y agoThankfully, there seems to be guidance that allows for common sense to intervene. "Mozilla recognizes that in some exceptional circumstances, revoking the affected certificates within the prescribed deadline may cause significant harm, such as...when the volume of revocations in a short period of time would result in a large cumulative impact to the web." He does have a good point that something quite similar happened a month ago[1], and should have trigged a deeper dive that would have exposed this as well. You could interpret that as LE needing some better discipline around process. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1708965 https://bugzilla.mozilla.org/show_bug.cgi?id=1708965
- SuchAnonMuchWow 5y agoYou are supposed to be able to revoke the certificate you misissue. LetsEncrypt automated the certificate creation, which resulted in very large number of certificate issued. But at the same time, it should be possible to revoke them all. If the revocation mechanism doesn't scale as much as the creation, this is a huge problem. This time, the misissued certificates probably won't cause large issues. But they are misissued nevertheless and should be revoked. If they are not revoked because it would be too complex and will break WebPKI, then what will happen when a larger issue happen that require the revocation ?
- ceejayoz 5y ago> If they are not revoked because it would be too complex and will break WebPKI, then what will happen when a larger issue happen that require the revocation ? As here, the severity of the issue should be weighed against the severity of revocation. "Half the internet needs to manually renew some certs early" is a reasonable approach if there's something risking exposure of sensitive data. It's not for "the certificate is valid for an extra second".
- gpm 5y ago> Grzegorz Prusak > Comment 5 • 6 hours ago > I do not understand the math behind the mentioned problem. > Is either the RFC or BR really specifying a DATE of 1s precision to be an INTERVAL rather than a POINT IN TIME? It specifies a granularity of points are allowed to be endpoints of a validity period, but these are still POINTS AFAICT. > It means that, for example 10:00:01 is AFTER 10:00, even though 10:00:01 is not a valid endpoint of a validity period (and it doesn't have to be, as it is a measurement result, not a validity period endpoint). > Does RFC or BR requires a browser to round down the measured current time to full seconds, before comparing them against the validity period? As a random member of the public, this seems like a reasonable interpretation to me... though the time format which expiry dates are required to be expressed in is explicitly not allowed to include fractional seconds... so maybe it is meant as an interval not a point in time.
- jcranmer 5y agoRFC 5280 says: > The universal time type, UTCTime, is a standard ASN.1 type intended for representation of dates and time. UTCTime specifies the year through the two low-order digits and time is specified to the precision of one minute or one second. [...] For the purposes of this profile, UTCTime values MUST be expressed in Greenwich Mean Time (Zulu) and MUST include seconds (i.e., times are YYMMDDHHMMSSZ), even where the number of seconds is zero. > For the purposes of this profile, GeneralizedTime values MUST be expressed in Greenwich Mean Time (Zulu) and MUST include seconds (i.e., times are YYYYMMDDHHMMSSZ), even where the number of seconds is zero. GeneralizedTime values MUST NOT include fractional seconds. In other words, the RFC specifies specifically requires the granularity of the time to be a second, and the notBefore and notAfter are worded such that is valid for all t in the range notBefore ≤ t ≤ notAfter. Computing the number of seconds this is valid for as notAfter - notBefore is a classic fencepost error.
- Grollicus 5y agoRFC 5280 further says: [1] > The validity period for a certificate is the period of time from notBefore through notAfter, inclusive. It says nothing about the comparison precision, just the storage format. It says CAs MUST encode validity dates as UTCTime / GeneralTime. It does not say this encoding must be used for comparison. So I don't think it's actually defined if 2020-01-02 03:04:05.01Z is actually <= 2020-01-02 03:04:05Z. That would mean this is not a 1 second-mistake but instead an infinitesimal mistake. [1] https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.5 https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2....
- raverbashing 5y agoSounds like one of those tricky technicalities For fun, I thought of a bad edge case: you create a certificate exactly 90 days before the end of the year (having your notAfter date of 2022-01-01-00:00:00Z) BUT you get a leap second on this year. Then it will last for 90 days + 1 second
- justin_oaks 5y agoWhich makes me wonder if other certificate authorities handle this scenario.
- thehappypm 5y agoThey actually mentioned that they use 90 days as defined by 2160 hours, so that would be a non-issue.
- varajelle 5y agoIt depends if the certificate itself contains an absolute timestamp for the expiration time, or a validity period.
- wiml 5y agoCertificates contain a pair of absolute times in a Gregorian representation. The ASN.1 time format does support leap seconds (in the usual way, e.g., a time of 23:59:60) but I don't know if X.500 or PKIX restricts that.
- chizhik-pyzhik 5y agoPeople who work with x.509 sure do make a lot of work for themselves
- wahern 5y agoIgnorance is bliss. Most projects and even many formal standards don't even think to consider such questions, let alone try to answer them. X.509 builds on ASN.1, which incorporates ISO 8601, which in turn incorporates IEC 60050-111. Unfortunately, as elucidated in this thread, the relevant portion of X.509 is still underspecified--it uses the term "inclusive" without resolving ambiguity related to intervals and granularity, which it could have done by using formulas provided by ASN.1, ISO 8601, or IEC 60060-111.
- jaas 5y agoHead of Let's Encrypt here. The question of whether or not revocation should happen has to be asked whenever a certificate compliance issue is being discussed, regardless of how serious the issue is. That is a normal part of the process of evaluating an incident thoroughly. We do not plan to revoke any certificates as a result of this issue. An aside - I love how informed many of the commenters here are, thanks to you all for helping to explain what's happening!
- sigzero 5y agoThat you for the clarification and the letsencrypt service.
- u8mybrownies 5y agoThank you for let’s encrypt. Very grateful for your role in society.
- zaxomi 5y agoHi! Is your time servers synchronized to GMT (mean solar time at the Royal Observatory in Greenwich) or UTC (Coordinated Universal Time)? The RFC requires GMT, but most time servers synchronizes to UTC. It could be a difference of up to 0.9 seconds between GMT and UTC.
- wahern 5y ago> It could be a difference of up to 0.9 seconds between GMT and UTC. Not in this context. In the context of ASN.1, NTP, POSIX, etc, GMT is a timezone equivalent to UTC+00:00. UT1 and DUT1 don't figure into it.
- fanf2 5y agoGMT was abolished at the end of 1971, when the Greenwich observatory started distributing UTC as its official timescale. At that time the observatory was based in Herstmonceux, so GMT was calculated from a model rather than being based on observations made from the transit instruments in Greenwich.
- akersten 5y ago
- zaxomi 5y agoIn my opinion the RFC should be changed from "notBefore ≤ t ≤ notAfter" to "notBefore ≤ t < notAfter".
- mulmen 5y agoShouldn't that be: "notBefore ≤ t ≤ notAfter" to "notBefore ≤ t < until" ?
- zaxomi 5y agoIt is the name of the fields in RFC 5280: Validity ::= SEQUENCE { notBefore Time, notAfter Time }
- mulmen 5y agoExactly. Making "notBefore" inclusive but "notAfter" exclusive is inconsistent and IMHO confusing. If the end time should be exclusive then it should be called "until" instead of "notAfter".
- account42 5y agoIt's only confusing because you interpret notBefore and notAfter as one-second time intervals when they really should be instants at the start of that second.
- zaxomi 5y agoIsn't GMT a time zone (mean solar time at the Royal Observatory in Greenwich) that can differ from UTC up to 0.9 seconds? So since the RFC specifies GMT it is already up to 0.9 seconds wrong?
- zaxomi 5y agoTom Scott has made a video: https://www.youtube.com/watch?v=yRz-Dl60Lfc https://www.youtube.com/watch?v=yRz-Dl60Lfc According to Encyclopaedia Britannica: > Greenwich Mean Time (GMT), the name for mean solar time of the longitude (0°) of the Royal Greenwich Observatory in England > ...in 1928 the International Astronomical Union changed the designation of the standard time of the Greenwich meridian to Universal Time. Universal Time remains in general use in a modified form as Coordinated Universal Time (UTC), which serves to accommodate the timekeeping differences that arise between atomic time (derived from atomic clocks) and solar time.
- davidhyde 5y agoNot related to this issue but I think that the expiration of LetsEncrypt's old root certificate "DST Root CA X3" on 21 Sep 2021 may cause quite a few problems. Old embedded systems may not have a way to update their trust anchors (like browsers do). Obviously not LetsEncrypt's fault but worth being aware of.
- pkamb 5y ago#ShrugOps
- Ensorceled 5y agoThis issue is NOT critical to my operations and is, in fact, irrelevant to them. I can NOT speak to any other users of letsencrypt. So I'm happy this was raised because maybe there is another, seemingly harmless incident that MAY impact my operations that will be brought up in the future.
- throwawayboise 5y agoA classic fencepost error. If you use SQL you should be aware of this if you use the BETWEEN operator; it will select values inclusive of the stated endpoints, which may or may not be your intuitive understanding of what "between" means.
- shakezula 5y agoI’ve never before heard this described as a fencepost problem but its perfect.
- deleted 5y ago[deleted]
- shreddit 5y ago> We will further improve our codebase […].This work will be completed by 2021-09-09 (inclusive). I chuckled. https://bugzilla.mozilla.org/show_bug.cgi?id=1715455 https://bugzilla.mozilla.org/show_bug.cgi?id=1715455
- computer23 5y agoThere are two hard things in computer science: cache invalidation, naming things , and off-by-one errors.
- failwhaleshark 5y agooff-by-1 error... ooops It will all be fixed in 90 days, so revoking is pointless for all but the edgiest of edge-cases.
- cascala 5y ago1 second Who cares?
- account42 5y agoThis is an outrage! I demand my money back!