7 ms·
Quick reminder from your friendly local SRE: never ever issue certificates that expire on weekends. Make certs expire in the middle of the afternoon on a busine
by jeffbee 6y ago
Quick reminder from your friendly local SRE: never ever issue certificates that expire on weekends. Make certs expire in the middle of the afternoon on a business day wherever your operators live and work. The cert in question expires at May 30 10:48:38 2020 GMT, which smells suspiciously like a fixed time after the cert was generated, rather than at a well-chosen point in time.
- encoderer 6y agoGreat tip. Did you notice that cert in this case was issued 20 years ago? It’s crazy to me that it was still being used to sign certs as recently as last week (according to twitter)
- jeffbee 6y agoOf course, but that doesn't really excuse them. My first experience with middle-of-Sunday-night SSL certificate expiration was in December 1998, and it was already a well-known doctrine by then. I'd expect a commercial certificate authority to have these kinds of things squared away.
- user5994461 6y agoMy experience with commercial CA is that they set the expiry exactly 1 year from creation. Doesn't matter if it's a week end or a holiday.
- sjtindell 6y agoGenerally that’s for Server certs, Roots and Intermediates will be multiple years from what I’ve seen. Roots in particular 10+ years.
- abiogenesis 6y ago1 Year is a relatively recent thing. Previously you were able to buy server certs with a 5 year expiration.
- jis 6y agoIt's actually worse. The new root (good I believe until 2038) uses the same key as the now expired certificate. It has to or it would not be possible to validate the certificates that were issued. And this new one is a root certificate installed in browsers! What "should" happen is that no certificate should be issued with an expiration date later than the issuing certificate. Then as the issuing certificate gets closer to expiration, a new one, with a new key pair, should be created and this new certificate should sign subordinate certificates.
- jis 6y agoSorry to reply to my own comment. But I want to clarify. Two certificates (at least) expired. The root named "AddTrust External CA Root" and a subordinate certificate with a subject of "USERTrust RSA Certification Authority." Both expired around the same time. The "USERTrust RSA Certification Authority" certificate signed yet another layer of intermediate certificates. The "USERTrust RSA Certification Authority" certificate was promoted to a self-signed certificate, now in the browser trust stores, using the same key pair as the original certificate that was signed by "AddTrust External CA Root." It has an expiration of 2038 (although that concept is a bit vague in a root certificate).
- josephcsible 6y agoThere's actually a third certificate for "USERTrust RSA Certification Authority", also using the same key pair, signed by a different root called "AAA Certificate Services". It looks like the intended replacement for the expiring one is this one, rather than the one where it's the root itself.
- dylz 6y agoIt is explicitly not a replacement, but some kind of legacy fallback that they don't want you to use, but exists for enterprise customers that absolutely can't get trust.
- josephcsible 6y ago
- imron 6y ago> It’s crazy to me that it was still being used to sign certs as recently as last week (according to twitter) It's likely because it was issued 20 years ago. People have been using it for 20 years and no-one realized it was about to stop working.
- cmbuckley 6y agoCertificates issued with this CA will have been cross-signed by the newer root certificate, but our CA (Sectigo) was sending the old chain in issuing emails as late as April this year, despite the cross-signed root being available for a long time.
- btgeekboy 6y agoIf you get to the point where the exact expiration date on the certificate matters, you've already lost the game.
- DavidSJ 6y agoDefense in depth.
- techslave 6y agoit’s more like blue m&m’s than an actual requirement
- alasdair_ 6y ago>it’s more like blue m&m’s than an actual requirement Did you mean Van Halen's famous "WARNING: ABSOLUTELY NO BROWN M&Ms" clause? https://www.snopes.com/fact-check/brown-out/ https://www.snopes.com/fact-check/brown-out/
- dialamac 6y agoProbably. It’s an honest mistake. For Gen Xers and early millennials blue m&ms are memorable for being added in the 90s with much ludicrous publicity.
- colechristensen 6y agoEngineering for failure is important, you should always set yourself up so that you have several lines of defence which can fail. Some lines of defence to make failing "impossible" others to make a fail softer, even when you think failing is impossible.
- lmilcin 6y agoAll my applications use a component that watches certs configured (everything in cert and trust store) and returns warning in telemetry from the application if any of the certificates is less than a week from expiration. This is checked periodically while the application runs. This not only makes sure we don't miss expiration but also ensures we don't forget to configure any of the application. We had a situation when the cert was replaced but the file was placed in incorrect path and was not actually used by the app. Having the app report on what is actually being in use is the best way to prevent this from ever happening.
- BurningFrog 6y agoThere is just no substitute for Reality!
- deleted 6y ago[deleted]
- kl4m 6y agoGood old "cert replaced but apache/nginx failed to reload" has bitten me more than once...
- Sebb767 6y agoMe too! Especially with the short expiration times of LetsEncrypt. But I really don't want to put `nginx -s reload` in the Cron, in case I'm tinkering with the configs and they're suddenly live (which only really happens at staging or at home of course, but still).
- beamatronic 6y agoAlso make a Calendar placeholder (like a fake meeting), invite a lot of folks or a distribution list, and turn on an alert for 24 hours ahead.
- folmar 6y agoFor long-lived certificates they will outlive your calendar tech. The bit rot leave maybe the events fine, but anything fancier like notifications will fad away. Source: BTDT 3 times in 7 years, and it was all with "Big Enterprise" grade products.
- kjaftaedi 6y ago"Middle of the afternoon" .. for who?
- hombre_fatal 6y agoIf you're going to ask a question about someone's comment, at least finish reading the sentence to make sure it's not immediately answered. Don't be part of the death of internet discourse.
- kjaftaedi 6y agoI read the comment several times. I am just of the opinion that "make it expire in the afternoon" doesn't apply to root certificates that are used across the entire world (i.e. - the topic of discussion)
- adrianmonk 6y ago> rather than at a well-chosen point in time So, you're saying that "I'm not going to be working here anymore by then... hahahaha" isn't well-chosen?
- jldugger 6y ago> Make certs expire in the middle of the afternoon on a business day wherever your operators live and work. If I can predict that 20 years into the future I wouldn't be in the SRE business.
- jeffbee 6y agoI think it was foreseeable even in dark days of the year 2000 that this certificate expired after business hours globally. But your statement is really pointing out that nobody should be making long-lived certificates.
- solarengineer 6y agoIf you were to issue certs for short durations, and also maintain a calendar of cert expiry, those certs could be renewed in a timely manner. In other scenarios where one would want to issue fresh certificates (receiving Ops control from other orgs, credentials refresh for what ever reason), one can still do so without waiting for the current certificates to expire.