24 ms·

I’ll still keep using Firefox since I recognize the importance of browser diversity and the hazards of a Chrome monoculture (that and vertical tabs), but, yikes

I’ll still keep using Firefox since I recognize the importance of browser diversity and the hazards of a Chrome monoculture (that and vertical tabs), but, yikes. Still, this type of oversight seems all too common even in large companies. I remember several cases from Fortune 500 companies in the past few years alone. What would be a good way to automate checking for them? Has anyone developed a tool designed specifically to avoid certificate expiry disasters?

wbl 7y ago
We scan our codebase for anything that looks like a cert and send emails when it gets close. Might not have helped here if it was an intermediate owned by a CA. There but for the grace of God go I.
- adtac 7y ago
  Why do you have certificates in your code to begin with?
  - justinclift 7y ago
    If you have your own CA for whatever reason, it's common to distribute the root and intermediate certs with your code so things can resolve. You don't ship the signing keys with the certs, as that would be bad. ;)
    - justinclift 7y ago
      s/resolve/validate/
- lvh 7y ago
  If you want to get rid of those and they're public certs: odds are they're in Certificate Transparency logs and you can monitor them from there.
  - tialaramex 7y ago
    Monitoring CT lets you verify that somebody renewed the certificate, but it doesn't verify they actually installed the replacement correctly. My employer (Kynd.io) currently monitors public web sites for customers so we can flag e.g. "Hey this site cert expires in a week! If it's dead probably just switch it off, otherwise renew the certificate" and we're in the process of integrating CT but mostly so we can say "You already have a newer cert but need to go install it" in our How To Fix instructions.
crazygringo 7y ago
That's a great question. I've never seen a bulletproof solution for organizational tasks that need to be done yearly. If someone's in charge... and both they and their manager happen to leave in the same year... and whatever system they had in place to remember (probably their personal calendars) is gone... and the manager's manager has 1,000 other things to remember... ...how does an organization ensure the task still gets done?
- emgee_1 7y ago
  Just one : Emacs orgmode
- Complexicate 7y ago
  "...how does an organization ensure the task still gets done?" With something almost stupidly simple and low-tech: checklists. (I'm reading "The Checklist Manifesto" right now, and the points it makes seem to fit perfectly with everything you mention.)
  - marcosdumay 7y ago
    An year is enough time for everybody that knows about the checklist to leave.
    - andrewflnr 7y ago
      Put "make sure someone else knows all this person's checklists" on the employee exit checklist.
      - craftinator 7y ago
        Put the checklist on the home page of the company website!
    - BuckRogers 7y ago
      We resolved this issue at my last company with sufficiently large mailing groups for cert renewal reminders. Once you get to 12 people on a mailing list, with new employees being added all the time, it's hard to miss. Usually a manager on that list is pinging people about it. There is the chance of the tragedy of the commons occurring, but I never saw it. Once you do this, the only checklist that matters are procedural checklists to add a new client or new cert to the renewal notification list. When you use a standard group email for all cert purchases, that one becomes tough to miss. In my 7 years of being involved, we never missed a cert renewal with this process for ~300 client sites with multiple or wildcard certs.
deleted 7y ago
[deleted]
ShinTakuya 7y ago
It's not that complicated, just add scheduled health checks to the same system you use for checking if the website and such is up. If the expiry date isn't updated within a week of expiry start paging engineers. I'm willing to bet Mozilla already does something like this but an engineer didn't set it up correctly for this certificate.
_wmd 7y ago
Let's not forget multiple mobile networks across Europe went down on the same day last year because Ericsson(?) let a cert expire on some internal management system that had not been updated. SSL cert renewal is one of the great unsolved problems in computer science edit: not Europe, just UK and Japan apparently: https://www.zdnet.com/article/ericsson-expired-certificate-caused-o2-and-softbank-outages/ https://www.zdnet.com/article/ericsson-expired-certificate-c...
- MrEldritch 7y ago
  There was also an issue last year where every single Oculus Rift was essentially bricked because they forgot to renew a cert (apparently, what with the chaos of the Rift launch and the Facebook acquisition between the cert issuance and expiration, they just kind of ... lost track). It took like two days before there was any kind of fix available, and they couldn't even roll it out automatically because the expiration had also disabled the auto-updating.
- AmericanChopper 7y ago
  >SSL cert renewal is one of the great unsolved problems in computer science Certificate expiry really only exists to make money for CAs. It doesn’t solve any security problem that CRLs don’t already solve (and solve better). There’s lots of unsolved problems relating to ‘how do you make a reliable PKI’, but cert expiry is really just an unrelated business requirement for CAs.
  - kevingadd 7y ago
    I'd argue it's a blunt hammer extra layer of defense, where if a certificate gets compromised and the owner never finds out at least it eventually stops working. This kind of compromise is pretty common.
  - jefftk 7y ago
    If it really was only to make money for CAs we'd see LetsEncrypt offering very long lifetime certs. But: * Very short lifetimes get people to automate, preventing problems where one cert lasts long enough to lose the institutional knowledge around it. * CRLs don't work. For performance you don't want to check for a revocation in serial with the request, and you don't want to block all browsing if the revocation list server is down. Revoking a cert will cover some users, but lots will still get "https://" https://" and no warnings.
kam 7y ago
ACME / Let's Encrypt go in the direction of making expiry happen so often that renewal gets automated, rather than a being a rare manual process that can be forgotten about. Not sure that's viable for a signing certificate like this, but that's the way to solve it for the web PKI.
- SomeHacker44 7y ago
  This is just abusive to the vast majority of users who do not care but still want to use SSL for their servers, frankly. I should be allowed to choose a near unlimited lifetime for my server's certificate if I don't care about the risks that may present.
  - httpsterio 7y ago
    As the service provider, you shouldn't get to decide. I think it's the users who can decide how long lived certs they're willing to trust.
    - Godel_unicode 7y ago
      That's cool and all, but what percentage of users do you think even know certs expire? I'd put the over/under at 1%.
  - lugg 7y ago
    Security tends towards the lowest common denominator. I'd rather you just figured out how to run a cron job. The problem comes if your keys ever get compromised or cracked all your historical traffic becomes vulnerable instead of just the most recent window.
    - gboudrias 7y ago
      Yeah "just" a cron job except the implementation changes several times a year. Somehow this automated process was more time-consuming than the previous, manual one.
minetest2048 7y ago
Talking about vertical tabs, I was in the middle of studying for an upcoming exam, then when I alt-tabbed back into Firefox, all of my tabs are missing with that unsupported addon error. Fortunately refreshing Firefox gave me back normal tabs, at a cost of uninstalling all of my addons. The problem is that Tree Style Tabs relies on userchrome.css edit to hide the tab bar, and when TST is forcibly removed there is no way to access the tabs, because that edited userchrome.css is still there. This is very disruptive. At least with the pre WebExtension addon TST itself hides the tab bar, so if TST is removed then the original tab bar comes back on automatically
- frosted-flakes 7y ago
  I have it set up so that the tab bar is only displayed if the menu bar is visible, and I can use the Alt key to toggle them together. https://github.com/eoger/tabcenter-redux/wiki/Custom-CSS-Tweaks#show-tab-strip-if-menu-bar-is-visible https://github.com/eoger/tabcenter-redux/wiki/Custom-CSS-Twe... #toolbar-menubar[inactive="true"] + #TabsToolbar { visibility: collapse !important; }
revvx 7y ago
> Still, this type of oversight seems all too common even in large companies. (...) Has anyone developed a tool designed specifically to avoid certificate expiry disasters? LetsEncrypt renewal is supposed to be automated. [1] I know of a company that hosted blogs for thousands of customers. They used LetsEncrypt, but the CTO considered automatic renewals a possible security risk, so they did it manually. Problem is, the expiration happened in a weekend and they "forgot" to update the certificates before that. Suffice to say that the next Monday wasn't pleasant. They automated after that. [1] https://letsencrypt.org/about/ https://letsencrypt.org/about/
- mc32 7y ago
  So did they conclude it wasn’t a security concern or did they conclude the security risk was worth the uptime?
  - mehrdadn 7y ago
    I'm curious as well. My intuition would be that it's not a concern, since servers already keep their private keys stored locally in order to be able to communicate with clients anyway? Being able to update them doesn't really seem to make things any different. But I feel like I could be missing something/not have thought through it properly. (I imagine security implications can get more complicated if a different server decrypts traffic vs. processes it, etc.)
    - revvx 7y ago
      The "manual" process used previously by the company already involved some form of automation, so it was more about trusting CertBot not to do anything horrendous. But now that you mention it, I wonder what's the opinion of security experts like tptacek on cert renewal automation.
      - inflatableDodo 7y ago
        We could attempt a summoning. Quick, make a wildly inaccurate claim about the correct way to implement an encryption library.
js2 7y ago
You can find lots of programs like this one to monitor certs: https://pypi.org/project/check-tls-certs/ https://pypi.org/project/check-tls-certs/ I run one daily from cron and have it email me a report with the days to expiration for the certs I’m responsible for, even for certs that auto renew. I don’t filter the email. Daily is not too frequent for it to go to my inbox, but frequent enough that I’ll notice if it doesn’t mail me. YMMV.
- dev_dull 7y ago
  Discovery of all the certs is what I think is the harder problem.
  - human20190310 7y ago
    I agree. What can be done to prevent developers from adding a certificate dependency without monitoring during the move-fast-and-break-things days of early development, which then sits for X years as developers come and go, and nobody notices until it fails?
    - technion 7y ago
      Whilst I'll say "disclaimer, this is my project", monitoring Certificate Transparency with CT Advisor has helped me find out about certificates marketing people deployed and expected me to maintain without my knowledge. [0] https://ctadvisor.lolware.net/ https://ctadvisor.lolware.net/
    - rhizome 7y ago
      >What can be done to prevent developers from adding a certificate dependency Discipline? Experience? PIP?
    - lvh 7y ago
      Certificate Transparency works pretty darn well for most usecases, we (Latacora) have found while trying to solve exactly this problem (or at least the figure out which certs exist that aren't being regularly re-issued part) :-)
otakucode 7y ago
>Has anyone developed a tool designed specifically to avoid certificate expiry disasters? Is anything more than a calendar reminder on the phone of someone important enough to shake the Earth and get it fixed For. Certain. needed? Like, say, the CEO, CTO, and CFO should at a minimum get a notification so they can ask if the refresh was done when necessary?
- tialaramex 7y ago
  Admin people. Often the most senior ones get the title "Personal Assistant (to senior person job title)" but not always. They're lead bureaucrats, and tracking things that need to be done and ensuring they get done, either by doing them themselves or assigning them to reliable underlings is the purpose of their role. Corporations are often not very good at putting the right people in these roles but good ones are invaluable. Since the Marvel Universe is everywhere, Pepper Potts is the archetype in that setting to give you an idea of why you'd need people like this. Tony Stark would be "too busy" to renew the certificates, but Pepper would make sure it gets done.
cesarb 7y ago
> I’ll still keep using Firefox since I recognize the importance of browser diversity Also, Chrome is not immune to "crashes for everyone at the same time" bugs. Like that time when the start of daylight saving time made it crash for a full day (a quick search tells me it probably was https://bugs.chromium.org/p/chromium/issues/detail?id=287821 https://bugs.chromium.org/p/chromium/issues/detail?id=287821).
- username223 7y ago
  > "crashes for everyone at the same time" bugs What else would you expect for auto-updating software that relies on the internet to work? It's a monoculture attached to a firehose of disease. This is exactly the same as "pushing out a security fix to all users," except it apparently wasn't intentional. You can't have one without the other.
  - craftinator 7y ago
    I love "firehose of disease", and will steal it. And I agree that bugs are bugs; every time you add a new capability, you add all the possible bugs that can occur with that capability.
- tssva 7y ago
  That bug seems to have affected only users on Android versions earlier than 4.3 and in Brazil or Chile.
luckylion 7y ago
Maybe we need more browser diversity than just two different teams with two different systems. Both are sitting very close to each other geographically, and both are produced in the same culture (as in silicon valley), so it would seem likely that, while they compete with each other, they will apply very similar answers to problems they face.
bartread 7y ago
> Still, this type of oversight seems all too common even in large companies. The npm self-signed certificate fiasco of early 2014 springs immediately to mind.
AmericanChopper 7y ago
There’s lots of monitoring services out there that do it. A long time ago I worked at place that used a service called site24x7 for cert and API monitoring. That was before Pingdom kinda got better than most API monitoring services, but I don’t know if they monitor cert expiry. Taking a look around, you’ll find lots of service providers, or tools you could use. But the main issue is all they do is tell a human being to do something, which they can still fail to do. Which is why automating cert rotation (with things like let’s encrypt or ACM) is arguably a better solution than monitoring it.
kitotik 7y ago
Systems designed around long TTLs make this problem worse. I love the default of 90 days for Let’s Encrypt. It forces some good discipline and hygiene. Wish there was a better solution for short lived CAs
mattbillenstein 7y ago
I built a tool for checking ssl certs some time ago: https://ismycertexpired.com https://ismycertexpired.com but I'm not checking intermediate certs...
rixed 7y ago
> Has anyone developed a tool designed specifically to avoid certificate expiry disasters? Not perfect, but I've added a TLS certificate extraction tool into a DPI that displays all visible certificates ordered by expiry date. One could then mirror all one's site traffic to it and let it run in the background. Coupled with some alerting tool it would catch most of those cases I guess. I could polish the tool a bit more if there is some interest, but anyone could do it as well. See https://github.com/rixed/junkie https://github.com/rixed/junkie and more specifically the plugin called 'sslogram'.
nothrabannosir 7y ago
Realistically: reduce your own cert renewal window to weekly, if not daily. This forces you to have a good renewal system in place and alerts you to failures long before actual expiration. Quixotically: make cert failure a randomised number, linearly related to how long ago the cert expired. This slowly introduces more and more failures, over a certain “grace period”, which makes the problem less of an extinction level event. It’s not a solution but it definitely would help.

wbl 7y ago

We scan our codebase for anything that looks like a cert and send emails when it gets close. Might not have helped here if it was an intermediate owned by a CA. There but for the grace of God go I.

adtac 7y ago
Why do you have certificates in your code to begin with?
- justinclift 7y ago
  If you have your own CA for whatever reason, it's common to distribute the root and intermediate certs with your code so things can resolve. You don't ship the signing keys with the certs, as that would be bad. ;)
  - justinclift 7y ago
    s/resolve/validate/
lvh 7y ago
If you want to get rid of those and they're public certs: odds are they're in Certificate Transparency logs and you can monitor them from there.
- tialaramex 7y ago
  Monitoring CT lets you verify that somebody renewed the certificate, but it doesn't verify they actually installed the replacement correctly. My employer (Kynd.io) currently monitors public web sites for customers so we can flag e.g. "Hey this site cert expires in a week! If it's dead probably just switch it off, otherwise renew the certificate" and we're in the process of integrating CT but mostly so we can say "You already have a newer cert but need to go install it" in our How To Fix instructions.

crazygringo 7y ago

That's a great question. I've never seen a bulletproof solution for organizational tasks that need to be done yearly. If someone's in charge... and both they and their manager happen to leave in the same year... and whatever system they had in place to remember (probably their personal calendars) is gone... and the manager's manager has 1,000 other things to remember... ...how does an organization ensure the task still gets done?

emgee_1 7y ago
Just one : Emacs orgmode
Complexicate 7y ago
"...how does an organization ensure the task still gets done?" With something almost stupidly simple and low-tech: checklists. (I'm reading "The Checklist Manifesto" right now, and the points it makes seem to fit perfectly with everything you mention.)
- marcosdumay 7y ago
  An year is enough time for everybody that knows about the checklist to leave.
  - andrewflnr 7y ago
    Put "make sure someone else knows all this person's checklists" on the employee exit checklist.
    - craftinator 7y ago
      Put the checklist on the home page of the company website!
  - BuckRogers 7y ago
    We resolved this issue at my last company with sufficiently large mailing groups for cert renewal reminders. Once you get to 12 people on a mailing list, with new employees being added all the time, it's hard to miss. Usually a manager on that list is pinging people about it. There is the chance of the tragedy of the commons occurring, but I never saw it. Once you do this, the only checklist that matters are procedural checklists to add a new client or new cert to the renewal notification list. When you use a standard group email for all cert purchases, that one becomes tough to miss. In my 7 years of being involved, we never missed a cert renewal with this process for ~300 client sites with multiple or wildcard certs.

deleted 7y ago

[deleted]

ShinTakuya 7y ago

It's not that complicated, just add scheduled health checks to the same system you use for checking if the website and such is up. If the expiry date isn't updated within a week of expiry start paging engineers. I'm willing to bet Mozilla already does something like this but an engineer didn't set it up correctly for this certificate.

_wmd 7y ago

Let's not forget multiple mobile networks across Europe went down on the same day last year because Ericsson(?) let a cert expire on some internal management system that had not been updated. SSL cert renewal is one of the great unsolved problems in computer science edit: not Europe, just UK and Japan apparently: https://www.zdnet.com/article/ericsson-expired-certificate-caused-o2-and-softbank-outages/ https://www.zdnet.com/article/ericsson-expired-certificate-c...

MrEldritch 7y ago
There was also an issue last year where every single Oculus Rift was essentially bricked because they forgot to renew a cert (apparently, what with the chaos of the Rift launch and the Facebook acquisition between the cert issuance and expiration, they just kind of ... lost track). It took like two days before there was any kind of fix available, and they couldn't even roll it out automatically because the expiration had also disabled the auto-updating.
AmericanChopper 7y ago
>SSL cert renewal is one of the great unsolved problems in computer science Certificate expiry really only exists to make money for CAs. It doesn’t solve any security problem that CRLs don’t already solve (and solve better). There’s lots of unsolved problems relating to ‘how do you make a reliable PKI’, but cert expiry is really just an unrelated business requirement for CAs.
- kevingadd 7y ago
  I'd argue it's a blunt hammer extra layer of defense, where if a certificate gets compromised and the owner never finds out at least it eventually stops working. This kind of compromise is pretty common.
- jefftk 7y ago
  If it really was only to make money for CAs we'd see LetsEncrypt offering very long lifetime certs. But: * Very short lifetimes get people to automate, preventing problems where one cert lasts long enough to lose the institutional knowledge around it. * CRLs don't work. For performance you don't want to check for a revocation in serial with the request, and you don't want to block all browsing if the revocation list server is down. Revoking a cert will cover some users, but lots will still get "https://" https://" and no warnings.

kam 7y ago

ACME / Let's Encrypt go in the direction of making expiry happen so often that renewal gets automated, rather than a being a rare manual process that can be forgotten about. Not sure that's viable for a signing certificate like this, but that's the way to solve it for the web PKI.

SomeHacker44 7y ago
This is just abusive to the vast majority of users who do not care but still want to use SSL for their servers, frankly. I should be allowed to choose a near unlimited lifetime for my server's certificate if I don't care about the risks that may present.
- httpsterio 7y ago
  As the service provider, you shouldn't get to decide. I think it's the users who can decide how long lived certs they're willing to trust.
  - Godel_unicode 7y ago
    That's cool and all, but what percentage of users do you think even know certs expire? I'd put the over/under at 1%.
- lugg 7y ago
  Security tends towards the lowest common denominator. I'd rather you just figured out how to run a cron job. The problem comes if your keys ever get compromised or cracked all your historical traffic becomes vulnerable instead of just the most recent window.
  - gboudrias 7y ago
    Yeah "just" a cron job except the implementation changes several times a year. Somehow this automated process was more time-consuming than the previous, manual one.

minetest2048 7y ago

Talking about vertical tabs, I was in the middle of studying for an upcoming exam, then when I alt-tabbed back into Firefox, all of my tabs are missing with that unsupported addon error. Fortunately refreshing Firefox gave me back normal tabs, at a cost of uninstalling all of my addons. The problem is that Tree Style Tabs relies on userchrome.css edit to hide the tab bar, and when TST is forcibly removed there is no way to access the tabs, because that edited userchrome.css is still there. This is very disruptive. At least with the pre WebExtension addon TST itself hides the tab bar, so if TST is removed then the original tab bar comes back on automatically

frosted-flakes 7y ago
I have it set up so that the tab bar is only displayed if the menu bar is visible, and I can use the Alt key to toggle them together. https://github.com/eoger/tabcenter-redux/wiki/Custom-CSS-Tweaks#show-tab-strip-if-menu-bar-is-visible https://github.com/eoger/tabcenter-redux/wiki/Custom-CSS-Twe... #toolbar-menubar[inactive="true"] + #TabsToolbar { visibility: collapse !important; }

revvx 7y ago

> Still, this type of oversight seems all too common even in large companies. (...) Has anyone developed a tool designed specifically to avoid certificate expiry disasters? LetsEncrypt renewal is supposed to be automated. [1] I know of a company that hosted blogs for thousands of customers. They used LetsEncrypt, but the CTO considered automatic renewals a possible security risk, so they did it manually. Problem is, the expiration happened in a weekend and they "forgot" to update the certificates before that. Suffice to say that the next Monday wasn't pleasant. They automated after that. [1] https://letsencrypt.org/about/ https://letsencrypt.org/about/

mc32 7y ago
So did they conclude it wasn’t a security concern or did they conclude the security risk was worth the uptime?
- mehrdadn 7y ago
  I'm curious as well. My intuition would be that it's not a concern, since servers already keep their private keys stored locally in order to be able to communicate with clients anyway? Being able to update them doesn't really seem to make things any different. But I feel like I could be missing something/not have thought through it properly. (I imagine security implications can get more complicated if a different server decrypts traffic vs. processes it, etc.)
  - revvx 7y ago
    The "manual" process used previously by the company already involved some form of automation, so it was more about trusting CertBot not to do anything horrendous. But now that you mention it, I wonder what's the opinion of security experts like tptacek on cert renewal automation.
    - inflatableDodo 7y ago
      We could attempt a summoning. Quick, make a wildly inaccurate claim about the correct way to implement an encryption library.

js2 7y ago

You can find lots of programs like this one to monitor certs: https://pypi.org/project/check-tls-certs/ https://pypi.org/project/check-tls-certs/ I run one daily from cron and have it email me a report with the days to expiration for the certs I’m responsible for, even for certs that auto renew. I don’t filter the email. Daily is not too frequent for it to go to my inbox, but frequent enough that I’ll notice if it doesn’t mail me. YMMV.

dev_dull 7y ago
Discovery of all the certs is what I think is the harder problem.
- human20190310 7y ago
  I agree. What can be done to prevent developers from adding a certificate dependency without monitoring during the move-fast-and-break-things days of early development, which then sits for X years as developers come and go, and nobody notices until it fails?
  - technion 7y ago
    Whilst I'll say "disclaimer, this is my project", monitoring Certificate Transparency with CT Advisor has helped me find out about certificates marketing people deployed and expected me to maintain without my knowledge. [0] https://ctadvisor.lolware.net/ https://ctadvisor.lolware.net/
  - rhizome 7y ago
    >What can be done to prevent developers from adding a certificate dependency Discipline? Experience? PIP?
  - lvh 7y ago
    Certificate Transparency works pretty darn well for most usecases, we (Latacora) have found while trying to solve exactly this problem (or at least the figure out which certs exist that aren't being regularly re-issued part) :-)

otakucode 7y ago

>Has anyone developed a tool designed specifically to avoid certificate expiry disasters? Is anything more than a calendar reminder on the phone of someone important enough to shake the Earth and get it fixed For. Certain. needed? Like, say, the CEO, CTO, and CFO should at a minimum get a notification so they can ask if the refresh was done when necessary?

tialaramex 7y ago
Admin people. Often the most senior ones get the title "Personal Assistant (to senior person job title)" but not always. They're lead bureaucrats, and tracking things that need to be done and ensuring they get done, either by doing them themselves or assigning them to reliable underlings is the purpose of their role. Corporations are often not very good at putting the right people in these roles but good ones are invaluable. Since the Marvel Universe is everywhere, Pepper Potts is the archetype in that setting to give you an idea of why you'd need people like this. Tony Stark would be "too busy" to renew the certificates, but Pepper would make sure it gets done.

cesarb 7y ago

> I’ll still keep using Firefox since I recognize the importance of browser diversity Also, Chrome is not immune to "crashes for everyone at the same time" bugs. Like that time when the start of daylight saving time made it crash for a full day (a quick search tells me it probably was https://bugs.chromium.org/p/chromium/issues/detail?id=287821 https://bugs.chromium.org/p/chromium/issues/detail?id=287821).

username223 7y ago
> "crashes for everyone at the same time" bugs What else would you expect for auto-updating software that relies on the internet to work? It's a monoculture attached to a firehose of disease. This is exactly the same as "pushing out a security fix to all users," except it apparently wasn't intentional. You can't have one without the other.
- craftinator 7y ago
  I love "firehose of disease", and will steal it. And I agree that bugs are bugs; every time you add a new capability, you add all the possible bugs that can occur with that capability.
tssva 7y ago
That bug seems to have affected only users on Android versions earlier than 4.3 and in Brazil or Chile.

luckylion 7y ago

Maybe we need more browser diversity than just two different teams with two different systems. Both are sitting very close to each other geographically, and both are produced in the same culture (as in silicon valley), so it would seem likely that, while they compete with each other, they will apply very similar answers to problems they face.

bartread 7y ago

> Still, this type of oversight seems all too common even in large companies. The npm self-signed certificate fiasco of early 2014 springs immediately to mind.

AmericanChopper 7y ago

There’s lots of monitoring services out there that do it. A long time ago I worked at place that used a service called site24x7 for cert and API monitoring. That was before Pingdom kinda got better than most API monitoring services, but I don’t know if they monitor cert expiry. Taking a look around, you’ll find lots of service providers, or tools you could use. But the main issue is all they do is tell a human being to do something, which they can still fail to do. Which is why automating cert rotation (with things like let’s encrypt or ACM) is arguably a better solution than monitoring it.

kitotik 7y ago

Systems designed around long TTLs make this problem worse. I love the default of 90 days for Let’s Encrypt. It forces some good discipline and hygiene. Wish there was a better solution for short lived CAs

mattbillenstein 7y ago

I built a tool for checking ssl certs some time ago: https://ismycertexpired.com https://ismycertexpired.com but I'm not checking intermediate certs...

rixed 7y ago

> Has anyone developed a tool designed specifically to avoid certificate expiry disasters? Not perfect, but I've added a TLS certificate extraction tool into a DPI that displays all visible certificates ordered by expiry date. One could then mirror all one's site traffic to it and let it run in the background. Coupled with some alerting tool it would catch most of those cases I guess. I could polish the tool a bit more if there is some interest, but anyone could do it as well. See https://github.com/rixed/junkie https://github.com/rixed/junkie and more specifically the plugin called 'sslogram'.

nothrabannosir 7y ago

Realistically: reduce your own cert renewal window to weekly, if not daily. This forces you to have a good renewal system in place and alerts you to failures long before actual expiration. Quixotically: make cert failure a randomised number, linearly related to how long ago the cert expired. This slowly introduces more and more failures, over a certain “grace period”, which makes the problem less of an extinction level event. It’s not a solution but it definitely would help.