4 ms·

Would be interesting to know how many people who've managed footprints for a reasonable period of time (say 5-10years) who haven't had a cert expire on them. Wo

by RowanH 8y ago

Would be interesting to know how many people who've managed footprints for a reasonable period of time (say 5-10years) who haven't had a cert expire on them. Wouldn't be surprised if it's single digit %ages. So many human & tech error factors lead to this occurring and they're all the same old things. Staffing changes, spam filters, ignored warnings, skipped emails...

stdplaceholder 8y ago
I know it's happened to every company where I've worked. It happens so rarely, though, that people don't have enough opportunity to learn from it. Even at Google they were on their Nth such outage for a large value of N before it became apparent that no certificate should ever expire at 23:59:59 on December 31, or otherwise outside of normal operating hours. Seriously 20 years of organizational knowledge required to get the company to understand that certs should expire at noon on a Wednesday to minimize time-to-repair in the inevitable event that one is allowed to lapse.
- jtl999 8y ago
  Instead of waiting last minute, you'd think a large company would have planning to renew certificates X amount of time before they expire. Alas I understand it's not that simple.
  - stdplaceholder 8y ago
    You'd certainly think so. If you have frontend probers that exercise your accessible endpoints (HTTP or whatever) then those probes should fail when the certificate expires in less than 30 days. I couldn't comment on whether an organization like Ericsson or O2 would be expected to have such probers.
    - jtl999 8y ago
      I should get on that for my own infrastructure. Thanks for the reminder.
  - jcrawfordor 8y ago
    In my small organization we had planning and multiple reminders to renew the cert well before it expired, and we did. Due to a miscommunication between myself and a coworker, the new cert sat ready for nearly two months without ever being added to the configuration (we were both certain the other had done it, naturally). There's a remarkable number of ways for this simple thing to go wrong. To prevent a future repeat, we got rid of our calendar reminders (which we started ignoring once we both thought the change had been made) and wrote a script that emailed us based on the time to expiration of the live cert. This is a much better method. Of course, give us enough years and I'm sure we'll manage to find a way to get this new setup wrong.
msbarnett 8y ago
A really easy way to avoid this in any environment with Continuous Integration style tests running on everyone's work: Add a test that just unconditionally fails on a certain date, like a week before your cert expires. Don't let any code review sign off on a merge of a fix the test until the new cert is in prod. Don't let any code promote between environments while tests are broken. The problem with emails and warnings is they're all ignorable and therefore completely unsuitable for managing something as critical as a cert. The secret is to create a straight-up error that absolutely interrupts every engineer in the organization's day until the cert is renewed. A development-halting error a week before cert expiration is a hell of a lot better than a business-halting error when it expires in prod.
- purple_ducks 8y ago
  > fails on a certain date Or just write a test that checks the date on the cert and conditionally fails. If it's within 4 weeks, send email to x,y,z. If it's within 2 weeks, send it to VP of x, y, z. combination of failing test and email should lessen chance of it going unnoticed(email could possibly fail for whatever reason).

by RowanH 8y ago

stdplaceholder 8y ago

I know it's happened to every company where I've worked. It happens so rarely, though, that people don't have enough opportunity to learn from it. Even at Google they were on their Nth such outage for a large value of N before it became apparent that no certificate should ever expire at 23:59:59 on December 31, or otherwise outside of normal operating hours. Seriously 20 years of organizational knowledge required to get the company to understand that certs should expire at noon on a Wednesday to minimize time-to-repair in the inevitable event that one is allowed to lapse.

jtl999 8y ago
Instead of waiting last minute, you'd think a large company would have planning to renew certificates X amount of time before they expire. Alas I understand it's not that simple.
- stdplaceholder 8y ago
  You'd certainly think so. If you have frontend probers that exercise your accessible endpoints (HTTP or whatever) then those probes should fail when the certificate expires in less than 30 days. I couldn't comment on whether an organization like Ericsson or O2 would be expected to have such probers.
  - jtl999 8y ago
    I should get on that for my own infrastructure. Thanks for the reminder.
- jcrawfordor 8y ago
  In my small organization we had planning and multiple reminders to renew the cert well before it expired, and we did. Due to a miscommunication between myself and a coworker, the new cert sat ready for nearly two months without ever being added to the configuration (we were both certain the other had done it, naturally). There's a remarkable number of ways for this simple thing to go wrong. To prevent a future repeat, we got rid of our calendar reminders (which we started ignoring once we both thought the change had been made) and wrote a script that emailed us based on the time to expiration of the live cert. This is a much better method. Of course, give us enough years and I'm sure we'll manage to find a way to get this new setup wrong.

msbarnett 8y ago

A really easy way to avoid this in any environment with Continuous Integration style tests running on everyone's work: Add a test that just unconditionally fails on a certain date, like a week before your cert expires. Don't let any code review sign off on a merge of a fix the test until the new cert is in prod. Don't let any code promote between environments while tests are broken. The problem with emails and warnings is they're all ignorable and therefore completely unsuitable for managing something as critical as a cert. The secret is to create a straight-up error that absolutely interrupts every engineer in the organization's day until the cert is renewed. A development-halting error a week before cert expiration is a hell of a lot better than a business-halting error when it expires in prod.

purple_ducks 8y ago
> fails on a certain date Or just write a test that checks the date on the cert and conditionally fails. If it's within 4 weeks, send email to x,y,z. If it's within 2 weeks, send it to VP of x, y, z. combination of failing test and email should lessen chance of it going unnoticed(email could possibly fail for whatever reason).