22 ms·

I get that LE is a single point of failure, but is it really that hard to automate cert renewals?

mdaniel 4y ago
My experience is that these questions are scratching at the wrong side of that problem: is it hard to automate? for sure no. Is it hard to monitor effectively and alert someone when it falls over? That's where the rabbit hole starts because there are a lot of "no, but"s hiding in that, not to mention the age-old "who watches the watchers?" Please don't misunderstand: I'm not arguing against the concept of doing something more often to drive down risk, but I do think it's not as "how hard can it be?" that folks make it out to be. It is trading one set of risks for another, and when the ACME goes off the rails, it takes out your global Internet provider, evidently
- nubinetwork 4y ago
  While I'm just one person, the only issues I've had with ACME stem from people registering false domains pointed at my IP, and an over enthusiatic WAF, which caused legitimate requests to get blocked.
- galleywest200 4y ago
  Every single time we deploy updates to our production code, we also simultaneously run a check against any main TLS certificates we use to see when they expire -- it is built into the scripts used to verify the deployment. We also have certificate checks running weekly that alert to slack channels. While we cannot automate the select few on Digicert, we can automate any of the self-signed certificates (you can use your own CA still in 2023) or any of the Let's Encrypt ones.
28304283409234 4y ago
No. It is really hard to convince the "Product Owner" that this automation is more important than the HR half-year review though (or whatever scrummy reason that has nothing to do with providing IT services and everything with building software products).
grecy 4y ago
Eh, I worked at the biggest Telco in Canada (50k employees), and we had failures all the time due to certs expiring. It's incredible what happens when an organization is big enough, the years roll by, teams change, people leave and stuff just gets forgotten about. Even stuff that has tens of thousands of customers bringing in millions in revenue a year just doesn't have an owner or a person accountable. Happens every week.

My experience is that these questions are scratching at the wrong side of that problem: is it hard to automate? for sure no. Is it hard to monitor effectively and alert someone when it falls over? That's where the rabbit hole starts because there are a lot of "no, but"s hiding in that, not to mention the age-old "who watches the watchers?" Please don't misunderstand: I'm not arguing against the concept of doing something more often to drive down risk, but I do think it's not as "how hard can it be?" that folks make it out to be. It is trading one set of risks for another, and when the ACME goes off the rails, it takes out your global Internet provider, evidently

nubinetwork 4y ago
While I'm just one person, the only issues I've had with ACME stem from people registering false domains pointed at my IP, and an over enthusiatic WAF, which caused legitimate requests to get blocked.
galleywest200 4y ago
Every single time we deploy updates to our production code, we also simultaneously run a check against any main TLS certificates we use to see when they expire -- it is built into the scripts used to verify the deployment. We also have certificate checks running weekly that alert to slack channels. While we cannot automate the select few on Digicert, we can automate any of the self-signed certificates (you can use your own CA still in 2023) or any of the Let's Encrypt ones.

28304283409234 4y ago

No. It is really hard to convince the "Product Owner" that this automation is more important than the HR half-year review though (or whatever scrummy reason that has nothing to do with providing IT services and everything with building software products).

grecy 4y ago

Eh, I worked at the biggest Telco in Canada (50k employees), and we had failures all the time due to certs expiring. It's incredible what happens when an organization is big enough, the years roll by, teams change, people leave and stuff just gets forgotten about. Even stuff that has tens of thousands of customers bringing in millions in revenue a year just doesn't have an owner or a person accountable. Happens every week.