4 ms·

To me, 10 years is long enough to completely forget how to fix the problem, once it becomes a problem in 10 years' time. Most people won't document well enough

To me, 10 years is long enough to completely forget how to fix the problem, once it becomes a problem in 10 years' time. Most people won't document well enough to even be aware that the 10-year deadline is approaching, much less how to fix it! When the deadline hits, everything will break, and then you basically have to reinvent the wheel to get it up and running again. In my opinion, if something can last 10 years, then it could probably just last indefinitely. Alternatively, have something that lasts a short time, but that is renewed automatically. I suppose that is the advantage of let's encrypt, for example, in that their certificates expire after 3 months, but that just means that I set up a Cron job to automatically renew them for me.

infl8ed 3y ago
I love love love the let's encrypt 3 months expire for this reason, it used to be such a pain to remember how to do the yearly/biyearly renew and now, as you say, it's just a Cron job.
- magicalhippo 3y ago
  At work, our customers need to get new certificates from a gov't agency every other year. Most of them have either completely forgotten about it and how to do it, or there's been a change of employees and the new ones didn't get the memo, so to speak. So it falls on us to remind them and guide them through the process. How I wish the gov't moved to a 3 month setup like Let's Encrypt.
  - jwestbury 3y ago
    I'll go further: Three months is too long. Secrets which are used to authenticate and identify should be rotated far more regularly, using infrastructure which treats them as effectively ephemeral. The industry has learned to do this -- and built the infrastructure to support it! -- for things like user credentials (see: extensive use of AWS IAM roles, rather than user creds). We should be making a push to treat certificates the same way. (That said, three months is better than any longer period. The shorter the rotation, the lower the risk -- but, more importantly, the stronger the impetus to build strong automation around the process.)
    - wongarsu 3y ago
      A three month expiration time with automatic renewal after two months (as letsencrypt recommends) is a sweet spot for me. When something breaks this gives you 30 days to figure out that something went wrong and to fix it with zero customer impact. The 30 day grace window is also long enough that let's encrypt will send you two emails (at the 19 day and 9 day thresholds) to make you aware that something might be going wrong. If we lowered the expiration time to say 3 days, with automatic renewal after 2 days, then any breakage on your side or downtime on let's encrypt's side would quickly escalate into https errors. That in turn would train users that those just happen, and make them ignore the big red scary page even when it's an actual attack. That sounds much worse than the small risk from a 30 day certificate.
      - vbezhenar 3y ago
        > If we lowered the expiration time to say 3 days, with automatic renewal after 2 days, then any breakage on your side or downtime on let's encrypt's side would quickly escalate into https errors. That in turn would train users that those just happen, and make them ignore the big red scary page even when it's an actual attack. That sounds much worse than the small risk from a 30 day certificate. That's already happened. I'm encountering LE errors on random websites so much that I don't care and automatically click through warnings. This is especially troublesome because my government keeps MITM me and I don't like it.
        JohnFen 3y ago
        This is my experience as well. I encounter cert errors more now than ever, and I tend to ignore them.
    - red-iron-pine 3y ago
      > The shorter the rotation, the lower the risk the lower the risk of compromised certs / keys. certainly not a lower risk of issues, or surprises. hopefully -- emphasis on hope -- this regular action becomes routine and easy enough to that it is a low risk behavior.
  - nonameiguess 3y ago
    Depending on the government agency, there may be a required level of ongoing identity and need verification that can't be automated. For personal PKI in the US DoD, for instance, you have to go in-person to an ID office on a military installation to get your common access card renewed. For server certs, there is obviously no way to make a server go somewhere physically, but you need a qualified sponsor to sign off on the request to the DoD PKI office, and who that person is will likely change over any multi-year span, as military command positions tend not to last more than a year and even the civilian offices still see fairly frequent turnover at the higher levels. Plus those people need to sign requests with their common access card, which requires them to periodically go to an ID office in-person.
- Aardwolf 3y ago
  I did this manually for a while with a reminder every 3 months but now it happens automagically with a cron job on my server as well and now I'm similarly at risk of forgetting how it worked in the first place
  - KronisLV 3y ago
    > now I'm similarly at risk of forgetting how it worked in the first place I once met a person at a client org who was generally opposed to automation due to risks of forgetting how things work and not always knowing what the internals behind those abstractions are. It was an interesting take. At the same time, something like Ansible and other methods of automation can be pretty useful and actually aid in documenting things. It's especially good if you can spare 10% of your time to put some notes down in Markdown files in a Git repo, or source/deploy most of your automation scripts from there as well.
- JohnFen 3y ago
  LE's short expiry is the primary reason why I don't use it. Yes, I know, automation is the approved solution for this, but it's not a great solution for me.
  - dijit 3y ago
    Agreed, when the tools stop working (which, they do) then suddenly what was swapping out a file instead becomes a big ordeal with fighting nginx .well-known bypass or trying to figure out why lets encrypt can't connect via IPv6 but everything else seems to be able to or, in my case, when certbot-auto stopped working and had no upgrade path on oBSD. my blog and personal website are down for this reason, I simply can't spend half-a-day at this point in my life figuring out how to do this on OpenBSD. So I'd rather just leave it dead at this point. Guess I could just buy an SSL certificate still, maybe I do that tonight.
    - WirelessGigabit 3y ago
      I use DNS-01. In fact, it's the only way I can do it as LE doesn't have access to my internal setup. And buying an SSL cert only gives you 368 days in Chrome / Apple browsers: https://support.apple.com/en-us/102028 https://support.apple.com/en-us/102028
      - dijit 3y ago
        DNS-01 is awkward with multiple TLDs and providers for a site. For me it’s like; blog.jharasym.com - namecheap blog.jharasym.dev - gandi blog.dijit.sh - self hosted with BIND
        justsomehnguy 3y ago
        'DNS Alias Mode', eg https://dan.langille.org/2019/02/01/acme-domain-alias-mode/ https://dan.langille.org/2019/02/01/acme-domain-alias-mode/
- tomjen3 3y ago
  Lets encrypt makes sense in a world where you have pet servers and can install random software like certbot and keep them running for years. In a world of containerized and immutable cattle servers its not a good solution. Especially not when you technically only need it for something that is internally accessible. Currently my homelab setup is based on running certbot locally in a docker and a calender entry - maybe I get annoyed enough and switch to my own cert at some point, but those are also a big pita.
BuildTheRobots 3y ago
My last boss was very keen on change control on any of our live systems; steps documented to the point where the cleaner could have run them. It seemed a bit excessive at the time, but there's something lovely about having copy/pasteable steps (and any notes) from when you had to do this a year ago. As for forgetting deadlines are approaching, just set up automated checks for cert and domain expiration. I wasn't just checking our infrastructure but also any remote APIs/interconnects we were using too. There were a handful of times where I'd contact providers with 7 days warning just to confirm they were aware their certs were expiring; it was slightly terrifying how often they weren't.
- thechao 3y ago
  I'm about to write a little "choose-your-own-adventure" script that walks new employees through our processes. We're about to begin our decade-ly turning-over, and the idea is that the we'll hand over the script and fix the "bugs" in the script, rather than explaining "out loud". Most of these process scripts are used on a weekly or monthly cadence, so the hope is they'll stay up-to-date.
Jach 3y ago
On the other hand, carefully written doc from 10 years ago might not even apply. So you're going to have to do things from scratch again regardless. Often, that's not really a big deal. And it can happen just as well to stable stuff or cron'd stuff that was working fine for x years until it stopped working.
RecycledEle 3y ago
Why isn't there a special log file for when things expire, so I can sort it by expiration date?
pc86 3y ago
I've had several problems like this at various jobs where something silently breaks, all hell breaks loose, and it takes the better part of an entire day from an entire team to figure out what 15-second change needs to be made to fix everything.

infl8ed 3y ago

I love love love the let's encrypt 3 months expire for this reason, it used to be such a pain to remember how to do the yearly/biyearly renew and now, as you say, it's just a Cron job.

magicalhippo 3y ago
At work, our customers need to get new certificates from a gov't agency every other year. Most of them have either completely forgotten about it and how to do it, or there's been a change of employees and the new ones didn't get the memo, so to speak. So it falls on us to remind them and guide them through the process. How I wish the gov't moved to a 3 month setup like Let's Encrypt.
- jwestbury 3y ago
  I'll go further: Three months is too long. Secrets which are used to authenticate and identify should be rotated far more regularly, using infrastructure which treats them as effectively ephemeral. The industry has learned to do this -- and built the infrastructure to support it! -- for things like user credentials (see: extensive use of AWS IAM roles, rather than user creds). We should be making a push to treat certificates the same way. (That said, three months is better than any longer period. The shorter the rotation, the lower the risk -- but, more importantly, the stronger the impetus to build strong automation around the process.)
  - wongarsu 3y ago
    A three month expiration time with automatic renewal after two months (as letsencrypt recommends) is a sweet spot for me. When something breaks this gives you 30 days to figure out that something went wrong and to fix it with zero customer impact. The 30 day grace window is also long enough that let's encrypt will send you two emails (at the 19 day and 9 day thresholds) to make you aware that something might be going wrong. If we lowered the expiration time to say 3 days, with automatic renewal after 2 days, then any breakage on your side or downtime on let's encrypt's side would quickly escalate into https errors. That in turn would train users that those just happen, and make them ignore the big red scary page even when it's an actual attack. That sounds much worse than the small risk from a 30 day certificate.
    - vbezhenar 3y ago
      > If we lowered the expiration time to say 3 days, with automatic renewal after 2 days, then any breakage on your side or downtime on let's encrypt's side would quickly escalate into https errors. That in turn would train users that those just happen, and make them ignore the big red scary page even when it's an actual attack. That sounds much worse than the small risk from a 30 day certificate. That's already happened. I'm encountering LE errors on random websites so much that I don't care and automatically click through warnings. This is especially troublesome because my government keeps MITM me and I don't like it.
      - JohnFen 3y ago
        This is my experience as well. I encounter cert errors more now than ever, and I tend to ignore them.
  - red-iron-pine 3y ago
    > The shorter the rotation, the lower the risk the lower the risk of compromised certs / keys. certainly not a lower risk of issues, or surprises. hopefully -- emphasis on hope -- this regular action becomes routine and easy enough to that it is a low risk behavior.
- nonameiguess 3y ago
  Depending on the government agency, there may be a required level of ongoing identity and need verification that can't be automated. For personal PKI in the US DoD, for instance, you have to go in-person to an ID office on a military installation to get your common access card renewed. For server certs, there is obviously no way to make a server go somewhere physically, but you need a qualified sponsor to sign off on the request to the DoD PKI office, and who that person is will likely change over any multi-year span, as military command positions tend not to last more than a year and even the civilian offices still see fairly frequent turnover at the higher levels. Plus those people need to sign requests with their common access card, which requires them to periodically go to an ID office in-person.
Aardwolf 3y ago
I did this manually for a while with a reminder every 3 months but now it happens automagically with a cron job on my server as well and now I'm similarly at risk of forgetting how it worked in the first place
- KronisLV 3y ago
  > now I'm similarly at risk of forgetting how it worked in the first place I once met a person at a client org who was generally opposed to automation due to risks of forgetting how things work and not always knowing what the internals behind those abstractions are. It was an interesting take. At the same time, something like Ansible and other methods of automation can be pretty useful and actually aid in documenting things. It's especially good if you can spare 10% of your time to put some notes down in Markdown files in a Git repo, or source/deploy most of your automation scripts from there as well.
JohnFen 3y ago
LE's short expiry is the primary reason why I don't use it. Yes, I know, automation is the approved solution for this, but it's not a great solution for me.
- dijit 3y ago
  Agreed, when the tools stop working (which, they do) then suddenly what was swapping out a file instead becomes a big ordeal with fighting nginx .well-known bypass or trying to figure out why lets encrypt can't connect via IPv6 but everything else seems to be able to or, in my case, when certbot-auto stopped working and had no upgrade path on oBSD. my blog and personal website are down for this reason, I simply can't spend half-a-day at this point in my life figuring out how to do this on OpenBSD. So I'd rather just leave it dead at this point. Guess I could just buy an SSL certificate still, maybe I do that tonight.
  - WirelessGigabit 3y ago
    I use DNS-01. In fact, it's the only way I can do it as LE doesn't have access to my internal setup. And buying an SSL cert only gives you 368 days in Chrome / Apple browsers: https://support.apple.com/en-us/102028 https://support.apple.com/en-us/102028
    - dijit 3y ago
      DNS-01 is awkward with multiple TLDs and providers for a site. For me it’s like; blog.jharasym.com - namecheap blog.jharasym.dev - gandi blog.dijit.sh - self hosted with BIND
      - justsomehnguy 3y ago
        'DNS Alias Mode', eg https://dan.langille.org/2019/02/01/acme-domain-alias-mode/ https://dan.langille.org/2019/02/01/acme-domain-alias-mode/
tomjen3 3y ago
Lets encrypt makes sense in a world where you have pet servers and can install random software like certbot and keep them running for years. In a world of containerized and immutable cattle servers its not a good solution. Especially not when you technically only need it for something that is internally accessible. Currently my homelab setup is based on running certbot locally in a docker and a calender entry - maybe I get annoyed enough and switch to my own cert at some point, but those are also a big pita.

BuildTheRobots 3y ago

My last boss was very keen on change control on any of our live systems; steps documented to the point where the cleaner could have run them. It seemed a bit excessive at the time, but there's something lovely about having copy/pasteable steps (and any notes) from when you had to do this a year ago. As for forgetting deadlines are approaching, just set up automated checks for cert and domain expiration. I wasn't just checking our infrastructure but also any remote APIs/interconnects we were using too. There were a handful of times where I'd contact providers with 7 days warning just to confirm they were aware their certs were expiring; it was slightly terrifying how often they weren't.

thechao 3y ago
I'm about to write a little "choose-your-own-adventure" script that walks new employees through our processes. We're about to begin our decade-ly turning-over, and the idea is that the we'll hand over the script and fix the "bugs" in the script, rather than explaining "out loud". Most of these process scripts are used on a weekly or monthly cadence, so the hope is they'll stay up-to-date.

Jach 3y ago

On the other hand, carefully written doc from 10 years ago might not even apply. So you're going to have to do things from scratch again regardless. Often, that's not really a big deal. And it can happen just as well to stable stuff or cron'd stuff that was working fine for x years until it stopped working.

RecycledEle 3y ago

Why isn't there a special log file for when things expire, so I can sort it by expiration date?

pc86 3y ago

I've had several problems like this at various jobs where something silently breaks, all hell breaks loose, and it takes the better part of an entire day from an entire team to figure out what 15-second change needs to be made to fix everything.