3 ms·
Typical certificate management practices for internal PKI are just absolutely set up to cause outages like this. The certificates get issued for a year, or two
by NovemberWhiskey 6y ago
Typical certificate management practices for internal PKI are just absolutely set up to cause outages like this. The certificates get issued for a year, or two years, or whatever. This is infrequent enough that it doesn't feel like it makes sense to automate the process, and then it becomes a run-book that only ever comes out once a year, it's way too easy to add additional services without remembering to monitor which certificates you deployed to them etc.
Start from the idea that you're going to issue certificates valid for 24 hours, and think how different your environment would need to look.
- darkwater 6y agoI know that automation is key here and all the benefits that doing thing very often bring BUT in the specific case of certificates issue if you have short lived certs now you have to ensure your CA system works perfectly, as its uptime now is the uptime of your whole platform. Yeah you can outsource it to AWS ACM and the like, or use Hashicorp Vault but still, it's something that before the change was totally static and now is an extra moving part. I'm not advocating against it, just exposing the whole story.
- ArchOversight 6y agoWith a short expiry, let's say 90 days, you should be renewing 30 days ahead of time, so at the 60 day mark you attempt to renew. This grants you 30 days to fix any problems and get the system back up.
- shuntress 6y agoThen you have to automate handling the 60-day renewal failure warnings which adds another moving part.
- johncolanduoni 6y agoIf you can’t reliably automate sending an unignorable message to some set of humans when something fails to happen, you’re going to have a tough time keeping anything actively developed online.
- NovemberWhiskey 6y agoIt depends on your organization, but in many ways the enterprise PKI CA is one of the easiest services to run at high availability. There are hardly any shared-data dependencies, so it's easy to scale; it's almost completely CPU bound with highly predictable demand, etc. Pretending it's "totally static" is exactly the problem. There are only two kinds of things in the software world - things that can stay the same until your next release, and things that need automation. "Almost completely static" is how your post mortem ends up on the front page of HN. A consideration of the full story also needs to include the risks associated with long-lived certificates. If you lose control of the private key associated with one, what do you do? Are you actually operating a CRL? Are any of your HTTPS clients actually checking the CRL? What would you do if a severe compromise were discovered that affected the signature algorithm you're using?
- marcosdumay 6y agoIt's worse than one not feeling it's worth it. Automating rare events is basically futile, because the next time your automation runs, everything will have changed and it will break.
- de6u99er 6y agoNot if you treat automation as a first class citizen.
- cwkoss 6y agoIf your automation has good logging and you have good alerting on logs, isn't it much better to see the automated process fail as a notification it needs to be done manually rather than relying on it being remembered? (Ideally, you'd remember and never set the alert off, but still great to have that extra layer.
- marcosdumay 6y agoIt's not much different from a notification telling you the activity is due. The difference is mostly a matter of what kind of notifications your organization ignores, and well, I've seen both cases. Anyway, the best is to shorten the certificates validity. The way Letsencrypt recommends is perfect, run it often and require several failures before anything breaks.
- cortesoft 6y agoIf that is the benefit, than why not just send an auto-reminder notification and skip the automation part?
- Already__Taken 6y agowhen your automation fails you get to start by fixing 1% of the task not 100%
- athorax 6y agoUgh, I feel this in my bones.
- qwertox 6y agoThe 3 month limit from Let's Encrypt was a blessing to me, as it forced me to monitor and automate all the renewals. I renew once a month, and if things should break, I have a two month window to fix the issues. Before that I would receive a Comodo SSL Certificate once a year via email and until then I always had forgotten what I had to do with it. What an unnecessary pain.
- atkbrah 6y agoI suppose it's the "green lock" that drives people to still use certificates issued non-automated way.
- lights0123 6y agoWhat browsers still show a green lock, or anything to differentiate EV certs? Only IE?
- NovemberWhiskey 6y agoExtended validation is dead; none of Chrome, Firefox or Safari show any distinguishing UI at this point.