7 ms·

Certificate management is such a huge problem for big organizations. Does anyone have guidelines on how to do it properly? You really need a way to track who is

by rb808 7y ago

Certificate management is such a huge problem for big organizations. Does anyone have guidelines on how to do it properly? You really need a way to track who is using what and how they automatically get renewed a long time before expiry. My view now is that certificates should be renewed very often - like every month, if you see something within 6 months of expiry there is a problem.

tyingq 7y ago
There is software like Venifi, but it requires everyone to follow a specific process. You still have outliers.
Znafon 7y ago
You can use something like Vault with the PKI backend and an audit engine set up. You will get a full trace of all certs delivered and an API endpoint to update them.
- rb808 7y ago
  https://www.vaultproject.io/docs/secrets/pki/index.html https://www.vaultproject.io/docs/secrets/pki/index.html thanks
hylaride 7y ago
I'd like to know as well. At my work we had to partner with a major bank and they demanded a fully signed (sigh) client certificate for the integration. The way their API works means that we have to do a planned hard-cutover before the certificate expires. There's a 60% chance that we'll have trouble reaching out to the right people when the time comes. Ugh.
- uxp 7y ago
  This is a common problem in healthcare insurance. API connections will stop working for about 3 weeks every year as each side performs a complicated dance of noticing the expiration and updating the systems to accept a new public certificate.
  - ehvatum 7y ago
    Seems to be a thing that happens at big companies in general. I add —-no-ssl-verify to everything and it saves my ass every single year.
    - hylaride 7y ago
      That's ok if you're not worried about man in the middle attacks, but it does make it a lot less secure.
- pnutjam 7y ago
  IMHO, 60% sounds low...
doublement 7y ago
A short period of renewal would mitigate the problem where every employee on the notification list has become a former employee / invalid email by the time the actual expiry occurs.
- r00fus 7y ago
  That kind of turnover seems crazy. You should also be using an email distribution list as the "notification target" which is kept up to date when someone joins or leaves the team...
allset_ 7y ago
You should know your assets. If you know your assets, it's pretty trivial to write a script or program that checks when their certificate expires, sticks it in a database, and alerts you when it's getting close to time to renew. Of course, if you don't know your assets, you hopefully at least know your IP space and can scan that.
_jal 7y ago
We take a belt-and-suspenders approach, mainly because I have a huge fear of the embarrassment of letting a cert (or domain name) expire. We built a DB to track them; nothing special, it sends email reminders. Been meaning to just publish calendar reminders, but haven't gotten to it. We also check expiry dates on the live certificates via one of our monitoring systems (Zabbix, but this isn't a built-in check) that will start yelling, I think, 2 weeks before expiry.
somurzakov 7y ago
check Venafi for scanning your network for certs and managing them you can also build scanner yourself on top of nmap and store discovered certs in one DB and link it with your configuration management db
jlgaddis 7y ago
> You really need a way to track who is using what and how they automatically get renewed a long time before expiry. If you have any sort of monitoring system in place, it should also be able to check for certificate expiration (or "validity" in general). Even the old antiquated nagios has had this ability since the dawn of time. As a sort of last resort and/or where no other solution is in place, you can do what I did on a customer's network a few years ago. Fortunately, there wasn't much "security" in place (internally, at least). nmap can easily 1) detect/identify TLS and 2) produce "grepable" output. I simply gathered a list of all IP subnets that were in use and scanned all of it. Afterwards, it was fairly simple to go through the results and figure out which hosts were running services using TLS on which ports. There are existing tools that can connect to services using TLS and gather the information from the presented certificate. If you can't find one that does what you need, that's no big deal, it really doesn't take much to write your own tool just for this purpose. (In fact, something like this -- running on a host that has access to connect to all of your TLS services -- might be nice to have even if you have a "proper" solution in place. If (when?) something falls through the cracks, you'd have this as a sort of "fallback" or "backup" to alert you to anything your proper solution might have missed.)
closeparen 7y ago
Fully automated issuance and rotation on the order of hours. Use stuff like SPIFFE [0]. Won't help you much with 3rd-party proprietary hardware but you can at least avoid dealing with long-lived certificate management for your own software. [0] https://spiffe.io/ https://spiffe.io/
Avamander 7y ago
Automate every device separately. I've written browser pupetteering scripts for that.

by rb808 7y ago

tyingq 7y ago

There is software like Venifi, but it requires everyone to follow a specific process. You still have outliers.

Znafon 7y ago

You can use something like Vault with the PKI backend and an audit engine set up. You will get a full trace of all certs delivered and an API endpoint to update them.

rb808 7y ago
https://www.vaultproject.io/docs/secrets/pki/index.html https://www.vaultproject.io/docs/secrets/pki/index.html thanks

hylaride 7y ago

I'd like to know as well. At my work we had to partner with a major bank and they demanded a fully signed (sigh) client certificate for the integration. The way their API works means that we have to do a planned hard-cutover before the certificate expires. There's a 60% chance that we'll have trouble reaching out to the right people when the time comes. Ugh.

uxp 7y ago
This is a common problem in healthcare insurance. API connections will stop working for about 3 weeks every year as each side performs a complicated dance of noticing the expiration and updating the systems to accept a new public certificate.
- ehvatum 7y ago
  Seems to be a thing that happens at big companies in general. I add —-no-ssl-verify to everything and it saves my ass every single year.
  - hylaride 7y ago
    That's ok if you're not worried about man in the middle attacks, but it does make it a lot less secure.
pnutjam 7y ago
IMHO, 60% sounds low...

doublement 7y ago

A short period of renewal would mitigate the problem where every employee on the notification list has become a former employee / invalid email by the time the actual expiry occurs.

r00fus 7y ago
That kind of turnover seems crazy. You should also be using an email distribution list as the "notification target" which is kept up to date when someone joins or leaves the team...

allset_ 7y ago

You should know your assets. If you know your assets, it's pretty trivial to write a script or program that checks when their certificate expires, sticks it in a database, and alerts you when it's getting close to time to renew. Of course, if you don't know your assets, you hopefully at least know your IP space and can scan that.

_jal 7y ago

We take a belt-and-suspenders approach, mainly because I have a huge fear of the embarrassment of letting a cert (or domain name) expire. We built a DB to track them; nothing special, it sends email reminders. Been meaning to just publish calendar reminders, but haven't gotten to it. We also check expiry dates on the live certificates via one of our monitoring systems (Zabbix, but this isn't a built-in check) that will start yelling, I think, 2 weeks before expiry.

somurzakov 7y ago

check Venafi for scanning your network for certs and managing them you can also build scanner yourself on top of nmap and store discovered certs in one DB and link it with your configuration management db

jlgaddis 7y ago

> You really need a way to track who is using what and how they automatically get renewed a long time before expiry. If you have any sort of monitoring system in place, it should also be able to check for certificate expiration (or "validity" in general). Even the old antiquated nagios has had this ability since the dawn of time. As a sort of last resort and/or where no other solution is in place, you can do what I did on a customer's network a few years ago. Fortunately, there wasn't much "security" in place (internally, at least). nmap can easily 1) detect/identify TLS and 2) produce "grepable" output. I simply gathered a list of all IP subnets that were in use and scanned all of it. Afterwards, it was fairly simple to go through the results and figure out which hosts were running services using TLS on which ports. There are existing tools that can connect to services using TLS and gather the information from the presented certificate. If you can't find one that does what you need, that's no big deal, it really doesn't take much to write your own tool just for this purpose. (In fact, something like this -- running on a host that has access to connect to all of your TLS services -- might be nice to have even if you have a "proper" solution in place. If (when?) something falls through the cracks, you'd have this as a sort of "fallback" or "backup" to alert you to anything your proper solution might have missed.)

closeparen 7y ago

Fully automated issuance and rotation on the order of hours. Use stuff like SPIFFE [0]. Won't help you much with 3rd-party proprietary hardware but you can at least avoid dealing with long-lived certificate management for your own software. [0] https://spiffe.io/ https://spiffe.io/

Avamander 7y ago

Automate every device separately. I've written browser pupetteering scripts for that.