5 ms·

I work for government and I can tell you the guys working infrastructure are still paying for shitty SSL certificates every year, in most cases for infrastructu

I work for government and I can tell you the guys working infrastructure are still paying for shitty SSL certificates every year, in most cases for infrastructure that doesn't even see the light of day (internal), and the reason for that is none other that not knowing any better, and being unable to get their head out of their asses for enough time to learn something novel and implement it in their workflow. So yeah, there are those types out there in thew wild still.

stego-tech 1y ago
In our defense, it’s because we’re expected to give everything a cert but often have no say on the security and cryptography capabilities of what’s brought onto the network in the first place, nevermind the manpower and time to build such an automated solution internally. Execs bringing in MFPs that don’t support TLS, PLCs that require SHA-1, routers with a packet buffer measured in single-digit integers but with a Java web GUI, all of these horrors and more are why it’s such a shitshow even today. Just because someone’s homelab is fully cert’d through Caddy and LE that they slapped together over a weekend two years ago, doesn’t mean the process is trivial or easy for the masses. Believe me, I’ve been fighting this battle internally my entire career and I hate it. I hate the shitty state of PKI today, and how the sole focus seems to be public-facing web services instead of, y’know, the other 90% of a network’s devices and resources. PKI isn’t a solved problem.
- stackskipton 1y ago
  It is solved but devices you are talking about refuse to get on board with the fix so here we are. Also, I used to do IT, I get it but what do you think the fix here is? You could also run your own CA that you push to all the devices and then you can cut certificates as long as you want.
  - Muromec 1y ago
    Where I am right now, if devices cant do what regulator and internal security asks, we find a new vendor.
- znpy 1y ago
  Don't take this as a snarky comment, but that sounds quite literally as "skill issue". Not in you personally, but in the environment you work in. > PKI isn’t a solved problem. PKI is largely a solved issue nowadays. Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need. It's been out for years now, integrating the root CA shouldn't be much of an issue via group policies (in windows, there are equivalents for mac os and gnu/linux i guess). > Just because someone’s homelab is fully cert’d through Caddy and LE that they slapped together over a weekend two years ago, doesn’t mean the process is trivial or easy for the masses. Quite the contrary: it means that the process is technically so trivial the masses can do it in an afternoon and live off it for years with little to no maintenance. Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology.
  - ecb_penguin 1y ago
    > but that sounds quite literally as "skill issue". Not in you personally, but in the environment you work in. You have no idea the environment they work in. The "skill issue" here is you thinking your basic knowledge of Vault matters. > Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need. They didn't tell you their needs, but you're convinced this vendor product solves it. Are you a non-technical CTO by chance? > there are equivalents for mac os and gnu/linux i guess You guess? I'm sensing a skill issue. Why would you say it's solved for their environment, "I guess??" > Quite the contrary: it means that the process is technically so trivial the masses can do it in an afternoon and live off it for years with little to no maintenance. I'm sensing you work in a low skill environment if you think "home lab trivial" translates to enterprise and defense. > Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology. Absolutely meaningless statement.
    - znpy 1y ago
- tetha 1y ago
  Aye. Between Lets Encrypt and compatible vendors, and e.g. Vault for internal CA-chains, /issuing/ certs in a secure, controlled and audited way (at least for the internal ones) is indeed a solved issue. We do have a lot of internal PKI chains running through vault and most of them are pushing 72 hour TTLs. If you can do that, do that. It's great. That being said, deploying and loading these certs is a fun adventure, even in somewhat modern software. We're discovering a surprising amount of fairly modern software which handles certs in stupid ways. For example, if I recall right, NodeJS applications tend to load certificates to secure a connection to PostgreSQL into memory and never bother to reload. libpq on the other hand loads them on connection startup. Even Spring wasn't able to properly reload certificates 3-4 years ago and this was only added in recent versions with dynamic certificates - a colleague patched this in back in the day. Our loadbalancers folded the ticket of "Hey, reload these three certs" into the issue of reloading the entire configuration, including opening, closing ports, handing TCP connections over transparently, and a billion other topics. Funny enough, if there is enough stuff running, there's not even an agreement on how to store stuff. Some stuff needs PEM-files, some stuff needs PKCS8 stores, some stuff wants 1-2 PKCS12 keyrings with certs and keys in there in separate entries. I even had to patch a PKCS12 handling library because one ruby library needed everything in one entry in a PKCS12 key store and there was no go-code under the sun to find to do this. So there is a sunny place in which PKI is indeed solved. And besides that there is a deep dark hole that goes deeper than it should on earth.
luma 1y ago
Alternatively, those people are dealing with legacy systems that are pathologically resistant to cert automation (looking SQUARELY AT YOU vmware) and elect for the longest lasting certs they can get their hands on to minimize the disruption. It’s generally best to assume experts in other fields are doing things for good reasons, and if you don’t understand the reason it might be something other than them being dumb.
- pelagicAustral 1y ago
  Yeah OK, touché, I was over the top... I just need my cup of coffee...
- ruszki 1y ago
  Generally agree, that we shouldn’t jump on conclusions, but experts in my own field are doing bad things mainly because of lack of knowledge. With the lack of will as a very close contender for the top position. “The lack of possibility” is the exception. When I argued here that Linux is still pain to maintain on my laptops, it wasn’t because it’s not possible. I just didn’t have enough willpower. People who jumped on me because of this were still idiots, because they thought that their even less knowledge (what were the error messages, my exact setup, etc) can be helpful for me, but still it would be wrong to state that I was unsuccessful because it’s not possible, or I had full knowledge back then. Especially now, that I’m running Linux full time. For example, I’m quite sure that you can solve this problem with a proxy, no matter what’s behind it. Maybe it’s infeasible, because you would need a custom proxy implementation, but it’s definitely possible.
- nickf 1y ago
  If there are systems that are that resistant to automation, the question should be 'does this system need a publicly-trusted server certificate, the same as a blog about cats or a Shopify shop?'. The answer is no. If it can't practically be automated, it near-certainly doesn't need to have a public cert on it.

stego-tech 1y ago

In our defense, it’s because we’re expected to give everything a cert but often have no say on the security and cryptography capabilities of what’s brought onto the network in the first place, nevermind the manpower and time to build such an automated solution internally. Execs bringing in MFPs that don’t support TLS, PLCs that require SHA-1, routers with a packet buffer measured in single-digit integers but with a Java web GUI, all of these horrors and more are why it’s such a shitshow even today. Just because someone’s homelab is fully cert’d through Caddy and LE that they slapped together over a weekend two years ago, doesn’t mean the process is trivial or easy for the masses. Believe me, I’ve been fighting this battle internally my entire career and I hate it. I hate the shitty state of PKI today, and how the sole focus seems to be public-facing web services instead of, y’know, the other 90% of a network’s devices and resources. PKI isn’t a solved problem.

stackskipton 1y ago
It is solved but devices you are talking about refuse to get on board with the fix so here we are. Also, I used to do IT, I get it but what do you think the fix here is? You could also run your own CA that you push to all the devices and then you can cut certificates as long as you want.
- Muromec 1y ago
  Where I am right now, if devices cant do what regulator and internal security asks, we find a new vendor.
znpy 1y ago
Don't take this as a snarky comment, but that sounds quite literally as "skill issue". Not in you personally, but in the environment you work in. > PKI isn’t a solved problem. PKI is largely a solved issue nowadays. Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need. It's been out for years now, integrating the root CA shouldn't be much of an issue via group policies (in windows, there are equivalents for mac os and gnu/linux i guess). > Just because someone’s homelab is fully cert’d through Caddy and LE that they slapped together over a weekend two years ago, doesn’t mean the process is trivial or easy for the masses. Quite the contrary: it means that the process is technically so trivial the masses can do it in an afternoon and live off it for years with little to no maintenance. Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology.
- ecb_penguin 1y ago
  > but that sounds quite literally as "skill issue". Not in you personally, but in the environment you work in. You have no idea the environment they work in. The "skill issue" here is you thinking your basic knowledge of Vault matters. > Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need. They didn't tell you their needs, but you're convinced this vendor product solves it. Are you a non-technical CTO by chance? > there are equivalents for mac os and gnu/linux i guess You guess? I'm sensing a skill issue. Why would you say it's solved for their environment, "I guess??" > Quite the contrary: it means that the process is technically so trivial the masses can do it in an afternoon and live off it for years with little to no maintenance. I'm sensing you work in a low skill environment if you think "home lab trivial" translates to enterprise and defense. > Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology. Absolutely meaningless statement.
  - znpy 1y ago
tetha 1y ago
Aye. Between Lets Encrypt and compatible vendors, and e.g. Vault for internal CA-chains, /issuing/ certs in a secure, controlled and audited way (at least for the internal ones) is indeed a solved issue. We do have a lot of internal PKI chains running through vault and most of them are pushing 72 hour TTLs. If you can do that, do that. It's great. That being said, deploying and loading these certs is a fun adventure, even in somewhat modern software. We're discovering a surprising amount of fairly modern software which handles certs in stupid ways. For example, if I recall right, NodeJS applications tend to load certificates to secure a connection to PostgreSQL into memory and never bother to reload. libpq on the other hand loads them on connection startup. Even Spring wasn't able to properly reload certificates 3-4 years ago and this was only added in recent versions with dynamic certificates - a colleague patched this in back in the day. Our loadbalancers folded the ticket of "Hey, reload these three certs" into the issue of reloading the entire configuration, including opening, closing ports, handing TCP connections over transparently, and a billion other topics. Funny enough, if there is enough stuff running, there's not even an agreement on how to store stuff. Some stuff needs PEM-files, some stuff needs PKCS8 stores, some stuff wants 1-2 PKCS12 keyrings with certs and keys in there in separate entries. I even had to patch a PKCS12 handling library because one ruby library needed everything in one entry in a PKCS12 key store and there was no go-code under the sun to find to do this. So there is a sunny place in which PKI is indeed solved. And besides that there is a deep dark hole that goes deeper than it should on earth.

luma 1y ago

Alternatively, those people are dealing with legacy systems that are pathologically resistant to cert automation (looking SQUARELY AT YOU vmware) and elect for the longest lasting certs they can get their hands on to minimize the disruption. It’s generally best to assume experts in other fields are doing things for good reasons, and if you don’t understand the reason it might be something other than them being dumb.

pelagicAustral 1y ago
Yeah OK, touché, I was over the top... I just need my cup of coffee...
ruszki 1y ago
Generally agree, that we shouldn’t jump on conclusions, but experts in my own field are doing bad things mainly because of lack of knowledge. With the lack of will as a very close contender for the top position. “The lack of possibility” is the exception. When I argued here that Linux is still pain to maintain on my laptops, it wasn’t because it’s not possible. I just didn’t have enough willpower. People who jumped on me because of this were still idiots, because they thought that their even less knowledge (what were the error messages, my exact setup, etc) can be helpful for me, but still it would be wrong to state that I was unsuccessful because it’s not possible, or I had full knowledge back then. Especially now, that I’m running Linux full time. For example, I’m quite sure that you can solve this problem with a proxy, no matter what’s behind it. Maybe it’s infeasible, because you would need a custom proxy implementation, but it’s definitely possible.
nickf 1y ago
If there are systems that are that resistant to automation, the question should be 'does this system need a publicly-trusted server certificate, the same as a blog about cats or a Shopify shop?'. The answer is no. If it can't practically be automated, it near-certainly doesn't need to have a public cert on it.