7 ms·

In our defense, it’s because we’re expected to give everything a cert but often have no say on the security and cryptography capabilities of what’s brought onto

In our defense, it’s because we’re expected to give everything a cert but often have no say on the security and cryptography capabilities of what’s brought onto the network in the first place, nevermind the manpower and time to build such an automated solution internally. Execs bringing in MFPs that don’t support TLS, PLCs that require SHA-1, routers with a packet buffer measured in single-digit integers but with a Java web GUI, all of these horrors and more are why it’s such a shitshow even today. Just because someone’s homelab is fully cert’d through Caddy and LE that they slapped together over a weekend two years ago, doesn’t mean the process is trivial or easy for the masses. Believe me, I’ve been fighting this battle internally my entire career and I hate it. I hate the shitty state of PKI today, and how the sole focus seems to be public-facing web services instead of, y’know, the other 90% of a network’s devices and resources. PKI isn’t a solved problem.

stackskipton 1y ago
It is solved but devices you are talking about refuse to get on board with the fix so here we are. Also, I used to do IT, I get it but what do you think the fix here is? You could also run your own CA that you push to all the devices and then you can cut certificates as long as you want.
- Muromec 1y ago
  Where I am right now, if devices cant do what regulator and internal security asks, we find a new vendor.
znpy 1y ago
Don't take this as a snarky comment, but that sounds quite literally as "skill issue". Not in you personally, but in the environment you work in. > PKI isn’t a solved problem. PKI is largely a solved issue nowadays. Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need. It's been out for years now, integrating the root CA shouldn't be much of an issue via group policies (in windows, there are equivalents for mac os and gnu/linux i guess). > Just because someone’s homelab is fully cert’d through Caddy and LE that they slapped together over a weekend two years ago, doesn’t mean the process is trivial or easy for the masses. Quite the contrary: it means that the process is technically so trivial the masses can do it in an afternoon and live off it for years with little to no maintenance. Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology.
- ecb_penguin 1y ago
  > but that sounds quite literally as "skill issue". Not in you personally, but in the environment you work in. You have no idea the environment they work in. The "skill issue" here is you thinking your basic knowledge of Vault matters. > Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need. They didn't tell you their needs, but you're convinced this vendor product solves it. Are you a non-technical CTO by chance? > there are equivalents for mac os and gnu/linux i guess You guess? I'm sensing a skill issue. Why would you say it's solved for their environment, "I guess??" > Quite the contrary: it means that the process is technically so trivial the masses can do it in an afternoon and live off it for years with little to no maintenance. I'm sensing you work in a low skill environment if you think "home lab trivial" translates to enterprise and defense. > Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology. Absolutely meaningless statement.
  - znpy 1y ago
    > You have no idea the environment they work in. The "skill issue" here is you thinking your basic knowledge of Vault matters. I've deployed Vault both at home and in two different companies, doing anything from pki, mutual-tls, secret storage and other stuff. > > Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need. > They didn't tell you their needs, but you're convinced this vendor product solves it. It was an example from the ecosystem of available tools but in general yes, Vault can do that. Mentioning FIPS compliance was about letting you know that the software can be used also in governative environments. It's not just a "homelab toy". > Are you a non-technical CTO by chance? Senior cloud engineer here. Worked anywhere from not-so-small companies (250 people, 100 engineers) to faangs (tens of thousands of engineers). > > there are equivalents for mac os and gnu/linux i guess > You guess? I'm sensing a skill issue. You're attacking me on a personal level because you can't argue otherwise. That's a common logical fallacy ("Ad Hominem" - https://www.britannica.com/topic/ad-hominem https://www.britannica.com/topic/ad-hominem). You basically have skill issue at debating =) > Why would you say it's solved for their environment, "I guess??" When you account Windows, Mac OS and Linux you're accounting for pretty much the totality of the desktop computing landscape. The last two macbooks I had for work came with the mac os equivalent of group policies with certificates installed etc etc. Enterprise-tier Linux distributions can do that as well (eg: Red Hat Enterprise Linux). > I'm sensing you work in a low skill environment if you think "home lab trivial" translates to enterprise and defense. Again, worked anywhere from companies with 250 people to FAANGs. You have skill issue at sensing, it seems. To get back to the point: homelab "triviality". In a way, yes. Large enterprises and even more defense can spend all the money not just for software but even for consulting from various company that can bring the skills to implement and maintain all the services that are needed, and train your people at that. Things become non trivial not on the base of technical issue, but on the base of organizational issues... If we talk government and defense... Do you know the US government has dedicated cloud regions (eg: https://aws.amazon.com/govcloud-us/ https://aws.amazon.com/govcloud-us/)? Do you really think that cloud providers offer those services at loss? Do you really think a few vault enterprise licenses are the issue there? And by the way, Vault is just an example of one of the possible solutions. It was meant to be an example but you clearly missed the point. > > Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology. > Absolutely meaningless statement. I think it's very meaningful. It's not 1995, cryptography isn't arcane anymore. We had hardware crypto acceleration in cpu since at least 2010 (AES-NI). The tooling is well established on both servers and clients. The skills are on the market ready to be hired (either via employment or via contracting). The issue is not technical in nature. Oh and by the way: I've worked closely with engineers working for the US government. I wasn't close to the US government (because I am not an US citizen) but they were. They were "close enough" that they had to work in a SCIF and could only interact with me via phone. The systems they were working on... Those systems had their own private CA (among other things). It's feasible. It's not a technical issue. If it's not done then it's an organizational issue.
tetha 1y ago
Aye. Between Lets Encrypt and compatible vendors, and e.g. Vault for internal CA-chains, /issuing/ certs in a secure, controlled and audited way (at least for the internal ones) is indeed a solved issue. We do have a lot of internal PKI chains running through vault and most of them are pushing 72 hour TTLs. If you can do that, do that. It's great. That being said, deploying and loading these certs is a fun adventure, even in somewhat modern software. We're discovering a surprising amount of fairly modern software which handles certs in stupid ways. For example, if I recall right, NodeJS applications tend to load certificates to secure a connection to PostgreSQL into memory and never bother to reload. libpq on the other hand loads them on connection startup. Even Spring wasn't able to properly reload certificates 3-4 years ago and this was only added in recent versions with dynamic certificates - a colleague patched this in back in the day. Our loadbalancers folded the ticket of "Hey, reload these three certs" into the issue of reloading the entire configuration, including opening, closing ports, handing TCP connections over transparently, and a billion other topics. Funny enough, if there is enough stuff running, there's not even an agreement on how to store stuff. Some stuff needs PEM-files, some stuff needs PKCS8 stores, some stuff wants 1-2 PKCS12 keyrings with certs and keys in there in separate entries. I even had to patch a PKCS12 handling library because one ruby library needed everything in one entry in a PKCS12 key store and there was no go-code under the sun to find to do this. So there is a sunny place in which PKI is indeed solved. And besides that there is a deep dark hole that goes deeper than it should on earth.

stackskipton 1y ago

It is solved but devices you are talking about refuse to get on board with the fix so here we are. Also, I used to do IT, I get it but what do you think the fix here is? You could also run your own CA that you push to all the devices and then you can cut certificates as long as you want.

Muromec 1y ago
Where I am right now, if devices cant do what regulator and internal security asks, we find a new vendor.

znpy 1y ago

Don't take this as a snarky comment, but that sounds quite literally as "skill issue". Not in you personally, but in the environment you work in. > PKI isn’t a solved problem. PKI is largely a solved issue nowadays. Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need. It's been out for years now, integrating the root CA shouldn't be much of an issue via group policies (in windows, there are equivalents for mac os and gnu/linux i guess). > Just because someone’s homelab is fully cert’d through Caddy and LE that they slapped together over a weekend two years ago, doesn’t mean the process is trivial or easy for the masses. Quite the contrary: it means that the process is technically so trivial the masses can do it in an afternoon and live off it for years with little to no maintenance. Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology.

ecb_penguin 1y ago
> but that sounds quite literally as "skill issue". Not in you personally, but in the environment you work in. You have no idea the environment they work in. The "skill issue" here is you thinking your basic knowledge of Vault matters. > Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need. They didn't tell you their needs, but you're convinced this vendor product solves it. Are you a non-technical CTO by chance? > there are equivalents for mac os and gnu/linux i guess You guess? I'm sensing a skill issue. Why would you say it's solved for their environment, "I guess??" > Quite the contrary: it means that the process is technically so trivial the masses can do it in an afternoon and live off it for years with little to no maintenance. I'm sensing you work in a low skill environment if you think "home lab trivial" translates to enterprise and defense. > Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology. Absolutely meaningless statement.
- znpy 1y ago
  > You have no idea the environment they work in. The "skill issue" here is you thinking your basic knowledge of Vault matters. I've deployed Vault both at home and in two different companies, doing anything from pki, mutual-tls, secret storage and other stuff. > > Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need. > They didn't tell you their needs, but you're convinced this vendor product solves it. It was an example from the ecosystem of available tools but in general yes, Vault can do that. Mentioning FIPS compliance was about letting you know that the software can be used also in governative environments. It's not just a "homelab toy". > Are you a non-technical CTO by chance? Senior cloud engineer here. Worked anywhere from not-so-small companies (250 people, 100 engineers) to faangs (tens of thousands of engineers). > > there are equivalents for mac os and gnu/linux i guess > You guess? I'm sensing a skill issue. You're attacking me on a personal level because you can't argue otherwise. That's a common logical fallacy ("Ad Hominem" - https://www.britannica.com/topic/ad-hominem https://www.britannica.com/topic/ad-hominem). You basically have skill issue at debating =) > Why would you say it's solved for their environment, "I guess??" When you account Windows, Mac OS and Linux you're accounting for pretty much the totality of the desktop computing landscape. The last two macbooks I had for work came with the mac os equivalent of group policies with certificates installed etc etc. Enterprise-tier Linux distributions can do that as well (eg: Red Hat Enterprise Linux). > I'm sensing you work in a low skill environment if you think "home lab trivial" translates to enterprise and defense. Again, worked anywhere from companies with 250 people to FAANGs. You have skill issue at sensing, it seems. To get back to the point: homelab "triviality". In a way, yes. Large enterprises and even more defense can spend all the money not just for software but even for consulting from various company that can bring the skills to implement and maintain all the services that are needed, and train your people at that. Things become non trivial not on the base of technical issue, but on the base of organizational issues... If we talk government and defense... Do you know the US government has dedicated cloud regions (eg: https://aws.amazon.com/govcloud-us/ https://aws.amazon.com/govcloud-us/)? Do you really think that cloud providers offer those services at loss? Do you really think a few vault enterprise licenses are the issue there? And by the way, Vault is just an example of one of the possible solutions. It was meant to be an example but you clearly missed the point. > > Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology. > Absolutely meaningless statement. I think it's very meaningful. It's not 1995, cryptography isn't arcane anymore. We had hardware crypto acceleration in cpu since at least 2010 (AES-NI). The tooling is well established on both servers and clients. The skills are on the market ready to be hired (either via employment or via contracting). The issue is not technical in nature. Oh and by the way: I've worked closely with engineers working for the US government. I wasn't close to the US government (because I am not an US citizen) but they were. They were "close enough" that they had to work in a SCIF and could only interact with me via phone. The systems they were working on... Those systems had their own private CA (among other things). It's feasible. It's not a technical issue. If it's not done then it's an organizational issue.

tetha 1y ago

Aye. Between Lets Encrypt and compatible vendors, and e.g. Vault for internal CA-chains, /issuing/ certs in a secure, controlled and audited way (at least for the internal ones) is indeed a solved issue. We do have a lot of internal PKI chains running through vault and most of them are pushing 72 hour TTLs. If you can do that, do that. It's great. That being said, deploying and loading these certs is a fun adventure, even in somewhat modern software. We're discovering a surprising amount of fairly modern software which handles certs in stupid ways. For example, if I recall right, NodeJS applications tend to load certificates to secure a connection to PostgreSQL into memory and never bother to reload. libpq on the other hand loads them on connection startup. Even Spring wasn't able to properly reload certificates 3-4 years ago and this was only added in recent versions with dynamic certificates - a colleague patched this in back in the day. Our loadbalancers folded the ticket of "Hey, reload these three certs" into the issue of reloading the entire configuration, including opening, closing ports, handing TCP connections over transparently, and a billion other topics. Funny enough, if there is enough stuff running, there's not even an agreement on how to store stuff. Some stuff needs PEM-files, some stuff needs PKCS8 stores, some stuff wants 1-2 PKCS12 keyrings with certs and keys in there in separate entries. I even had to patch a PKCS12 handling library because one ruby library needed everything in one entry in a PKCS12 key store and there was no go-code under the sun to find to do this. So there is a sunny place in which PKI is indeed solved. And besides that there is a deep dark hole that goes deeper than it should on earth.