4 ms·

“Cultural” technical issues are so frustrating to me. A certificate is fundamentally just a type of credential, like a password, but for historical reasons they

“Cultural” technical issues are so frustrating to me. A certificate is fundamentally just a type of credential, like a password, but for historical reasons they’re treated like getting citizenship papers. There’s always this ceremony even in scenarios where it makes zero sense — such as internal-use certificates used for a gRPC API server behind a load balancer. Why - for the love of God why - can’t I just obtain a cert like this directly out of a secret store such as an Azure Key Vault!? These things are already full hardware security modules (HSMs) with all of the capabilities required to run just about anything short of a public Root CA and maybe even that too. But no. NO! Script it yourself. Make a root cert, “upload” it, make a cert, sign it, upload it, link it, renew it, re-configure it, and on and on. Oh… you wanted a CRL too? A 1kb file? Ha-ha! No. Make it and host it yourself!! It’s absurd. So many services depend on a simple CA->cert setup: VPNs, API auth, load balancer back-ends, clusters, etc… But my mark my words: no cloud will have a trivial turnkey solution this decade. This is because running a CA is culturally accepted to be a hard problem. It is! If you’re DigiCert. It isn’t if you’re building a three-server internal use cluster. But the problem is hard, you see? That accepted fact! Everyone knows it! Ceremony is required. We can’t just hand your sever a 1kb credential file! That would be… unconventional! It’s just not the way things are done, so stop asking.

stanleydrew 3y ago
This is a decent rant, and I mostly share your frustration. But At least GCP and AWS have certificate authority products which essentially do work the way you want them to: https://cloud.google.com/certificate-authority-service https://cloud.google.com/certificate-authority-service https://aws.amazon.com/private-ca/ https://aws.amazon.com/private-ca/ Azure may well have one too, I just don't use their service.
- firesteelrain 3y ago
  Azure will let you tap into a HSM too
- jiggawatts 3y ago
  You just answered a rant about too much ceremony by linking to an entire service that costs a staggering USD 400 / month. That’s the cost of an FTE in like half the world… to issue what are essentially “fat passwords”. Imagine with a straight face trying to sell someone a “password generator service” when in reality that’s just a one line script snippet. The disconnect between the physical reality on the ground and how it is treated by industry is just absurd. Ref: https://aws.amazon.com/private-ca/pricing/ https://aws.amazon.com/private-ca/pricing/ Getting a signed cert issued should be a one-liner script referencing an AWS KMS or an Azure KeyVault!
gorkish 3y ago
It’s about damn time that we should be able to get intermediate signing certs that are domain limited from a public ca and issue out own “real” certs. This is fully supported in the standards. Anyone offering this product affordably yet?
- diarrhea 3y ago
  Interesting idea. What use cases would this make easier or even possible? At first, it sounds like more work (you’re now your own CA but without full freedom of a truly self hosted one).
  - Nextgrid 3y ago
    The main advantage is that this CA and all downstream certificates would be globally-trusted (limited to the domain), which is not the case for a custom CA. Security-wise it shouldn't be any worse than wildcard certificates which are already a thing. It would actually improve things, because the user can now issue downstream certificates much more granularly without having to interact with the root CA (so you can issue fully offline for an intranet, or issue extremely-short-lived certificates that would otherwise run afoul of root CA's rate-limits).
    - jiggawatts 3y ago
      It would improve things for everybody but the certificate authorities. They're selling something with a marginal cost of zero for $50 each. A wildcard certificate costs more not because it is materially different or "harder" to issue, but because it replaces many $50 certificates. Thus, it "must" cost more, or everybody would just use wildcard certificates everywhere and reduce profits at the large public CAs. It is actually possible to get a domain-specific CA like the one you're thinking of. I saw one at a large department of education. It allowed unlimited issuance of certificates such as HTTPS, mail-signing, document-signing, and some other types that could be locked to a DNS domain. However, there was still a cost per certificate and the up-front cost was huge, something like $100K.
diarrhea 3y ago
A password is a secret. A certificate is fully public (it’s useless if not). I don’t see how they’re similar in that dimension. Or how certificates are possible without a third party (a core reason for all the ceremony). Passwords get away with not needing a third party involved because they’re a prearranged process. Certificates aren’t, I need to be able to hit any website at any point, for the first time.
- jiggawatts 3y ago
  The key phrase here is internal use. Think authentication between two micro services, not HTTPS for someblog.com. The third party in this case is just a file: the internal CA key. Try to think of this in the following terms: the issued certs are just signed tokens, nothing more. They’re also conveniently a public-private key pair, but that’s not the point. The point is that if you used a 1kb random password for service to service auth and someone tried to sell you a $400/mo service to generate them, you’d laugh in their face and then have security escort them from the building. Sprinkle a tiny bit of cryptography on top and suddenly people think $4,800 annually is a bargain for 1kb passwords. I can buy a decent used car for that kind of money.
  - matthewaveryusa 3y ago
    PKI can provide non-repudiation while signed tokens and API keys cannot. There's a big difference transmitting a bearer token vs establishing a TLS connection.
    - jiggawatts 3y ago
      I get that you don't get it... that's my point. A public-private key pair is clever cryptography, yes, but INTERNALLY within a network they're not Magic(tm) that requires a multi-billion dollar market cap company to issue with some Indian call centre verifying my identity papers. The same cryptographic algorithm has two wildly different uses: one that is $0.0000001 in value, and one that requires a third party organisation that needs to pay their employees and can maybe justify asking for tens of dollars. (Narrator: Let's Encrypt showed that they can't justify this either.) People conflate the two and then try to charge $50 for the $0.0000001 use-case, which is a markup of five million percent. That's what's upsetting. It's just so absurd, and people blink slowly and then start suggesting $40 options as-if that discount somehow makes it okay. Or they start talking about "all the things you get" for that $50, when it just doesn't apply. There should be a trivial set of commands along the lines of: New-AzKeyVaultRootCertificate -VaultName 'xyzinternal' -Name 'ContosoAPIServiceRoot' New-AzKeyVaultSignedCertificate -VaultName 'xyzinternal' -RootCertificateName 'ContosoAPIServiceRoot' -DnsName 'apisvc1352.internal.cloud' You can emulate the above with a 20-line script now, but it's fiddly, and doesn't cooperate with Bicep deployment templates. Similarly, there ought to be a built-in renewal mechanism (which is JUST 'cron' for the love of God!), but instead requires Azure Functions, layers of complex authorisations, and who knows what else...
mdekkers 3y ago
> Ceremony is required. Ceremony, as in “a well established, rigorously implemented, meticulously documented, and diligently audited process” is required to establish and maintain trust. A CA promises that folks showing up bearing their certificate _are who they say they are_ - this doesn’t mean it is perfect (there are plenty of cases where public CAs went wrong) and some random person can place their trust in that assertion. The whole _point_ of a PKI is trust, and why would this be any different for internal systems?
- jiggawatts 3y ago
  Trust can be established in many different ways. Through paperwork (literally the case for some public CAs!), via automation, or some other means. The difficulty in running a CA at a very large scale (public or private) is that the root certificate revocation becomes prohibitively difficult. For this reason, its security is critical and much ceremony is warranted. A CA doesn’t have to be a giant basket with millions of eggs in it. I have scenarios where I need a CA cert just so it can sign two (2) leaf certificates to make some load balancer happy, or to establish a site to site VPN or whatever. This is just "marking" the issued certs with a common attribute so that a load balancer can trust 'n' servers without having to have 'n' distinct certificates in its trust list. That scenario does not need the same ceremony as DigiCert or the Microsoft kernel driver signing root certificate. It’s just a “shared secret” in a Cloudformation/Terraform/Bicep template. If that CA gets compromised, I can just re-run the template to generate and deploy a new one! No ceremony. Just press play. Not every gets married like Beyonce at an exotic island destination with 500 rich & famous guests. Some people just go down to the local marriage registrar office, get the paperwork done, and go home.
  - mdekkers 3y ago
    > The difficulty in running a CA at a very large scale (public or private) is that the root certificate revocation becomes prohibitively difficult. For this reason, its security is critical and much ceremony is warranted. Unless the end-to-end, from the CA to the final certs, are managed by yourself or your immediate team, the scale would be irrelevant, and some ceremony would be required, as would be some processes that are internal to the team. My personal approach to avoid much of this is to use really short-lived certs to avoid the revocation theatre, but that doesn’t avoid the need to ensure that the CA is fundamentally secure. > If that CA gets compromised, I can just re-run the template to generate and deploy a new one! No ceremony. Just press play. Indeed! That, and re-distribute and implement trust for your CA Chain to all machines that need it. Your examples work fine (and I agree with) for small-scale use-cases, but if you run PKI for anything except yourself, you have to take care of the P for the KI. Look, I am no fan of the rent-seeking bullshit the big public PKI vendors make us go through, and the incredibly high cost per certificate model. Charge me a reasonable fee for verification, and charge me a reasonable fee to set up my certificate vending endpoint, and be done. The same goes for internal use-cases. But the fact remains that verification and process are required. Fortunately, ACME for internal use and sane APIs and workflows such as Vault or smallstep provide make things very easy and cost effective. As for $400 per month for Azure certificates? Run your own instance, with your own (supposed) management processes, and see how it tallies up…

stanleydrew 3y ago

This is a decent rant, and I mostly share your frustration. But At least GCP and AWS have certificate authority products which essentially do work the way you want them to: https://cloud.google.com/certificate-authority-service https://cloud.google.com/certificate-authority-service https://aws.amazon.com/private-ca/ https://aws.amazon.com/private-ca/ Azure may well have one too, I just don't use their service.

firesteelrain 3y ago
Azure will let you tap into a HSM too
jiggawatts 3y ago
You just answered a rant about too much ceremony by linking to an entire service that costs a staggering USD 400 / month. That’s the cost of an FTE in like half the world… to issue what are essentially “fat passwords”. Imagine with a straight face trying to sell someone a “password generator service” when in reality that’s just a one line script snippet. The disconnect between the physical reality on the ground and how it is treated by industry is just absurd. Ref: https://aws.amazon.com/private-ca/pricing/ https://aws.amazon.com/private-ca/pricing/ Getting a signed cert issued should be a one-liner script referencing an AWS KMS or an Azure KeyVault!

gorkish 3y ago

It’s about damn time that we should be able to get intermediate signing certs that are domain limited from a public ca and issue out own “real” certs. This is fully supported in the standards. Anyone offering this product affordably yet?

diarrhea 3y ago
Interesting idea. What use cases would this make easier or even possible? At first, it sounds like more work (you’re now your own CA but without full freedom of a truly self hosted one).
- Nextgrid 3y ago
  The main advantage is that this CA and all downstream certificates would be globally-trusted (limited to the domain), which is not the case for a custom CA. Security-wise it shouldn't be any worse than wildcard certificates which are already a thing. It would actually improve things, because the user can now issue downstream certificates much more granularly without having to interact with the root CA (so you can issue fully offline for an intranet, or issue extremely-short-lived certificates that would otherwise run afoul of root CA's rate-limits).
  - jiggawatts 3y ago
    It would improve things for everybody but the certificate authorities. They're selling something with a marginal cost of zero for $50 each. A wildcard certificate costs more not because it is materially different or "harder" to issue, but because it replaces many $50 certificates. Thus, it "must" cost more, or everybody would just use wildcard certificates everywhere and reduce profits at the large public CAs. It is actually possible to get a domain-specific CA like the one you're thinking of. I saw one at a large department of education. It allowed unlimited issuance of certificates such as HTTPS, mail-signing, document-signing, and some other types that could be locked to a DNS domain. However, there was still a cost per certificate and the up-front cost was huge, something like $100K.

diarrhea 3y ago

A password is a secret. A certificate is fully public (it’s useless if not). I don’t see how they’re similar in that dimension. Or how certificates are possible without a third party (a core reason for all the ceremony). Passwords get away with not needing a third party involved because they’re a prearranged process. Certificates aren’t, I need to be able to hit any website at any point, for the first time.

jiggawatts 3y ago
The key phrase here is internal use. Think authentication between two micro services, not HTTPS for someblog.com. The third party in this case is just a file: the internal CA key. Try to think of this in the following terms: the issued certs are just signed tokens, nothing more. They’re also conveniently a public-private key pair, but that’s not the point. The point is that if you used a 1kb random password for service to service auth and someone tried to sell you a $400/mo service to generate them, you’d laugh in their face and then have security escort them from the building. Sprinkle a tiny bit of cryptography on top and suddenly people think $4,800 annually is a bargain for 1kb passwords. I can buy a decent used car for that kind of money.
- matthewaveryusa 3y ago
  PKI can provide non-repudiation while signed tokens and API keys cannot. There's a big difference transmitting a bearer token vs establishing a TLS connection.
  - jiggawatts 3y ago
    I get that you don't get it... that's my point. A public-private key pair is clever cryptography, yes, but INTERNALLY within a network they're not Magic(tm) that requires a multi-billion dollar market cap company to issue with some Indian call centre verifying my identity papers. The same cryptographic algorithm has two wildly different uses: one that is $0.0000001 in value, and one that requires a third party organisation that needs to pay their employees and can maybe justify asking for tens of dollars. (Narrator: Let's Encrypt showed that they can't justify this either.) People conflate the two and then try to charge $50 for the $0.0000001 use-case, which is a markup of five million percent. That's what's upsetting. It's just so absurd, and people blink slowly and then start suggesting $40 options as-if that discount somehow makes it okay. Or they start talking about "all the things you get" for that $50, when it just doesn't apply. There should be a trivial set of commands along the lines of: New-AzKeyVaultRootCertificate -VaultName 'xyzinternal' -Name 'ContosoAPIServiceRoot' New-AzKeyVaultSignedCertificate -VaultName 'xyzinternal' -RootCertificateName 'ContosoAPIServiceRoot' -DnsName 'apisvc1352.internal.cloud' You can emulate the above with a 20-line script now, but it's fiddly, and doesn't cooperate with Bicep deployment templates. Similarly, there ought to be a built-in renewal mechanism (which is JUST 'cron' for the love of God!), but instead requires Azure Functions, layers of complex authorisations, and who knows what else...

mdekkers 3y ago

> Ceremony is required. Ceremony, as in “a well established, rigorously implemented, meticulously documented, and diligently audited process” is required to establish and maintain trust. A CA promises that folks showing up bearing their certificate _are who they say they are_ - this doesn’t mean it is perfect (there are plenty of cases where public CAs went wrong) and some random person can place their trust in that assertion. The whole _point_ of a PKI is trust, and why would this be any different for internal systems?

jiggawatts 3y ago
Trust can be established in many different ways. Through paperwork (literally the case for some public CAs!), via automation, or some other means. The difficulty in running a CA at a very large scale (public or private) is that the root certificate revocation becomes prohibitively difficult. For this reason, its security is critical and much ceremony is warranted. A CA doesn’t have to be a giant basket with millions of eggs in it. I have scenarios where I need a CA cert just so it can sign two (2) leaf certificates to make some load balancer happy, or to establish a site to site VPN or whatever. This is just "marking" the issued certs with a common attribute so that a load balancer can trust 'n' servers without having to have 'n' distinct certificates in its trust list. That scenario does not need the same ceremony as DigiCert or the Microsoft kernel driver signing root certificate. It’s just a “shared secret” in a Cloudformation/Terraform/Bicep template. If that CA gets compromised, I can just re-run the template to generate and deploy a new one! No ceremony. Just press play. Not every gets married like Beyonce at an exotic island destination with 500 rich & famous guests. Some people just go down to the local marriage registrar office, get the paperwork done, and go home.
- mdekkers 3y ago
  > The difficulty in running a CA at a very large scale (public or private) is that the root certificate revocation becomes prohibitively difficult. For this reason, its security is critical and much ceremony is warranted. Unless the end-to-end, from the CA to the final certs, are managed by yourself or your immediate team, the scale would be irrelevant, and some ceremony would be required, as would be some processes that are internal to the team. My personal approach to avoid much of this is to use really short-lived certs to avoid the revocation theatre, but that doesn’t avoid the need to ensure that the CA is fundamentally secure. > If that CA gets compromised, I can just re-run the template to generate and deploy a new one! No ceremony. Just press play. Indeed! That, and re-distribute and implement trust for your CA Chain to all machines that need it. Your examples work fine (and I agree with) for small-scale use-cases, but if you run PKI for anything except yourself, you have to take care of the P for the KI. Look, I am no fan of the rent-seeking bullshit the big public PKI vendors make us go through, and the incredibly high cost per certificate model. Charge me a reasonable fee for verification, and charge me a reasonable fee to set up my certificate vending endpoint, and be done. The same goes for internal use-cases. But the fact remains that verification and process are required. Fortunately, ACME for internal use and sane APIs and workflows such as Vault or smallstep provide make things very easy and cost effective. As for $400 per month for Azure certificates? Run your own instance, with your own (supposed) management processes, and see how it tallies up…