7 ms·

What if your Pods need to trust self-signed certificates?

40 points by alexellisuk 3y ago

nijave 3y ago
You can also do sidecar proxy pattern where the the sidecar handles TLS termination (incoming or outgoing). That's basically service mesh light. You can also use MutatingWebhooks to inject initContainers or side cars to achieve that pattern. MutatingWebhooks increase the runtime complexity since you're jamming more config in but they can handle vendor/3rd party software. Depending on the CNI, insecure TLS flags might not actually be that bad. There are some CNIs floating around that can handle encryption. In addition, software defined networks like AWS might not offer encryption but they do guarantee messages are authenticated so you can trust the source IP
- vbezhenar 3y ago
  IMO that's ideal situation: when service does not encode any routing information and just talks to another service via fake DNS name. And actual routing is performed by infrastructure tools which can implement TLS, collect metrics, collect full HTTP logs if asked, implement retries, caching and so on. This is about outgoing connections and incoming connections. It could also save on container size theoretically. Implementing TLS is a daunting task and requires many kilobytes of code. Implementing HTTP is easier task. So you can strip TLS implementation and achieve lighter containers.
  - adql 3y ago
    You save shit all if every container comes with sidecar doing the actual TLS, it's extra component to debug and extra small latency added.
    - pojzon 3y ago
      Or you can leverage ebpf like cillium.
    - nijave 3y ago
      In practice it's not that much. I think our Envoy sidecars take, maybe, 8mb of RAM when the apps are usually 500-1000MB. They usually add afaik <1ms latency. In addition, you can implement client side load balancing through Envoy so you can hit your services directly without intermediary load balancers. It adds complexity with the side car but usually you get standardized RED metrics regardless of if the service implemented them. You can also have the sidecar do connection pooling and maintain persistent or long lived TCP connections which ends up speeding things up without doing that at the app level.
- adql 3y ago
  > You can also do sidecar proxy pattern where the the sidecar handles TLS termination (incoming or outgoing). That's basically service mesh light. That's a horrid idea, why would you do that ?
  - otterley 3y ago
    It’s an extremely common pattern. Decoupling TLS allows the developer to focus on the application and it makes for far easier development and testing.
jhugo 3y ago
Don’t bake trust stores into container images. At work we have a single place where trusted certs (commercial CAs plus our company root CA) are configured, and then those are mounted in the appropriate format (big PEM file for /etc/ssl, trust store for Java apps, etc) as a volume in deployments in all clusters. This means that trust store updates are not dependent on the OS vendor, they’re consistent between native/Java/etc, and updates don’t require rebuilding container images.
- zrail 3y ago
  This is essentially the pattern I've settled on for my homelab. I build a container image on a cron that contains all of my certificates. It has a small entrypoint script that copies the certs into a volume, and then that volume is mounted read-only into every other container that needs certs. When certificates rotate the system builds a new image which gets pulled down by watchtower, which then in turn handles dependency management and restarts things as needed.
  - parlortricks 3y ago
    do you have any doco for this or links to read over, i like the sound of this but too inexperienced in docker to set it up.
    - zrail 3y ago
      Not right now. I'll write something up when I have a few minutes.
  - MereInterest 3y ago
    What’s the purpose of the container image in this setup? It sounds like it exists just to make the certificate directory, so the container feels like overkill when compared to a tarball, or a self-extracting archive.
    - zrail 3y ago
      You're right, it's a little weird. I wrote a short essay about my setup[1] but the tl;dr is that I wanted certificates distributed in the same way every other thing on my machines is distributed. I wanted my homeprod setup to be as hands off as possible while still allowing easy management. Each physical host is running Alpine. During provisioning I install docker, Tailscale, and manually start a "root" container that runs[2] docker compose and then starts a cron daemon. The compose commands include one or more "stack" files and are generated based on a yaml file listing the stacks for each host. Watchtower runs with a 30 second cycle time to keep everything updated, including the root container. Adding or updating services means committing and pushing a change to the root container repo, then CI builds and pushes a new image. Watchtower picks up the new image and restarts the root container, which re-runs Compose which in turn starts, stops, modifies, etc anything that's changed. For certificates, I tried a number of different things but ultimately settled on the method I described earlier. The purpose of the container image is to 1) transport the certificates and install them in the right spot and 2) be updatable automatically with Watchtower. Certificate changes are very similar to the root container, except the git repo self-modifies upon renewals (yes I keep private keys committed to git, it's a homelab, it's really not a big deal). [1]: https://www.petekeen.net/homeprod-management-with-docker https://www.petekeen.net/homeprod-management-with-docker [2]: https://github.com/peterkeen/docker-compose-stack/blob/main/run_compose.sh#L40 https://github.com/peterkeen/docker-compose-stack/blob/main/...
adql 3y ago
That means you designed your architecture incorrectly; get an internal CA and automate it, then pod can just have long term root CA embedded in it.
- SOLAR_FIELDS 3y ago
  That’s kind of the same problem as mentioned here though, you still have to mount that root CA in all your pods. An internal CA is also, while achievable with cert-manager these days, not exactly trivial to set up.
  - diarrhea 3y ago
    I’ve had good success with step-ca. It was easy enough to be called trivial actually!
    - SOLAR_FIELDS 3y ago
      I actually used smallstep when I rolled my own CA as well. They have nice tooling.The most annoying part of the whole ordeal is dealing with cert formats IMO
  - deathanatos 3y ago
    > An internal CA is also, while achievable with cert-manager these days, not exactly trivial to set up. …it's not exactly hard, though? Install cert-manager, and it's all of 3 resources? 1 self-signing Issuer, one Certificate (the CA cert), and then an Issuer that issues certs under that CA cert. Then any service that wants an internal cert adds a Certificate issued by that Issuer.
    - SOLAR_FIELDS 3y ago
      Simple if you’re already familiar with cert-manager. The annoying part is getting it mounted in every pod that needs it and needs to consume it, which is the point the original article made. Node has its own way of doing it, Java has its own way of doing it, etc etc and they all have their own little ideosyncracies that you have to worry about.
flerchin 3y ago
In the age of cheap certs, it's not clear to me why we have these enterprises with self-signed certificates. It definitely is a thing, but it seems to be tech debt that we should fix.
- adql 3y ago
  Having to communicate with outside is kinda overkill if you just want to have container A talking to container B. But the solution is internal CA, not self signed certs that defeat near-entire point of encrypting communication.
  - flerchin 3y ago
    Yes an internal CA is better than a self-signed-cert. However, I've come across, quite often, where an internal CA causes problems that are only at that enterprise. Using the established public CAs is generally more resilient, and there's _no reason to not_.
    - czx4f4bd 3y ago
      Unfortunately, enterprises are intensely risk averse. I worked at a place where we could only generate proper certs by manually submitting a ticket to IT and waiting probably days to get it back, giving us zero hope of applying meaningful automation. Getting certs from a proper CA was absolutely forbidden, despite our lobbying, so in a lot of cases self-signed certs were the only option if we wanted to automate.
      - geraldwhen 3y ago
        Where I work it’s months, and fights, and maybe even several meetings. And that’s only if you fill out the correct form the exact right way. Any slight deviation and your ticket is closed weeks after you opened it with the field you got wrong.
        mirekrusin 3y ago
sgtcodfish 3y ago
Plug (but it's open source and free - and mentioned in the article!): We've been trying to address this in Kubernetes with trust-manager. [1] Trust bundles need to be a runtime concern and they need to support trusting both the old a new version of a cert to safely allow for rotation. It's pretty simple but it seems to work well! trust-manager also supports pulling in the Mozilla trust bundle which most Linux distros (and therefore most containers) use! Handling trust of private [2] certificates is done poorly generally across many orgs and platforms, not just Kubernetes. There are lots of ways of shooting yourself in the foot - particularly when it comes to rotating CA certificates. I think there's a lot of space here for new solutions here! [1] https://cert-manager.io/docs/projects/trust-manager/ https://cert-manager.io/docs/projects/trust-manager/ [2] I try to avoid "self-signed" in this use case because its literal meaning is that the certificate signs itself using its own key, which is what root certificates do. The Let's Encrypt ISRG X1 root certificate is self-signed but it's definitely not what I'd call a 'private CA'; see https://letsencrypt.org/certificates/ https://letsencrypt.org/certificates/
- rad_gruchalski 3y ago
  Thanks for the tip re trust manager and kudos for cert-manager. Works really well.
8organicbits 3y ago
I've been working on a different approach to try to make it as easy as possible to use certificates from public CAs for private networks. I think it's really hard to get all your trust stores updated with private CA certs; there's just too many of them. I've encountered devices that weren't properly configured at every company I've worked at. Any time you've got a BYOD policy, a variety of mobile devices, or engineers are spinning up their own VMs (which don't copy the trust store of the host) you're going to see certificate warning messages. Back when every computer ran Windows and was chained to the desk it was easier for IT departments to update trust stores, but the world has changed and it's a losing battle. I think the better approach is to just use certificates from public CAs on private networks. There's some big issues to address: how does the public CA verify internal servers which don't permit external connections, is leaking internal subdomain names a security issue? But I think these are solvable. getlocalcert.net[1] is my project to simplify the process. Currently you can register a subdomain and use the ACME DNS-01 protocol via my site to issue certificates from from providers like Let's Encrypt. All you need to issue a certificate is a getlocalcert API key and the ability to connect out-bound to the Let's Encrypt API and the getlocalcert API. It supports a couple ACME clients and it should support cert-manager, mentioned in the article, but I haven't had a chance to test and properly document it[2]. I'm not doing anything that other subdomain registrars or DNS providers can't do, but I'm trying to address the public-CA-on-private-server niche as best as possible. Longer term I'm looking to add bring-your-own-domain support and other improvements. [1] https://www.getlocalcert.net/ https://www.getlocalcert.net/ [2] https://docs.getlocalcert.net/acme-clients/cert-manager/ https://docs.getlocalcert.net/acme-clients/cert-manager/

40 points by alexellisuk 3y ago

You can also do sidecar proxy pattern where the the sidecar handles TLS termination (incoming or outgoing). That's basically service mesh light. You can also use MutatingWebhooks to inject initContainers or side cars to achieve that pattern. MutatingWebhooks increase the runtime complexity since you're jamming more config in but they can handle vendor/3rd party software. Depending on the CNI, insecure TLS flags might not actually be that bad. There are some CNIs floating around that can handle encryption. In addition, software defined networks like AWS might not offer encryption but they do guarantee messages are authenticated so you can trust the source IP

vbezhenar 3y ago
IMO that's ideal situation: when service does not encode any routing information and just talks to another service via fake DNS name. And actual routing is performed by infrastructure tools which can implement TLS, collect metrics, collect full HTTP logs if asked, implement retries, caching and so on. This is about outgoing connections and incoming connections. It could also save on container size theoretically. Implementing TLS is a daunting task and requires many kilobytes of code. Implementing HTTP is easier task. So you can strip TLS implementation and achieve lighter containers.
- adql 3y ago
  You save shit all if every container comes with sidecar doing the actual TLS, it's extra component to debug and extra small latency added.
  - pojzon 3y ago
    Or you can leverage ebpf like cillium.
  - nijave 3y ago
    In practice it's not that much. I think our Envoy sidecars take, maybe, 8mb of RAM when the apps are usually 500-1000MB. They usually add afaik <1ms latency. In addition, you can implement client side load balancing through Envoy so you can hit your services directly without intermediary load balancers. It adds complexity with the side car but usually you get standardized RED metrics regardless of if the service implemented them. You can also have the sidecar do connection pooling and maintain persistent or long lived TCP connections which ends up speeding things up without doing that at the app level.
adql 3y ago
> You can also do sidecar proxy pattern where the the sidecar handles TLS termination (incoming or outgoing). That's basically service mesh light. That's a horrid idea, why would you do that ?
- otterley 3y ago
  It’s an extremely common pattern. Decoupling TLS allows the developer to focus on the application and it makes for far easier development and testing.

jhugo 3y ago

Don’t bake trust stores into container images. At work we have a single place where trusted certs (commercial CAs plus our company root CA) are configured, and then those are mounted in the appropriate format (big PEM file for /etc/ssl, trust store for Java apps, etc) as a volume in deployments in all clusters. This means that trust store updates are not dependent on the OS vendor, they’re consistent between native/Java/etc, and updates don’t require rebuilding container images.

zrail 3y ago
This is essentially the pattern I've settled on for my homelab. I build a container image on a cron that contains all of my certificates. It has a small entrypoint script that copies the certs into a volume, and then that volume is mounted read-only into every other container that needs certs. When certificates rotate the system builds a new image which gets pulled down by watchtower, which then in turn handles dependency management and restarts things as needed.
- parlortricks 3y ago
  do you have any doco for this or links to read over, i like the sound of this but too inexperienced in docker to set it up.
  - zrail 3y ago
    Not right now. I'll write something up when I have a few minutes.
- MereInterest 3y ago
  What’s the purpose of the container image in this setup? It sounds like it exists just to make the certificate directory, so the container feels like overkill when compared to a tarball, or a self-extracting archive.
  - zrail 3y ago
    You're right, it's a little weird. I wrote a short essay about my setup[1] but the tl;dr is that I wanted certificates distributed in the same way every other thing on my machines is distributed. I wanted my homeprod setup to be as hands off as possible while still allowing easy management. Each physical host is running Alpine. During provisioning I install docker, Tailscale, and manually start a "root" container that runs[2] docker compose and then starts a cron daemon. The compose commands include one or more "stack" files and are generated based on a yaml file listing the stacks for each host. Watchtower runs with a 30 second cycle time to keep everything updated, including the root container. Adding or updating services means committing and pushing a change to the root container repo, then CI builds and pushes a new image. Watchtower picks up the new image and restarts the root container, which re-runs Compose which in turn starts, stops, modifies, etc anything that's changed. For certificates, I tried a number of different things but ultimately settled on the method I described earlier. The purpose of the container image is to 1) transport the certificates and install them in the right spot and 2) be updatable automatically with Watchtower. Certificate changes are very similar to the root container, except the git repo self-modifies upon renewals (yes I keep private keys committed to git, it's a homelab, it's really not a big deal). [1]: https://www.petekeen.net/homeprod-management-with-docker https://www.petekeen.net/homeprod-management-with-docker [2]: https://github.com/peterkeen/docker-compose-stack/blob/main/run_compose.sh#L40 https://github.com/peterkeen/docker-compose-stack/blob/main/...

adql 3y ago

That means you designed your architecture incorrectly; get an internal CA and automate it, then pod can just have long term root CA embedded in it.

SOLAR_FIELDS 3y ago
That’s kind of the same problem as mentioned here though, you still have to mount that root CA in all your pods. An internal CA is also, while achievable with cert-manager these days, not exactly trivial to set up.
- diarrhea 3y ago
  I’ve had good success with step-ca. It was easy enough to be called trivial actually!
  - SOLAR_FIELDS 3y ago
    I actually used smallstep when I rolled my own CA as well. They have nice tooling.The most annoying part of the whole ordeal is dealing with cert formats IMO
- deathanatos 3y ago
  > An internal CA is also, while achievable with cert-manager these days, not exactly trivial to set up. …it's not exactly hard, though? Install cert-manager, and it's all of 3 resources? 1 self-signing Issuer, one Certificate (the CA cert), and then an Issuer that issues certs under that CA cert. Then any service that wants an internal cert adds a Certificate issued by that Issuer.
  - SOLAR_FIELDS 3y ago
    Simple if you’re already familiar with cert-manager. The annoying part is getting it mounted in every pod that needs it and needs to consume it, which is the point the original article made. Node has its own way of doing it, Java has its own way of doing it, etc etc and they all have their own little ideosyncracies that you have to worry about.

flerchin 3y ago

In the age of cheap certs, it's not clear to me why we have these enterprises with self-signed certificates. It definitely is a thing, but it seems to be tech debt that we should fix.

adql 3y ago
Having to communicate with outside is kinda overkill if you just want to have container A talking to container B. But the solution is internal CA, not self signed certs that defeat near-entire point of encrypting communication.
- flerchin 3y ago
  Yes an internal CA is better than a self-signed-cert. However, I've come across, quite often, where an internal CA causes problems that are only at that enterprise. Using the established public CAs is generally more resilient, and there's _no reason to not_.
  - czx4f4bd 3y ago
    Unfortunately, enterprises are intensely risk averse. I worked at a place where we could only generate proper certs by manually submitting a ticket to IT and waiting probably days to get it back, giving us zero hope of applying meaningful automation. Getting certs from a proper CA was absolutely forbidden, despite our lobbying, so in a lot of cases self-signed certs were the only option if we wanted to automate.
    - geraldwhen 3y ago
      Where I work it’s months, and fights, and maybe even several meetings. And that’s only if you fill out the correct form the exact right way. Any slight deviation and your ticket is closed weeks after you opened it with the field you got wrong.
      - mirekrusin 3y ago

sgtcodfish 3y ago

Plug (but it's open source and free - and mentioned in the article!): We've been trying to address this in Kubernetes with trust-manager. [1] Trust bundles need to be a runtime concern and they need to support trusting both the old a new version of a cert to safely allow for rotation. It's pretty simple but it seems to work well! trust-manager also supports pulling in the Mozilla trust bundle which most Linux distros (and therefore most containers) use! Handling trust of private [2] certificates is done poorly generally across many orgs and platforms, not just Kubernetes. There are lots of ways of shooting yourself in the foot - particularly when it comes to rotating CA certificates. I think there's a lot of space here for new solutions here! [1] https://cert-manager.io/docs/projects/trust-manager/ https://cert-manager.io/docs/projects/trust-manager/ [2] I try to avoid "self-signed" in this use case because its literal meaning is that the certificate signs itself using its own key, which is what root certificates do. The Let's Encrypt ISRG X1 root certificate is self-signed but it's definitely not what I'd call a 'private CA'; see https://letsencrypt.org/certificates/ https://letsencrypt.org/certificates/

rad_gruchalski 3y ago
Thanks for the tip re trust manager and kudos for cert-manager. Works really well.

8organicbits 3y ago

I've been working on a different approach to try to make it as easy as possible to use certificates from public CAs for private networks. I think it's really hard to get all your trust stores updated with private CA certs; there's just too many of them. I've encountered devices that weren't properly configured at every company I've worked at. Any time you've got a BYOD policy, a variety of mobile devices, or engineers are spinning up their own VMs (which don't copy the trust store of the host) you're going to see certificate warning messages. Back when every computer ran Windows and was chained to the desk it was easier for IT departments to update trust stores, but the world has changed and it's a losing battle. I think the better approach is to just use certificates from public CAs on private networks. There's some big issues to address: how does the public CA verify internal servers which don't permit external connections, is leaking internal subdomain names a security issue? But I think these are solvable. getlocalcert.net[1] is my project to simplify the process. Currently you can register a subdomain and use the ACME DNS-01 protocol via my site to issue certificates from from providers like Let's Encrypt. All you need to issue a certificate is a getlocalcert API key and the ability to connect out-bound to the Let's Encrypt API and the getlocalcert API. It supports a couple ACME clients and it should support cert-manager, mentioned in the article, but I haven't had a chance to test and properly document it[2]. I'm not doing anything that other subdomain registrars or DNS providers can't do, but I'm trying to address the public-CA-on-private-server niche as best as possible. Longer term I'm looking to add bring-your-own-domain support and other improvements. [1] https://www.getlocalcert.net/ https://www.getlocalcert.net/ [2] https://docs.getlocalcert.net/acme-clients/cert-manager/ https://docs.getlocalcert.net/acme-clients/cert-manager/