3 ms·

Don’t bake trust stores into container images. At work we have a single place where trusted certs (commercial CAs plus our company root CA) are configured, and

by jhugo 3y ago

Don’t bake trust stores into container images. At work we have a single place where trusted certs (commercial CAs plus our company root CA) are configured, and then those are mounted in the appropriate format (big PEM file for /etc/ssl, trust store for Java apps, etc) as a volume in deployments in all clusters. This means that trust store updates are not dependent on the OS vendor, they’re consistent between native/Java/etc, and updates don’t require rebuilding container images.

zrail 3y ago
This is essentially the pattern I've settled on for my homelab. I build a container image on a cron that contains all of my certificates. It has a small entrypoint script that copies the certs into a volume, and then that volume is mounted read-only into every other container that needs certs. When certificates rotate the system builds a new image which gets pulled down by watchtower, which then in turn handles dependency management and restarts things as needed.
- parlortricks 3y ago
  do you have any doco for this or links to read over, i like the sound of this but too inexperienced in docker to set it up.
  - zrail 3y ago
    Not right now. I'll write something up when I have a few minutes.
- MereInterest 3y ago
  What’s the purpose of the container image in this setup? It sounds like it exists just to make the certificate directory, so the container feels like overkill when compared to a tarball, or a self-extracting archive.
  - zrail 3y ago
    You're right, it's a little weird. I wrote a short essay about my setup[1] but the tl;dr is that I wanted certificates distributed in the same way every other thing on my machines is distributed. I wanted my homeprod setup to be as hands off as possible while still allowing easy management. Each physical host is running Alpine. During provisioning I install docker, Tailscale, and manually start a "root" container that runs[2] docker compose and then starts a cron daemon. The compose commands include one or more "stack" files and are generated based on a yaml file listing the stacks for each host. Watchtower runs with a 30 second cycle time to keep everything updated, including the root container. Adding or updating services means committing and pushing a change to the root container repo, then CI builds and pushes a new image. Watchtower picks up the new image and restarts the root container, which re-runs Compose which in turn starts, stops, modifies, etc anything that's changed. For certificates, I tried a number of different things but ultimately settled on the method I described earlier. The purpose of the container image is to 1) transport the certificates and install them in the right spot and 2) be updatable automatically with Watchtower. Certificate changes are very similar to the root container, except the git repo self-modifies upon renewals (yes I keep private keys committed to git, it's a homelab, it's really not a big deal). [1]: https://www.petekeen.net/homeprod-management-with-docker https://www.petekeen.net/homeprod-management-with-docker [2]: https://github.com/peterkeen/docker-compose-stack/blob/main/run_compose.sh#L40 https://github.com/peterkeen/docker-compose-stack/blob/main/...
andrewstuart2 3y ago
I've built a small MutatingAdmissionWebhook controller [0] that handles this, via a pod annotation whose value is a secret with `ca.crt` inside, and it uses the (mostly) de facto standard openssl variables to configure the libraries, so that it works across pretty much everything I've tried it with off the shelf. I build a bundle (though I may just move to trust-manager [1]) and replicate it into all namespaces with kubernetes-replicator [2], and then I can annotate any pod with [0] https://github.com/microcumulus/ca-injector https://github.com/microcumulus/ca-injector [1] https://github.com/cert-manager/trust-manager https://github.com/cert-manager/trust-manager [2] https://github.com/mittwald/kubernetes-replicator https://github.com/mittwald/kubernetes-replicator
- andrewstuart2 3y ago
  *...with the name of that secret and SSL works for almost everything out of the box. As I've found things that don't work (nodejs was one), I've just updated that code and it's made life pretty great for anywhere I'm running a CA in vault etc.
tikkabhuna 3y ago
I've been thinking about doing exactly this at my place. I was thinking about Linux and Java. What other formats do we need to worry about? I'm thinking Windows?
- mirekrusin 3y ago
  Probably pfx.
vngzs 3y ago
If you want good integrity controls, treating the CA bundles as secrets (e.g., storing them in k8s secrets or something like Vault) can help there. We don't actually care much about confidentiality controls, but any secret manager worth its salt is going to have integrity controls as well. This gives you easy updates and lets you reuse containers across environments while remaining defensive against attackers adding their own CA to the store.

by jhugo 3y ago

zrail 3y ago

This is essentially the pattern I've settled on for my homelab. I build a container image on a cron that contains all of my certificates. It has a small entrypoint script that copies the certs into a volume, and then that volume is mounted read-only into every other container that needs certs. When certificates rotate the system builds a new image which gets pulled down by watchtower, which then in turn handles dependency management and restarts things as needed.

parlortricks 3y ago
do you have any doco for this or links to read over, i like the sound of this but too inexperienced in docker to set it up.
- zrail 3y ago
  Not right now. I'll write something up when I have a few minutes.
MereInterest 3y ago
What’s the purpose of the container image in this setup? It sounds like it exists just to make the certificate directory, so the container feels like overkill when compared to a tarball, or a self-extracting archive.
- zrail 3y ago
  You're right, it's a little weird. I wrote a short essay about my setup[1] but the tl;dr is that I wanted certificates distributed in the same way every other thing on my machines is distributed. I wanted my homeprod setup to be as hands off as possible while still allowing easy management. Each physical host is running Alpine. During provisioning I install docker, Tailscale, and manually start a "root" container that runs[2] docker compose and then starts a cron daemon. The compose commands include one or more "stack" files and are generated based on a yaml file listing the stacks for each host. Watchtower runs with a 30 second cycle time to keep everything updated, including the root container. Adding or updating services means committing and pushing a change to the root container repo, then CI builds and pushes a new image. Watchtower picks up the new image and restarts the root container, which re-runs Compose which in turn starts, stops, modifies, etc anything that's changed. For certificates, I tried a number of different things but ultimately settled on the method I described earlier. The purpose of the container image is to 1) transport the certificates and install them in the right spot and 2) be updatable automatically with Watchtower. Certificate changes are very similar to the root container, except the git repo self-modifies upon renewals (yes I keep private keys committed to git, it's a homelab, it's really not a big deal). [1]: https://www.petekeen.net/homeprod-management-with-docker https://www.petekeen.net/homeprod-management-with-docker [2]: https://github.com/peterkeen/docker-compose-stack/blob/main/run_compose.sh#L40 https://github.com/peterkeen/docker-compose-stack/blob/main/...

andrewstuart2 3y ago

I've built a small MutatingAdmissionWebhook controller [0] that handles this, via a pod annotation whose value is a secret with `ca.crt` inside, and it uses the (mostly) de facto standard openssl variables to configure the libraries, so that it works across pretty much everything I've tried it with off the shelf. I build a bundle (though I may just move to trust-manager [1]) and replicate it into all namespaces with kubernetes-replicator [2], and then I can annotate any pod with [0] https://github.com/microcumulus/ca-injector https://github.com/microcumulus/ca-injector [1] https://github.com/cert-manager/trust-manager https://github.com/cert-manager/trust-manager [2] https://github.com/mittwald/kubernetes-replicator https://github.com/mittwald/kubernetes-replicator

andrewstuart2 3y ago
*...with the name of that secret and SSL works for almost everything out of the box. As I've found things that don't work (nodejs was one), I've just updated that code and it's made life pretty great for anywhere I'm running a CA in vault etc.

tikkabhuna 3y ago

I've been thinking about doing exactly this at my place. I was thinking about Linux and Java. What other formats do we need to worry about? I'm thinking Windows?

mirekrusin 3y ago
Probably pfx.

vngzs 3y ago

If you want good integrity controls, treating the CA bundles as secrets (e.g., storing them in k8s secrets or something like Vault) can help there. We don't actually care much about confidentiality controls, but any secret manager worth its salt is going to have integrity controls as well. This gives you easy updates and lets you reuse containers across environments while remaining defensive against attackers adding their own CA to the store.