4 ms·

I work on Flux, and for the reasons you mentioned we advocate for a strict separation between CI and CD, where CI is responsible for building and testing images

by yebyen 5y ago

I work on Flux, and for the reasons you mentioned we advocate for a strict separation between CI and CD, where CI is responsible for building and testing images and pushing out releases, and CD is responsible for production. There is no crossover between these roles, and this makes it very secure. You use a policy CRD called ImagePolicy to declare what versions should be matched and deployed automatically. You can have them deployed directly to production, or if it makes you more comfortable with your contractors, to an "auto-PR" branch which simply queues them up for a release engineer with the required juice to approve and merge the changes to prod. There is no deploy job in CI, since the deployment in prod is whatever happens to be on the main branch in the config repo. This is a protected branch which only releng can merge changes into, usually only when the release checks are satisfied. Flux (which scans any image repos) updates those manifests when there is a release, either directly or via pull request. This strategy I think would be able to address your Security concerns regarding contractors, by reducing the responsibility of your CI system to only CI, and not to handle CD anymore. This is the subject of my talk at KubeCon[1], although you might not be able to get that from the title, (I'm presenting this topic for a Jenkins audience, but the focus is on Flux and how Flux works) so I'm hoping it should be applicable to a broader audience, certainly inclusive of GitHub users :) [1]: https://kccncna2021.sched.com/event/lV0V/gitopsjenkins-ci-with-declarative-everything-kingdon-barrett-weaveworks?iframe=no https://kccncna2021.sched.com/event/lV0V/gitopsjenkins-ci-wi...

lmeyerov 5y ago
Yep -- and it goes deeper, even for CI. The CI/CD secrets vulnerability is just the most burning example. Ex: Network segmentation for CI. We only expect CI to communicate with dockerhub, conda/pypi, and the CI service (ex: azure). Something similar for CD. That _should_ be settable via GHA, but isn't. In a world of weird npm/python scripts and github action marketplace... scary we can't. Self-hosted runners can in theory do this via custom network policies, but it's a PITA for something ~everyone should be doing out-of-the-box. Edit: HN won't let me respond to the below. Imagine something like IP theft, wanting to mine our repo for everything labeled "security", dig into our version #'s for viable CVEs, or force-push some git commits (incl. history rewriting). Defense-in-depth says we shouldn't make that unnecessarily easy for anyone who plants a backdoor that runs during CI's `npm install` / `RUN xyz` phase. Network + RBAC are basically table stakes for almost everyone building enterprise software, so GHA doesn't have to reinvent the wheel here, just do it + make it friendly.
- yebyen 5y ago
  I think those problems solve themselves when you can say "CI doesn't need to communicate with production at all anymore" – the Flux controllers live on the cluster, or on one cluster which is in position to communicate with others on the network in case of multi-cluster. They do not have any outward facing interfaces (except one), they live on the Kube API. (The Flux API is the CRD within the Kubernetes API.) The reason for CI to communicate with production is to handle deployments, right? So take away that responsibility, and now you can safely isolate GitHub Server and shunt it into a network that only has access to outside artifacts that are necessary, push access to an image registry, and that's it. Deployment status notifications can go from Flux to CI, so you can read about your deployments via commit status messages, but the communication is one-way. Changes are always pulled into the cluster and never pushed. You can set up webhooks in both directions, of course, for faster round trips (it makes a big difference for devs when iterating to not have to do a context switch and manually trigger a deploy, or to wait seconds not minutes and get immediate feedback after git push) but many will also opt not to do this in order to further confine and constrain to limit attack surfaces, it works just as well so long as they can live with some 5-10 minute waits for polling to do the job. Edit: heaven forbid we have a conversation, Hacker News won't allow fast replies :D thanks for giving examples
orlovs 5y ago
+100000. any more or less serious case deployment must be decoupled from build. It brings some challenges, but deploy and build are two different cycles what you cant put on the same assembly area. It can be triggered, but it should not be same pipelin

by yebyen 5y ago

lmeyerov 5y ago

Yep -- and it goes deeper, even for CI. The CI/CD secrets vulnerability is just the most burning example. Ex: Network segmentation for CI. We only expect CI to communicate with dockerhub, conda/pypi, and the CI service (ex: azure). Something similar for CD. That _should_ be settable via GHA, but isn't. In a world of weird npm/python scripts and github action marketplace... scary we can't. Self-hosted runners can in theory do this via custom network policies, but it's a PITA for something ~everyone should be doing out-of-the-box. Edit: HN won't let me respond to the below. Imagine something like IP theft, wanting to mine our repo for everything labeled "security", dig into our version #'s for viable CVEs, or force-push some git commits (incl. history rewriting). Defense-in-depth says we shouldn't make that unnecessarily easy for anyone who plants a backdoor that runs during CI's `npm install` / `RUN xyz` phase. Network + RBAC are basically table stakes for almost everyone building enterprise software, so GHA doesn't have to reinvent the wheel here, just do it + make it friendly.

yebyen 5y ago
I think those problems solve themselves when you can say "CI doesn't need to communicate with production at all anymore" – the Flux controllers live on the cluster, or on one cluster which is in position to communicate with others on the network in case of multi-cluster. They do not have any outward facing interfaces (except one), they live on the Kube API. (The Flux API is the CRD within the Kubernetes API.) The reason for CI to communicate with production is to handle deployments, right? So take away that responsibility, and now you can safely isolate GitHub Server and shunt it into a network that only has access to outside artifacts that are necessary, push access to an image registry, and that's it. Deployment status notifications can go from Flux to CI, so you can read about your deployments via commit status messages, but the communication is one-way. Changes are always pulled into the cluster and never pushed. You can set up webhooks in both directions, of course, for faster round trips (it makes a big difference for devs when iterating to not have to do a context switch and manually trigger a deploy, or to wait seconds not minutes and get immediate feedback after git push) but many will also opt not to do this in order to further confine and constrain to limit attack surfaces, it works just as well so long as they can live with some 5-10 minute waits for polling to do the job. Edit: heaven forbid we have a conversation, Hacker News won't allow fast replies :D thanks for giving examples

orlovs 5y ago

+100000. any more or less serious case deployment must be decoupled from build. It brings some challenges, but deploy and build are two different cycles what you cant put on the same assembly area. It can be triggered, but it should not be same pipelin