4 ms·

We've been loving GHA: CI/CD-as-code, pull requests capture deploy history (CI, stage/restage, deploy), and labeling a PR with 'release' is enough to generate o

We've been loving GHA: CI/CD-as-code, pull requests capture deploy history (CI, stage/restage, deploy), and labeling a PR with 'release' is enough to generate our on-prem + multi-cloud artifacts. Our main gotchas are roughly: - GH-hosted runners have too little RAM/HD for big docker software. They push you to self-hosted runners for that, which is fine in theory, but GHA/Azure doesn't actually support serverless runners, so that falls flat in practice. We don't want to be turning machines on/off, that's GHA's job. We experimented with GHA -> Packer -> Azure for serverless, but it was slow and Packer frequently leaves zombie machines, so we went back to tweaking the low-RAM runners provided by our enterprise plan. - Security: We want contactors etc. to be able to run limited GHA CI jobs and use that quota, but not higher-trust GHA CD ones. This is tricky at a configuration level. Ex: It seems like we'd need to funny things like main repo for CI w/ CI secrets, and a separate repo for CD w/ CD secrets, and only give untrusted folks access to the CD-cred repo. We've thought of other possibilities as well, but in general, it's frightening. - Big Docker images: We do spend more time than I'd like messing with optimizing Docker caching as GPU containers are embarrassingly big (we use dockerhub vs github's due to sizes/pricing/etc), think both multi-stage containers + multi-step jobs (monorepo/microservices). I think they're in a good position to speed that up! I'm optimistic about these, but tricky to align with MS/GH PM personal team priorities :)

yebyen 5y ago
I work on Flux, and for the reasons you mentioned we advocate for a strict separation between CI and CD, where CI is responsible for building and testing images and pushing out releases, and CD is responsible for production. There is no crossover between these roles, and this makes it very secure. You use a policy CRD called ImagePolicy to declare what versions should be matched and deployed automatically. You can have them deployed directly to production, or if it makes you more comfortable with your contractors, to an "auto-PR" branch which simply queues them up for a release engineer with the required juice to approve and merge the changes to prod. There is no deploy job in CI, since the deployment in prod is whatever happens to be on the main branch in the config repo. This is a protected branch which only releng can merge changes into, usually only when the release checks are satisfied. Flux (which scans any image repos) updates those manifests when there is a release, either directly or via pull request. This strategy I think would be able to address your Security concerns regarding contractors, by reducing the responsibility of your CI system to only CI, and not to handle CD anymore. This is the subject of my talk at KubeCon[1], although you might not be able to get that from the title, (I'm presenting this topic for a Jenkins audience, but the focus is on Flux and how Flux works) so I'm hoping it should be applicable to a broader audience, certainly inclusive of GitHub users :) [1]: https://kccncna2021.sched.com/event/lV0V/gitopsjenkins-ci-with-declarative-everything-kingdon-barrett-weaveworks?iframe=no https://kccncna2021.sched.com/event/lV0V/gitopsjenkins-ci-wi...
- lmeyerov 5y ago
  Yep -- and it goes deeper, even for CI. The CI/CD secrets vulnerability is just the most burning example. Ex: Network segmentation for CI. We only expect CI to communicate with dockerhub, conda/pypi, and the CI service (ex: azure). Something similar for CD. That _should_ be settable via GHA, but isn't. In a world of weird npm/python scripts and github action marketplace... scary we can't. Self-hosted runners can in theory do this via custom network policies, but it's a PITA for something ~everyone should be doing out-of-the-box. Edit: HN won't let me respond to the below. Imagine something like IP theft, wanting to mine our repo for everything labeled "security", dig into our version #'s for viable CVEs, or force-push some git commits (incl. history rewriting). Defense-in-depth says we shouldn't make that unnecessarily easy for anyone who plants a backdoor that runs during CI's `npm install` / `RUN xyz` phase. Network + RBAC are basically table stakes for almost everyone building enterprise software, so GHA doesn't have to reinvent the wheel here, just do it + make it friendly.
  - yebyen 5y ago
    I think those problems solve themselves when you can say "CI doesn't need to communicate with production at all anymore" – the Flux controllers live on the cluster, or on one cluster which is in position to communicate with others on the network in case of multi-cluster. They do not have any outward facing interfaces (except one), they live on the Kube API. (The Flux API is the CRD within the Kubernetes API.) The reason for CI to communicate with production is to handle deployments, right? So take away that responsibility, and now you can safely isolate GitHub Server and shunt it into a network that only has access to outside artifacts that are necessary, push access to an image registry, and that's it. Deployment status notifications can go from Flux to CI, so you can read about your deployments via commit status messages, but the communication is one-way. Changes are always pulled into the cluster and never pushed. You can set up webhooks in both directions, of course, for faster round trips (it makes a big difference for devs when iterating to not have to do a context switch and manually trigger a deploy, or to wait seconds not minutes and get immediate feedback after git push) but many will also opt not to do this in order to further confine and constrain to limit attack surfaces, it works just as well so long as they can live with some 5-10 minute waits for polling to do the job. Edit: heaven forbid we have a conversation, Hacker News won't allow fast replies :D thanks for giving examples
Pathogen-David 5y ago
> but not higher-trust GHA CD ones They improved this quite a bit last June. https://github.blog/changelog/2021-06-22-github-actions-environments-environment-protection-rules-and-environment-secrets-are-generally-available/ https://github.blog/changelog/2021-06-22-github-actions-envi... Basically you can define deployment environments which have their own secrets and configure it so only authorized users can approve workflows which access those environments.
- lmeyerov 5y ago
  Ah this is a good step. I hadn't looked since the env var vulns earlier in the ~year. Add in GH runner domain/vpc safelisting and a lot of the basics get solved!
  - jeremyepling 5y ago
    I think these will solve that. If not, let me know. https://github.com/github/roadmap/issues/249 https://github.com/github/roadmap/issues/249 https://github.com/github/roadmap/issues/220 https://github.com/github/roadmap/issues/220
    - lmeyerov 5y ago
      the first commenter was right on the env thing helping with ci/cd secret isolation the open issue is that GHA env is vulnerable to typical npm/pip/conda/etc supply chain attacks. it lets them do funny network requests (both data exfil + integrity, and even quota wastage / DoS): - GitHub hosted runners should allow inbound/outbound domain safelists, full isolation w internal repo safelisting, or something else like it - and/or self-hosted runners should have an Azure serverless mode where the host owner can do their own presets. less good b/c trickier to maintain typical safelists currently , GHA is default-allow for supply chain attacks =\ in theory folks can try docker-in-docker to limit network access, or the equiv lower, and layer on internal repos to prevent 0-days, but this gets gnarly quickly and more sanely handled by the managed CI env we are all paying for

yebyen 5y ago

I work on Flux, and for the reasons you mentioned we advocate for a strict separation between CI and CD, where CI is responsible for building and testing images and pushing out releases, and CD is responsible for production. There is no crossover between these roles, and this makes it very secure. You use a policy CRD called ImagePolicy to declare what versions should be matched and deployed automatically. You can have them deployed directly to production, or if it makes you more comfortable with your contractors, to an "auto-PR" branch which simply queues them up for a release engineer with the required juice to approve and merge the changes to prod. There is no deploy job in CI, since the deployment in prod is whatever happens to be on the main branch in the config repo. This is a protected branch which only releng can merge changes into, usually only when the release checks are satisfied. Flux (which scans any image repos) updates those manifests when there is a release, either directly or via pull request. This strategy I think would be able to address your Security concerns regarding contractors, by reducing the responsibility of your CI system to only CI, and not to handle CD anymore. This is the subject of my talk at KubeCon[1], although you might not be able to get that from the title, (I'm presenting this topic for a Jenkins audience, but the focus is on Flux and how Flux works) so I'm hoping it should be applicable to a broader audience, certainly inclusive of GitHub users :) [1]: https://kccncna2021.sched.com/event/lV0V/gitopsjenkins-ci-with-declarative-everything-kingdon-barrett-weaveworks?iframe=no https://kccncna2021.sched.com/event/lV0V/gitopsjenkins-ci-wi...

lmeyerov 5y ago
Yep -- and it goes deeper, even for CI. The CI/CD secrets vulnerability is just the most burning example. Ex: Network segmentation for CI. We only expect CI to communicate with dockerhub, conda/pypi, and the CI service (ex: azure). Something similar for CD. That _should_ be settable via GHA, but isn't. In a world of weird npm/python scripts and github action marketplace... scary we can't. Self-hosted runners can in theory do this via custom network policies, but it's a PITA for something ~everyone should be doing out-of-the-box. Edit: HN won't let me respond to the below. Imagine something like IP theft, wanting to mine our repo for everything labeled "security", dig into our version #'s for viable CVEs, or force-push some git commits (incl. history rewriting). Defense-in-depth says we shouldn't make that unnecessarily easy for anyone who plants a backdoor that runs during CI's `npm install` / `RUN xyz` phase. Network + RBAC are basically table stakes for almost everyone building enterprise software, so GHA doesn't have to reinvent the wheel here, just do it + make it friendly.
- yebyen 5y ago
  I think those problems solve themselves when you can say "CI doesn't need to communicate with production at all anymore" – the Flux controllers live on the cluster, or on one cluster which is in position to communicate with others on the network in case of multi-cluster. They do not have any outward facing interfaces (except one), they live on the Kube API. (The Flux API is the CRD within the Kubernetes API.) The reason for CI to communicate with production is to handle deployments, right? So take away that responsibility, and now you can safely isolate GitHub Server and shunt it into a network that only has access to outside artifacts that are necessary, push access to an image registry, and that's it. Deployment status notifications can go from Flux to CI, so you can read about your deployments via commit status messages, but the communication is one-way. Changes are always pulled into the cluster and never pushed. You can set up webhooks in both directions, of course, for faster round trips (it makes a big difference for devs when iterating to not have to do a context switch and manually trigger a deploy, or to wait seconds not minutes and get immediate feedback after git push) but many will also opt not to do this in order to further confine and constrain to limit attack surfaces, it works just as well so long as they can live with some 5-10 minute waits for polling to do the job. Edit: heaven forbid we have a conversation, Hacker News won't allow fast replies :D thanks for giving examples

Pathogen-David 5y ago

> but not higher-trust GHA CD ones They improved this quite a bit last June. https://github.blog/changelog/2021-06-22-github-actions-environments-environment-protection-rules-and-environment-secrets-are-generally-available/ https://github.blog/changelog/2021-06-22-github-actions-envi... Basically you can define deployment environments which have their own secrets and configure it so only authorized users can approve workflows which access those environments.

lmeyerov 5y ago
Ah this is a good step. I hadn't looked since the env var vulns earlier in the ~year. Add in GH runner domain/vpc safelisting and a lot of the basics get solved!
- jeremyepling 5y ago
  I think these will solve that. If not, let me know. https://github.com/github/roadmap/issues/249 https://github.com/github/roadmap/issues/249 https://github.com/github/roadmap/issues/220 https://github.com/github/roadmap/issues/220
  - lmeyerov 5y ago
    the first commenter was right on the env thing helping with ci/cd secret isolation the open issue is that GHA env is vulnerable to typical npm/pip/conda/etc supply chain attacks. it lets them do funny network requests (both data exfil + integrity, and even quota wastage / DoS): - GitHub hosted runners should allow inbound/outbound domain safelists, full isolation w internal repo safelisting, or something else like it - and/or self-hosted runners should have an Azure serverless mode where the host owner can do their own presets. less good b/c trickier to maintain typical safelists currently , GHA is default-allow for supply chain attacks =\ in theory folks can try docker-in-docker to limit network access, or the equiv lower, and layer on internal repos to prevent 0-days, but this gets gnarly quickly and more sanely handled by the managed CI env we are all paying for