5 ms·

> With this approach, a single PR can atomically merge code and update one or more release specifications (triggering CD for those components), if and when that

> With this approach, a single PR can atomically merge code and update one or more release specifications (triggering CD for those components), if and when that is a sensible thing to do. How do you avoid the chicken-and-egg problem? Like if the k8s manifest contains a container tag, and the tag is created by CI when the PR is merged to main, it would seem you can’t add code and deploy that code in the same PR.

derefr 2y ago
I haven't seen this approach in action personally, but I assume you'd set it up similarly to how Homebrew builds + pins bottles for formulae: - PR creation (or any update to the PR branch by a human) would trigger a CI workflow to build+push the container; - if this succeeds, the workflow would push a commit to the PR feature branch that pins the container ref; - and the base branch would have a branch protection rule that makes the success of this CI workflow a prerequisite for PR mergeability. In the explicit-GitOps-in-monorepo case, you'd probably want to do this by having the PR author modify the release spec file of each component they want to build a release for, replacing the old static ref with a temporary symbolic ref meaning "hey CI system, calculate this." Then the CI's added commit would rewrite those temporary symbolic refs into new static ones. --- Although, that's all assuming that you even want to do CI builds in the first place. If you're developing containerized server software — rather than an operating system, a web browser, etc — then it shouldn't matter where you're building, and there isn't much impetus for deterministic, auditable builds, either. So why bottleneck builds by shoving them all onto some wimpy CI system—when every engineer already has a powerful workstation that can do builds often 4x faster, sitting there idle? Here's what I call "local-first explicit GitOps": 1. you give your engineers the ability to push refs to the container registry (but you never use symbolic refs, always static manifest SHAs, so a malicious engineer can't do anything dangerous with this. 2. you add a script to your monorepo, that an engineer can run against their feature branch, that'll notice their current branch's local symbolic refs, and build+push containers locally in order to rewrite those into static refs. 3. On success, the script can use `gh` (or equivalent) to trigger PR creation. Now every PR will come into existence already pinning valid new build artifacts by their static refs!
someone654 2y ago
If you use helm charts or similar, the version of the image would be a variable that is updated outside the repo. ArgoCD can do this with the Application resource. For example: build code and image, update git reference and version number in Application when build done. You then get atomic updates of both code and configuration.
claytonjy 2y ago
I just implemented something like this in Github Actions for a single python application which includes 2 containers and a model repository, deployed to Kubernetes via Argo from the in-repo Helm chart. The same semantic version string is in pyproject.toml and values.yaml. These are enforced to be the same via pre-commit hook. Environment-specific versions are in env-specific values files like values-prod.yaml. I have a build workflow which triggers on PRs that change certain files, computes either a pre- or post-release tag by appending the git hash to the server tag, and pushes the images and model repo using that tag. This will re-trigger as the PR is updated. Users deploy these pre/post-releases in a few clicks with a manual github action which updates the branch and tag in our dev cluster Argo. The PR can’t be merged unless the version is bumped. When the PR is merged, a “promote” workflow runs which re-tags the latest image and model repo with the release version (pre-release without git hash suffix). This is fast, which means there’s no risk of image backoff loop from Argo trying to sync before artifacts are available. Most PRs don’t touch the version in values-prod.yaml, so merges to main auto-deploy to staging while prod requires another PR updating only values-prod.yaml. Folks complain a bit about manually bumping the version, and needing 2 PRs to take a change to prod, but it’s pretty smooth otherwise!
- mtoohig 2y ago
  I would like to see the details of this, do you have the source available publicly or a blog post about it?
  - claytonjy 2y ago
    Sadly no; internal work stuff.

derefr 2y ago

I haven't seen this approach in action personally, but I assume you'd set it up similarly to how Homebrew builds + pins bottles for formulae: - PR creation (or any update to the PR branch by a human) would trigger a CI workflow to build+push the container; - if this succeeds, the workflow would push a commit to the PR feature branch that pins the container ref; - and the base branch would have a branch protection rule that makes the success of this CI workflow a prerequisite for PR mergeability. In the explicit-GitOps-in-monorepo case, you'd probably want to do this by having the PR author modify the release spec file of each component they want to build a release for, replacing the old static ref with a temporary symbolic ref meaning "hey CI system, calculate this." Then the CI's added commit would rewrite those temporary symbolic refs into new static ones. --- Although, that's all assuming that you even want to do CI builds in the first place. If you're developing containerized server software — rather than an operating system, a web browser, etc — then it shouldn't matter where you're building, and there isn't much impetus for deterministic, auditable builds, either. So why bottleneck builds by shoving them all onto some wimpy CI system—when every engineer already has a powerful workstation that can do builds often 4x faster, sitting there idle? Here's what I call "local-first explicit GitOps": 1. you give your engineers the ability to push refs to the container registry (but you never use symbolic refs, always static manifest SHAs, so a malicious engineer can't do anything dangerous with this. 2. you add a script to your monorepo, that an engineer can run against their feature branch, that'll notice their current branch's local symbolic refs, and build+push containers locally in order to rewrite those into static refs. 3. On success, the script can use `gh` (or equivalent) to trigger PR creation. Now every PR will come into existence already pinning valid new build artifacts by their static refs!

someone654 2y ago

If you use helm charts or similar, the version of the image would be a variable that is updated outside the repo. ArgoCD can do this with the Application resource. For example: build code and image, update git reference and version number in Application when build done. You then get atomic updates of both code and configuration.

claytonjy 2y ago

I just implemented something like this in Github Actions for a single python application which includes 2 containers and a model repository, deployed to Kubernetes via Argo from the in-repo Helm chart. The same semantic version string is in pyproject.toml and values.yaml. These are enforced to be the same via pre-commit hook. Environment-specific versions are in env-specific values files like values-prod.yaml. I have a build workflow which triggers on PRs that change certain files, computes either a pre- or post-release tag by appending the git hash to the server tag, and pushes the images and model repo using that tag. This will re-trigger as the PR is updated. Users deploy these pre/post-releases in a few clicks with a manual github action which updates the branch and tag in our dev cluster Argo. The PR can’t be merged unless the version is bumped. When the PR is merged, a “promote” workflow runs which re-tags the latest image and model repo with the release version (pre-release without git hash suffix). This is fast, which means there’s no risk of image backoff loop from Argo trying to sync before artifacts are available. Most PRs don’t touch the version in values-prod.yaml, so merges to main auto-deploy to staging while prod requires another PR updating only values-prod.yaml. Folks complain a bit about manually bumping the version, and needing 2 PRs to take a change to prod, but it’s pretty smooth otherwise!

mtoohig 2y ago
I would like to see the details of this, do you have the source available publicly or a blog post about it?
- claytonjy 2y ago
  Sadly no; internal work stuff.