4 ms·

GitHub Actions with act to test stuff locally is actually pretty usable. We use on-prem k8s-hosted runners to get access to servers/clusters with limited intern

GitHub Actions with act to test stuff locally is actually pretty usable. We use on-prem k8s-hosted runners to get access to servers/clusters with limited internet access, works great. But as always with “”gitops”” themed tools I think it’s pretty awkward to handle rollbacks. Either you take the stance that master is the source of truth and let the CI/CD tooling revert commits if deployment fails, or you store that state elsewhere and allow e.g manifests to diverge from git. I would be curious to hear how other people do it

bilalq 5y ago
Like all terms in the industry, "gitops" gets defined a little differently by everyone. In our case, we use trunk-based development where "main" is the branch that is approved for release, but it does not necessarily point to what is currently released. Instead, we use git tags for that. On merge to main, we kick off pipeline executions (we use AWS CodePipeline here, but Github actions with a concurrency group id set would achieve something similar). Non-prod stages just push lightweight tags in the format of `$STAGE-YYYY.MM.DD.hhmm` on successful deployment. Prod stages are similar, but we publish a Github release declaration using the Github API rather than just pushing a git tag. In the case of rollback, you either push new tags at older commits or roll forward with a revert commit (but still using tags as the tracking mechanism).
- ynouri 5y ago
  Thanks for sharing! We are a small company and right now we release (from our monorepo) on every PR merge to main. As the engineering team continues growing, we anticipate this approach will become unsustainable soon, and are looking for alternatives. I wonder if the tag based release is a widespread industry pattern (especially for growing monorepos)? Would be curious to learn more about it
  - DrewRWx 5y ago
    I'm on a team that implemented a hybrid tag-based release system on a monorepo and it is working well. External releases are built on release branches off of main with their own git tag rules, but I want to touch on internal releases off of main. Between feature branches and main we have GitHub status checks that gate bad PRs from getting in. Once the PRs are in main, we build a nightly full suite of components and put them through various levels of interop testing. Once our system receives the signal that all builds and interop testing had passed, it applies a Last Known Good (LKG) tag to the git commit the components were built from. After that, various systems including artifact links for QA and auto-merge jobs from main, are set to use the LKG tag to ensure they are getting good builds and code.
  - bilalq 5y ago
    I was honestly shocked that I couldn't find many blog posts or articles about managing this process. The tag based mechanism was one I came up with on my own because we needed something. We're also a small team of 5 engineers, with many small microservices that each have their own pipelines. The problem with main being the release tracker is that it doesn't work if you have multiple environment stages. Stage specific branches work, but then you don't get history.

bilalq 5y ago

Like all terms in the industry, "gitops" gets defined a little differently by everyone. In our case, we use trunk-based development where "main" is the branch that is approved for release, but it does not necessarily point to what is currently released. Instead, we use git tags for that. On merge to main, we kick off pipeline executions (we use AWS CodePipeline here, but Github actions with a concurrency group id set would achieve something similar). Non-prod stages just push lightweight tags in the format of `$STAGE-YYYY.MM.DD.hhmm` on successful deployment. Prod stages are similar, but we publish a Github release declaration using the Github API rather than just pushing a git tag. In the case of rollback, you either push new tags at older commits or roll forward with a revert commit (but still using tags as the tracking mechanism).

ynouri 5y ago
Thanks for sharing! We are a small company and right now we release (from our monorepo) on every PR merge to main. As the engineering team continues growing, we anticipate this approach will become unsustainable soon, and are looking for alternatives. I wonder if the tag based release is a widespread industry pattern (especially for growing monorepos)? Would be curious to learn more about it
- DrewRWx 5y ago
  I'm on a team that implemented a hybrid tag-based release system on a monorepo and it is working well. External releases are built on release branches off of main with their own git tag rules, but I want to touch on internal releases off of main. Between feature branches and main we have GitHub status checks that gate bad PRs from getting in. Once the PRs are in main, we build a nightly full suite of components and put them through various levels of interop testing. Once our system receives the signal that all builds and interop testing had passed, it applies a Last Known Good (LKG) tag to the git commit the components were built from. After that, various systems including artifact links for QA and auto-merge jobs from main, are set to use the LKG tag to ensure they are getting good builds and code.
- bilalq 5y ago
  I was honestly shocked that I couldn't find many blog posts or articles about managing this process. The tag based mechanism was one I came up with on my own because we needed something. We're also a small team of 5 engineers, with many small microservices that each have their own pipelines. The problem with main being the release tracker is that it doesn't work if you have multiple environment stages. Stage specific branches work, but then you don't get history.