3 ms·

Ok, but the more interesting part - how did you solve the CI/CD part and how does it compare to a multirepo?

by siva7 2y ago

CharlieDigital 2y ago
Most CI/CD platforms will allow specification of targeted triggers. For example, in GitHub[0]: name: ".NET - PR Unit Test" on: ## Only execute these unit tests when a file in this directory changes. pull_request: branches: [main] paths: [src/services/publishing/**.cs, src/tests/unit/**.cs] So we set up different workflows that kick off based on the sets of files that change. [0] https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore https://docs.github.com/en/actions/writing-workflows/workflo...
- victorNicollet 2y ago
  I'm not familiar with GitHub Actions, but we reverted our migration to Bitbucket Pipelines because of a nasty side-effect of conditional execution: if a commit triggers test suite T1 but not T2, and T1 is successful, Bitbucket displays that commit with a green "everything is fine" check mark, regardless of the status of T2 on any ancestors of that commit. That is, the green check mark means "the changes in this commit did not break anything that was not already broken", as opposed to the more useful "the repository, as of this commit, passes all tests".
  - ants_everywhere 2y ago
    isn't that generally what you want? the check mark tells you the commit didn't break anything. if something was already broken it should have either blocked the commit that broke it or there's a flake somewhere that you can only locate by periodically running tests independent of any PR activity.
  - daelon 2y ago
    Is it a side effect if it's also the primary effect?
  - plorkyeran 2y ago
    I would find it extremely confusing and unhelpful if tests in the parent commit which weren't rerun for a PR because nothing relevant was touched marked the PR as red. Why would you even want that? That's not something which is relevant to evaluating the PR and would make you get in the habit of ignoring failures. If you split something into multiple repositories then surely you wouldn't mark PRs on one of them as red just because tests are failing in a different one?
    - victorNicollet 2y ago
      I suppose our development process is a bit unusual. The meaning we give to "the commit is green" is not "this PR can be merged" but "this can be deployed to production", and it is used for the purpose of selecting a release candidate several times a week. It is a statement about the entire state of the project as of that commit, rather than just the changes introduced in that commit. I can understand the frustration of creating a PR from a red commit on the main branch, and having that PR be red as well as a result. I can't say this has happened very often, though: red commits on the main branch are very rare, and new branches tend to be started right after a deployment, so it's overwhelmingly likely that the PR will be rooted at a green commit. When it does happen, the time it takes to push a fix (or a revert) to the main branch is usually much shorter than the time for a review of the PR, which means it is possible to rebase the PR on top of a green commit as part of the normal PR acceptance timeline.
      - plorkyeran 2y ago
        Going off the PR status to determine if the end result is deployable is not reliable. A non-FF merge can have both the base commit and the PR be green but the merged result fail. You need to run your full test suite on the merged result at some point before deployment; either via a commit queue or post-merge testing.
        victorNicollet 2y ago
        I agree ! We use the commit status instead of the PR status. A non-FF merge commit, being a commit, would have its own status separate from the status of its parents.
- hk1337 2y ago
  Even AWS CodeBuild (or CodePipeline) allows you to do this now. It didn't before but it's a fairly recent update.
  - CharlieDigital 2y ago
    As a prior user of AWS Code*, I can appreciate that you qualified that with "Even" LMAO
devjab 2y ago
I don’t think CI/CD should really be a big worry as far as mono-repositories go as you can setup different pipelines and different flows with different configurations. Something you’re probably already doing if you have multiple repos. In my experience the article is right when it tells you there isn’t that big of a difference. We have all sorts of repositories, some of which are basically mono-repositories for their business domain. We tend to separate where it “makes sense” which for us means that it’s when what we put into repositories is completely separate from everything else. We used to have a lot of micro-repositories and it wasn’t that different to be honest. We grouped more of them together to make it easier for us to be DORA compliant in terms of the bureaucracy it adds to your documentation burden. Technically I hardly notice.
- JamesSwift 2y ago
  In my limited-but-not-nothing experience working with mono vs multi repo of the same projects, CI/CD definitely was one of the harder pieces to solve. Its highly dependent on your frameworks and CI provider on just how straightforward it is going to be, and most of them are "not very straightforward". The basic way most work is to run full CI on every change. This quickly becomes a huge speedbump to deployment velocity until a solution for "only run what is affected" is found.
  - bluGill 2y ago
    The problem with "only run what is affected" is it is really easy to have something that is affected but doesn't seem like it should be (that is whatever tools you have to detect is it affected say it isn't). So if you have such a system you must have regular rebuild everything jobs as well to verify you didn't break something unexpected. I'm not against only run what is affected, it is a good answer. It just has failings that you need to be aware of.
    - JamesSwift 2y ago
      Yeah thats a good point. Especially for an overly-dynamic runtime like ruby/rails, theres just not usually a clean way to cordon off sections of code. On the other hand, using nx in an angular project was pretty amazing.
victorNicollet 2y ago
Wouldn't CI be easier with a monorepo ? Testing integration across multiple repositories (triggered by changes in any of them) seems more complex than just adding another test suite to a single repo.
- bluGill 2y ago
  Pros and cons. Both can be used successfully, but there are different problems to each. If you have a large project you will have a tool teams to deal with the problems of your solution.
habosa 2y ago
I think this is one of the big reasons so many companies end up using Bazel as part of scaling their monorepos. It has many faults but one thing it does perfectly is “build and test everything affected by this commit”
IshKebab 2y ago
You use a build system that sandboxes dependencies (Bazel, Buck2, Please, etc.) so that you know there are no undeclared dependencies. Then you can simply query the build system "what could have been affected by this change" and only build/test those things.

Ok, but the more interesting part - how did you solve the CI/CD part and how does it compare to a multirepo?

by siva7 2y ago

Ok, but the more interesting part - how did you solve the CI/CD part and how does it compare to a multirepo?

Most CI/CD platforms will allow specification of targeted triggers. For example, in GitHub[0]: name: ".NET - PR Unit Test" on: ## Only execute these unit tests when a file in this directory changes. pull_request: branches: [main] paths: [src/services/publishing/**.cs, src/tests/unit/**.cs] So we set up different workflows that kick off based on the sets of files that change. [0] https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore https://docs.github.com/en/actions/writing-workflows/workflo...

victorNicollet 2y ago
I'm not familiar with GitHub Actions, but we reverted our migration to Bitbucket Pipelines because of a nasty side-effect of conditional execution: if a commit triggers test suite T1 but not T2, and T1 is successful, Bitbucket displays that commit with a green "everything is fine" check mark, regardless of the status of T2 on any ancestors of that commit. That is, the green check mark means "the changes in this commit did not break anything that was not already broken", as opposed to the more useful "the repository, as of this commit, passes all tests".
- ants_everywhere 2y ago
  isn't that generally what you want? the check mark tells you the commit didn't break anything. if something was already broken it should have either blocked the commit that broke it or there's a flake somewhere that you can only locate by periodically running tests independent of any PR activity.
- daelon 2y ago
  Is it a side effect if it's also the primary effect?
- plorkyeran 2y ago
  I would find it extremely confusing and unhelpful if tests in the parent commit which weren't rerun for a PR because nothing relevant was touched marked the PR as red. Why would you even want that? That's not something which is relevant to evaluating the PR and would make you get in the habit of ignoring failures. If you split something into multiple repositories then surely you wouldn't mark PRs on one of them as red just because tests are failing in a different one?
  - victorNicollet 2y ago
    I suppose our development process is a bit unusual. The meaning we give to "the commit is green" is not "this PR can be merged" but "this can be deployed to production", and it is used for the purpose of selecting a release candidate several times a week. It is a statement about the entire state of the project as of that commit, rather than just the changes introduced in that commit. I can understand the frustration of creating a PR from a red commit on the main branch, and having that PR be red as well as a result. I can't say this has happened very often, though: red commits on the main branch are very rare, and new branches tend to be started right after a deployment, so it's overwhelmingly likely that the PR will be rooted at a green commit. When it does happen, the time it takes to push a fix (or a revert) to the main branch is usually much shorter than the time for a review of the PR, which means it is possible to rebase the PR on top of a green commit as part of the normal PR acceptance timeline.
    - plorkyeran 2y ago
      Going off the PR status to determine if the end result is deployable is not reliable. A non-FF merge can have both the base commit and the PR be green but the merged result fail. You need to run your full test suite on the merged result at some point before deployment; either via a commit queue or post-merge testing.
      - victorNicollet 2y ago
        I agree ! We use the commit status instead of the PR status. A non-FF merge commit, being a commit, would have its own status separate from the status of its parents.
hk1337 2y ago
Even AWS CodeBuild (or CodePipeline) allows you to do this now. It didn't before but it's a fairly recent update.
- CharlieDigital 2y ago
  As a prior user of AWS Code*, I can appreciate that you qualified that with "Even" LMAO

devjab 2y ago

I don’t think CI/CD should really be a big worry as far as mono-repositories go as you can setup different pipelines and different flows with different configurations. Something you’re probably already doing if you have multiple repos. In my experience the article is right when it tells you there isn’t that big of a difference. We have all sorts of repositories, some of which are basically mono-repositories for their business domain. We tend to separate where it “makes sense” which for us means that it’s when what we put into repositories is completely separate from everything else. We used to have a lot of micro-repositories and it wasn’t that different to be honest. We grouped more of them together to make it easier for us to be DORA compliant in terms of the bureaucracy it adds to your documentation burden. Technically I hardly notice.

JamesSwift 2y ago
In my limited-but-not-nothing experience working with mono vs multi repo of the same projects, CI/CD definitely was one of the harder pieces to solve. Its highly dependent on your frameworks and CI provider on just how straightforward it is going to be, and most of them are "not very straightforward". The basic way most work is to run full CI on every change. This quickly becomes a huge speedbump to deployment velocity until a solution for "only run what is affected" is found.
- bluGill 2y ago
  The problem with "only run what is affected" is it is really easy to have something that is affected but doesn't seem like it should be (that is whatever tools you have to detect is it affected say it isn't). So if you have such a system you must have regular rebuild everything jobs as well to verify you didn't break something unexpected. I'm not against only run what is affected, it is a good answer. It just has failings that you need to be aware of.
  - JamesSwift 2y ago
    Yeah thats a good point. Especially for an overly-dynamic runtime like ruby/rails, theres just not usually a clean way to cordon off sections of code. On the other hand, using nx in an angular project was pretty amazing.

victorNicollet 2y ago

Wouldn't CI be easier with a monorepo ? Testing integration across multiple repositories (triggered by changes in any of them) seems more complex than just adding another test suite to a single repo.

bluGill 2y ago
Pros and cons. Both can be used successfully, but there are different problems to each. If you have a large project you will have a tool teams to deal with the problems of your solution.

habosa 2y ago

I think this is one of the big reasons so many companies end up using Bazel as part of scaling their monorepos. It has many faults but one thing it does perfectly is “build and test everything affected by this commit”

IshKebab 2y ago

You use a build system that sandboxes dependencies (Bazel, Buck2, Please, etc.) so that you know there are no undeclared dependencies. Then you can simply query the build system "what could have been affected by this change" and only build/test those things.